Skip to content

LASSO and Adaptive LASSO

How a model determines which variables are relevant is a question of feature selection and regularization.

One of the most widely used approaches is LASSO (Least Absolute Shrinkage and Selection Operator), introduced by Tibshirani in 1996.

LASSO performs automatic feature selection: its \(\ell_1\) penalty shrinks coefficients toward zero and sets some of them exactly to zero.

Shrinkage, however, introduces bias in the estimated coefficients, which need not be offset by the accompanying reduction in variance.

In his 2006 paper “The Adaptive LASSO and Its Oracle Properties”, Hui Zou showed that standard LASSO can be inconsistent for variable selection.

Because the method applies the same penalty weight to every coefficient, it can over-shrink large coefficients while retaining noise variables, particularly when the predictors are strongly correlated.

To address this, he proposed the adaptive LASSO. Instead of penalizing all coefficients equally, it assigns each coefficient its own weight:

\[\hat\beta^{\text{adaptive}} = \arg\min_{\beta} \; \frac{1}{2n} \vert\vert X \beta - y \vert\vert_2^2 + \alpha \sum_{j=1}^{p} w_j \vert \beta_j \vert, \qquad w_j = \frac{1}{\vert \hat\beta_j \vert^{\gamma}}, \; \gamma > 0\]

The weights are derived from an initial estimate \(\hat\beta\), for example from OLS or ridge regression:

  • large initial coefficients \(\vert \hat\beta_j \vert\) receive smaller penalties;
  • small initial coefficients receive stronger penalties.

Under suitable conditions, the method attains the oracle property asymptotically: in large samples it behaves as if it had known all along which features truly matter.

Coefficient estimates from LASSO and Adaptive LASSO compared with the true values

The plot illustrates this idea:

  • Grey circles: the true coefficient values
  • Blue crosses: estimates from the standard LASSO
  • Orange triangles: estimates from the adaptive LASSO

The standard LASSO visibly shrinks several coefficients toward zero and distorts some of the larger ones, whereas the adaptive LASSO recovers the large true coefficients more accurately and zeroes out the irrelevant ones more reliably.