Skip to content

Multi-task regression with the LASSO

Some machine learning models can learn several related prediction tasks at once, a paradigm known as multi-task learning.

Rather than fitting each task in isolation, joint learning can exploit structure shared across tasks and thereby improve feature selection.

A classic approach is Multi-Task LASSO, introduced by Obozinski, Taskar, and Jordan (2006).

The standard LASSO applies \(\ell_1\) regularization to each task separately, encouraging sparsity in each model's coefficients.

The multi-task LASSO extends the idea by applying a group penalty instead, the mixed \(\ell_{2,1}\) norm \(\sum_j \lVert B_{j\cdot} \rVert_2\) over the rows of the coefficient matrix \(B\) (features × tasks):

  • The \(\ell_2\) norm groups each feature's coefficients across tasks.

  • The \(\ell_1\) norm is then applied across these groups.

This encourages a shared sparsity pattern, so that the same features tend to be selected, or dropped, jointly across all models.

The difference becomes clear when the learned coefficients are visualized as heatmaps (top row) alongside the resulting selection patterns (bottom row):

Coefficient heatmaps and selection masks for independent and multi-task LASSO

Coefficient heatmaps (top row)

  • Ground truth: the true importance of each feature across tasks.

  • Independent LASSO: estimates are comparatively poor in this simulation, since each task is treated on its own.

  • Multi-task LASSO: recovers the underlying structure far more accurately.

Binary selection masks (bottom row)

  • Independent LASSO: selection is inconsistent — the same feature may be picked for some tasks and dropped for others.

  • Multi-task LASSO: selects the same features across all tasks by construction, and here that pattern matches the ground truth.

By sharing information between tasks, the multi-task LASSO can yield more stable feature selection and better performance, particularly when the data per task is limited.