Skip to content

Introducing NLRS: Non-Standard Linear Regression Methods with Sparsity

During my PhD, I worked on feature selection and regularization in the context of index tracking. Reading up on methods such as multi-task learning, the adaptive LASSO, and support vector regression (SVR), I kept wanting to try these ideas out directly in my own workflows.

Most of these models are available in libraries such as scikit-learn. scikit-learn does not, however, readily accommodate requirements that deviate from these standard cases.

For example:

  • Multi-task regression with non-standard losses
  • Adaptive regularization for feature selection
  • Positivity constraints on coefficients
  • SVR with custom penalties

Combining several of these requirements usually means implementing the optimization problem from scratch.

I therefore implemented them in CVXPY; the result is the NLRS (Non-Standard Linear Regression Methods with Sparsity) Python package.

NLRS formulates linear regression and SVR as convex optimization problems, so that loss functions, penalties, and constraints can be combined freely while remaining compatible with scikit-learn tooling (pipelines, GridSearchCV) and Optuna-based tuning.

The source code is available on GitHub.

Many of the individual building blocks already exist in scikit-learn. What NLRS adds is the freedom to combine them:

  • Adaptive regularization (adaptive LASSO and elastic net)
  • Positivity constraints for every model and loss
  • Any supported loss (Huber, quantile, ε-insensitive, MAE) paired with any supported penalty; for a combined \(\ell_1\ell_2\) penalty the adaptive weights apply to the \(\ell_1\) component only
  • Multi-task regression with robust losses

Example: robust multi-task feature selection with a linear SVR

Coefficient heatmaps and selection masks for multi-task LASSO and multi-task SVR

The figure above shows a simulated multi-task regression problem with pronounced outliers in the targets.

Coefficient heatmaps (top row)

  • Ground truth: the true coefficient values across tasks.
  • Multi-task LASSO: the squared loss is sensitive to outliers, which leaves the coefficient estimates noisy and distorted.
  • Multi-task SVR: the ε-insensitive loss dampens the influence of outliers and recovers a much cleaner signal.

Binary selection masks (bottom row)

  • Multi-task LASSO: performs poorly in the presence of outliers.
  • Multi-task SVR: recovers the correct sparse structure, selecting the same features consistently across tasks.

In this simulation the multi-task SVR combines accurate estimation, robustness to outliers, and consistent feature selection across tasks.