NLRS: Non-Standard Linear Regression Methods with Sparsity

A Python Library for Non-Standard Linear Regression Methods with Sparsity.

The source code is available on GitHub.

Description

The nlrs library formulates linear regression and support vector regression (SVR) as convex optimization problems with a range of regularizers: the sparsity-inducing Lasso and Elastic Net, as well as the non-sparse Ridge.

Built on the Python-embedded modeling language CVXPY, nlrs complements scikit-learn by adding modeling constraints, loss functions, and adaptive penalties that scikit-learn does not cover. Because the problems are solved through CVXPY, nlrs interfaces natively with open-source solvers such as ECOS, OSQP, Clarabel, and SCS, as well as commercial ones such as CPLEX and Gurobi.

Key capabilities

• Adaptive penalties: models support adaptive=True, which scales \(\ell_1\) or \(\ell_2\) penalties by inverse-magnitude weights of the form \(\omega_j = 1/|\hat\beta_j|^{\gamma}\) with \(\gamma > 0\) (often \(\gamma = 1\)), where \(\hat\beta\) comes from a previous fit, for example a linear regression or RidgeCV. Note that the \(\ell_2\) variants shrink the coefficients but do not yield sparse solutions. For combined l1_l2 penalties, the adaptive weights apply to the \(\ell_1\) component only.
• Constraints: positive=True restricts all estimated coefficients to \(\geq 0\).
• Multi-task regression: MultiTaskRegressor supports grouped shrinkage (Lasso, Elastic Net, SVR) by applying the mixed \(\ell_{2,1}\) norm, so that feature relevance is assessed jointly across all target variables.
• scikit-learn compatibility: all models inherit from BaseEstimator and RegressorMixin and are designed to work with scikit-learn cross-validation, pipelines, and grid search.

---

Documentation

Read the documentation on GitLab Pages.

Installation

You can install nlrs locally by cloning the repository and installing it via pip:

git clone https://github.com/trholy/nlrs
cd nlrs
pip install .

---

Methods

The formulations below represent the objectives minimized via CVXPY.

Standard linear models (nlrs.linear_model.linear)

LinearRegression

• Loss: \(\vert\vert X \beta - y \vert\vert_2^2\)
• Constraints: fit_intercept, positive=True

Lasso

• Loss: \(\frac{1}{2n} \vert\vert X \beta - y \vert\vert_2^2\)
• Penalties:
  • \(\ell_1\)-penalty: \(\alpha \cdot \vert\vert \beta \vert\vert_1\)
  • Adaptive \(\ell_1\)-penalty: \(\alpha \cdot \sum_{j=1}^{p} \omega_j | \beta_j |\)

Ridge

• Loss: \(\vert\vert X \beta - y \vert\vert_2^2\)
• Penalties:
  • \(\ell_2\)-penalty: \(\alpha \cdot \vert\vert \beta \vert\vert_2^2\)
  • Adaptive \(\ell_2\)-penalty: \(\alpha \cdot \sum_{j=1}^{p} \omega_j \beta_j^2\)

ElasticNet

• Loss: \(\frac{1}{2n} \vert\vert X \beta - y \vert\vert_2^2\)
• Penalties: combines \(\ell_1\) (scaled by \(\alpha \cdot \ell_1\text{-ratio}\)) and \(\ell_2\) (scaled by \(\tfrac{1}{2} \cdot \alpha \cdot (1 - \ell_1\text{-ratio})\)), i.e. with \(\rho = \ell_1\text{-ratio}\):

\[\frac{1}{2n} \vert\vert X \beta - y \vert\vert_2^2 + \alpha \rho \vert\vert \beta \vert\vert_1 + \frac{\alpha (1 - \rho)}{2} \vert\vert \beta \vert\vert_2^2\]

MultiTaskRegressor

• Loss functions: squared_error, epsilon_insensitive, squared_epsilon_insensitive, huber, quantile, mae (alias: mean_absolute_error).
• Penalties: for a coefficient matrix \(B\) (features × tasks), the \(\ell_{2,1}\) penalty \(\sum_j \|B_{j\cdot}\|_2\) sums the \(\ell_2\) norms of the rows, i.e. of each feature's coefficients across all tasks. This removes a feature from all tasks jointly.
• Adaptive weights: not supported for multi-task regression.

Support vector machines (nlrs.linear_model.svm)

LinearSVR

Uses the epsilon-insensitive loss \(L_\epsilon(r)=\max(0,|r|-\epsilon)\): residuals with \(|r|\le\epsilon\) incur zero loss.

• Loss functions: all losses from nlrs.objectives.losses.get_loss_expr are supported: squared_error, sum_squares, epsilon_insensitive, squared_epsilon_insensitive, huber, quantile, mae (alias mean_absolute_error). With squared_error, the model is effectively a penalized least-squares regression rather than an SVR.
• Penalties: \(\ell_1\), \(\ell_2\), and elastic net penalties are built in. Adaptive weights apply to a pure \(\ell_1\) or \(\ell_2\) penalty; for the combined \(\ell_1\ell_2\) penalty, only the \(\ell_1\) part is weighted.
• A note on the intercept: nlrs.linear_model.svm.LinearSVR does not regularize the intercept, whereas scikit-learn and liblinear do regularize the intercept feature (intercept_scaling).

---

Project structure

./
├── .gitignore
├── .gitlab-ci.yml
├── LICENSE
├── README.md
├── THIRD_PARTY_LICENSES.txt
├── docs
│   ├── linear_model
│   │   ├── base.md
│   │   ├── linear.md
│   │   └── svm.md
│   ├── objectives
│   │   ├── losses.md
│   │   └── regularizers.md
│   ├── solvers
│   │   └── base.md
│   └── utils
│       ├── selection.md
│       └── validation.md
├── examples
│   ├── adaptive_lasso.py
│   ├── multitask_lasso.py
│   └── multitask_svr.py
├── mkdocs.yml
├── pyproject.toml
├── setup.py
├── src
│   └── nlrs
│       ├── __init__.py
│       ├── linear_model
│       │   ├── __init__.py
│       │   ├── base.py
│       │   ├── linear.py
│       │   └── svm.py
│       ├── objectives
│       │   ├── __init__.py
│       │   ├── losses.py
│       │   └── regularizers.py
│       ├── solvers
│       │   ├── __init__.py
│       │   └── base.py
│       └── utils
│           ├── __init__.py
│           ├── selection.py
│           └── validation.py
└── tests
    ├── test_AdaptiveWeights.py
    ├── test_adaptive.py
    ├── test_adaptive_advanced.py
    ├── test_collinear.py
    ├── test_edge_cases.py
    ├── test_linear.py
    ├── test_losses.py
    ├── test_multitask.py
    ├── test_multitask_advanced.py
    ├── test_regularizers.py
    ├── test_sklearn_compat.py
    ├── test_svm.py
    └── test_validation.py

---

License

This project is licensed under the MIT License. See the LICENSE file for more details.