Lasso coefficient for some features is higher than Linear Regression Coefficient

Question

I'm using Lasso Regularization to avoid overfitting & multicollinearity between two features (X1 and X2), nowing that I have 14 independent features. I got some good results for some features, Lasso was able to reduce the coefficient to 0, but for other features the linear regression coefficient was less than Lasso (same thing for Ridge).

lr = LinearRegression()
lr.fit(X, Y)
lr_coeff = lr.coef_
lr_intercept = lr.intercept_

lasso = Lasso(alpha=10)
lasso.fit(X, Y)
lasso_coeff = lasso.coef_
lasso_intercept = lasso.intercept_

Result:

    lr_coeff  lr_intercept  lasso_coeff  lasso_intercept
0   0.968567      16.01858     0.000000       103.471224
1   1.743420      16.01858     1.730920       103.471224
2   5.221518      16.01858     3.931450       103.471224
3   4.769328      16.01858     3.186003       103.471224
4   6.341612      16.01858     4.265931       103.471224
5   2.272504      16.01858     1.277541       103.471224
6   3.104016      16.01858     1.648253       103.471224
7   1.418943      16.01858     0.667189       103.471224
8   1.144834      16.01858     0.000000       103.471224
9   0.138457      16.01858     0.000000       103.471224
10  1.272995      16.01858     0.693323       103.471224
11  0.188450      16.01858     0.503958       103.471224
12 -2.334245      16.01858    -0.167953       103.471224
13 -0.475823      16.01858     0.124608       103.471224
14  0.489548      16.01858     0.512034       103.471224

Sincerely,

To be honest I think it is perfectly fine. – German Demidov Feb 14 '22 at 10:38 — German Demidov, Feb 14 '22 at 10:38

score 11 · Accepted Answer · answered Feb 14 '22 at 10:58

As German Demidov notes, this is perfectly fine. The Lasso will shrink some of your coefficients to zero, but it does not have the property of shrinking all coefficients compared to the OLS estimate. Rather, it may increase some coefficients to "compensate" for the ones it has shrunk. There is nothing to worry about. (A very good question, though.)

Sextus Empiricus · Answer 2 · 2022-02-14T11:54:22.353

Lasso coefficients can shrink again while you get closer to the OLS solution.

See for instance: Why under joint least squares direction is it possible for some coefficients to decrease in LARS regression?

Here is an image of the relationship between the coefficients and the error. The lasso balances the error (depicted by the green surface) and the size of the coefficients (the red surface). For a given amount of regularisation it might be that some some parameters 'overshoot' and are larger than the actual OLS. By having these parameters larger you will have other parameters lower.

This situation happens when one parameter can take the role of several others. In that case, initially this parameter will be able to model the outcome very well even with a small coefficient (and one that is above the true model coefficient), but if you allow the total of coefficients to be larger then the others may catch up.

A clear illustration of this principle is in this question where a coefficient that should be zero is initially possitive. This happens because the parameter models the outcome better than the true model when the penalty is high: Is Ridge more robust than Lasso on feature selection?

Thank you, this is very helpful! – El Mehdi OUAFIQ Feb 14 '22 at 15:01 — El Mehdi OUAFIQ, Feb 14 '22 at 15:01

Lasso coefficient for some features is higher than Linear Regression Coefficient

2 Answers2