0

I am using the glmnet package in R. When I set the alpha value = 0, I would expect that no variables are selected. When I look at the coefficients some of them are set to zero. What could be the explanation for this behaviour?

Pat
  • 11
  • 1

2 Answers2

0

I found an explanation why I get some coefficients set to zero in my empirical data set. The reason seems to be, that some of my variables have the same information content over rows. I am working with SNP data. So in the case where a variable is set to zero they are e.g. homozygous for all indviduals. If I remove all this "monomorphic" positions I end up with the same amount of variables, which remain after fitting the ridge model and removing coefficients which were set to zero. So my assumption now is, as some predictors have no variance, the respective regression coefficients are not assessable and thus set to zero.

Pat
  • 11
  • 1
-2

You should have a look at the glmnet's vignette. Having alpha set to zero does not mean that no variables are selected. It means that you only use the $\ell_2$-penalization. Let me know if you need more explanation ?

In fact, $\ell_1$ penalization tends to give you sparse estimates. If you have a look on the picture, you can see that it usually goes on the edge (i.e. forcing some coefficients to be zero). The $\ell_2$ penalization is supposed to "regroup variables" (harder to explain it with this picture).

comparison $\ell_1$ and $\ell_2$ estimation

Maybe I don't understand your question, but why did you think it would not select variables ?

Julien D.
  • 114
  • 3
  • Some explanation would be appreciated. I thought the l2-penalty can not set variables to zero. – Pat May 28 '14 at 09:25
  • I updated my answer. Yes, l2-penalty can set a variable to zero. Actually you can see it as a constraint, i.e., it tries to find a solution along the ellipsoids which minimizes the equation AND is in the black region. – Julien D. May 28 '14 at 09:38
  • I have no strong statistical background, my assumption was based on literatur. [Here](http://stats.stackexchange.com/questions/74542/why-does-the-lasso-provide-variable-selection) is an explanation why ridge can not select variables, but probably I do not get the point. – Pat May 28 '14 at 09:52
  • Oh, I think I know what you mean. $\ell_1$ penalty tends to give you sparse estimates. It means that some of the coefficients are going to be zero, therefore you are going to pick only the coefficients with a value different from zero. If you only use $\ell_2$ penalty, you will not have this "behavior". This does not mean you cannot do "variable selection". The only different thing you have to do is to choose an $\epsilon$ and select the variables that are bigger than this $\epsilon$. You know what I mean ? – Julien D. May 28 '14 at 11:41
  • Yes I think I know what you mean. But how can it be that some of my coefficients are zero using the l2 penelty. They just should be shrunken towards zero. But thank you so fare for your effort! – Pat May 28 '14 at 11:54
  • $\ell_2$ penalty does not mean your coefficients cannot be zero. You can see the $\ell_2$ penalty as a constraint on your coefficient. You can have more details on why this is equivalent [here](http://math.stackexchange.com/questions/335306/why-are-additional-constraint-and-penalty-term-equivalent-in-ridge-regression) – Julien D. May 28 '14 at 14:27
  • 1
    With most real data sets it would highly unusual for any $l_2$ coefficients to be *exactly* zero (within numerical roundoff error). – whuber May 28 '14 at 15:43