4

I'm doing variable selection using the Lasso.

To explain my response variable I have several predictors, both categorical and numerical, but I have problems to explain the process that underlies when Lasso selects only a category from a variable with several categories.

For example, one of my predictors is a categorial variable with four levels, and Lasso just selects one of them. So, Lasso is working with the whole variable (the four categories) but some may "enter" and some not. How can I explain this? It's something related with Analysis of Covariance?

I hope my question makes sense and I would appreciate a not very mathematical answer.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Alien
  • 41
  • 3
  • 2
    It would be typical to treat categorical variables as factors, all of whose levels are in or not-in at the same time. I believe functions in the R packages `grpreg`, `grplasso` and `glmnet` can all do this. – Glen_b Sep 10 '14 at 00:50

2 Answers2

2

You are doing lasso the wrong way. A categorical variable with four levels is represented by three dummy variables. Those three dummys together represents one variable, and should be treated as such. To do that with the lasso, use the group lasso, as discussed in Why use group lasso instead of lasso?.

There are many posts about the group lasso in here.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
1

Say, you have categories A, B, C, D. It turns out that as you go from A to non-A, the response varies substantially, but as you switch levels inside {B, C, D}, the response doesn't change much. Therefore, to explain the variance of response, it is helpful to know whether the predictor is at A or not, but having more concrete information about the levels of predictor is not that helpful.

James
  • 2,600
  • 1
  • 14
  • 26
  • 1
    That may be, but it doesnt make sense logically to include only some of the indicators and not others. Choice of coding might also influence results then! The only logical solution is to use software that treats factors correctly. – kjetil b halvorsen Sep 11 '14 at 08:29