3

I am trying to apply glmnet's lasso to a set of features in which there are multiple categorical variables with multiple levels. My intention is to let lasso reduce some of the coefficients of the features down to 0, so that they can be thrown out. Some of my categorical predictors have as many as 50 levels.

The result is that the glmnet is throwing out only some of the levels of the categorical predictors, but keeping some others. My understanding is that this is incorrect - the dummified levels are all a part of the same predictor - so one needs to throw out the entire predictor (with all it's levels) or keep all of them. The data set is large, maybe 600,000 rows. I am trying to predict a binary outcome between two classes.

Here is an example of my code:

library(glmnet)
x <- model.matrix(project.status~., data = data_train)
y <- data_train$project.status

lasso.net <- cv.glmnet(x, y, alpha = 1, family = "binomial", nfolds = 5, 
type.measure = "auc")

and the output:

>coef(lasso.net)

 86 x 1 sparse Matrix of class "dgCMatrix"
                                                       1
(Intercept)                                    -9.099069e-02
(Intercept)                                     .           
project.gradeGrades 6-8                         .           
project.gradeGrades 9-12                        6.891742e-02
project.gradeGrades PreK-2                      3.107257e-03
project.resourceBooks                           .           
project.resourceClassroom Basics                8.106422e-01
project.resourceComputers & Tablets             5.849269e-01
project.resourceEducational Kits & Games        8.442034e-01
project.resourceFlexible Seating                5.031112e-01
project.resourceTrips                           .           
project.resourceVisitors                        .           
project.cost                                   -5.286631e-04
school.metro.typesuburban                       1.991501e-02
school.metro.typetown                          -8.060338e-02
school.metro.typeurban                          2.380249e-01
school.percent.lunch                            4.178175e-04
school.stateAlaska                              .           
school.stateArizona                            -2.095588e-01
school.stateArkansas                           -1.652419e-01
school.stateCalifornia                          1.209260e-03
school.stateColorado                            6.299693e-02
school.stateConnecticut                         1.186827e-01
school.stateDelaware                            1.829217e-01
school.stateDistrict of Columbia                4.099672e-01
school.stateFlorida                             .
school.stateGeorgia                            -2.292140e-01

I've not included the entire output (it's long) but hopefully this presents my issue. The school.state predictor is a categorical predictor with the 50 states. I essentially want to see if I can throw this predictor out, but instead of zeroing the entire predictor, it is only zeroing out some of the states, and keeping the others. Likewise with the project resource and project grade (it's essentially a charity project, I am trying to predict whether they met their funding goal or not).

Karolis Koncevičius
  • 4,282
  • 7
  • 30
  • 47
Marcel
  • 1,200
  • 6
  • 24
  • My understanding is that lasso doesn't work that way. Look at the penalty term--there is no concept of groups of coefficients. Maybe you want to look into the group lasso? – The Laconic Jun 15 '18 at 22:04
  • In linear regression each "dummy" variable is coded as a separate covariate. So it's not surprise that some of them are being thrown out while others are left in. In your example each state has a coefficient that will be estimated. So LASSO can "decide" to shrink some of them, but not others. – Karolis Koncevičius Jun 15 '18 at 22:24
  • Is there then a way to encode the categorical predictors so that this doesn't happen? Is that even the correct choice? – Marcel Jun 15 '18 at 22:33
  • As @TheLaconic pointed out - something called "group lasso" seems to designed for this. I.E. here: https://stats.stackexchange.com/q/214325/18417 – Karolis Koncevičius Jun 15 '18 at 22:45

1 Answers1

1

Answering my own question for posterity.

Indeed, it looks like "Group Lasso" is the correct procedure here. A number of packages are available in R, including grplasso, grpreg, and gglasso. Most seem to operate on producing a vector of consecutive numbers that denote which variables should be grouped together or not. It seems relatively intuitive to produce columns for your dummified categorical predictor, then group those dummies together and implement a group lasso, which should then zero out all of the dummies at once or keep them.

Marcel
  • 1,200
  • 6
  • 24