7

When feeding a categorical variable into glmnet do I code n or n-1 dummy variables?

For instance if using days of the week as an independent variable would I use 6 dummies or 7?

If the answer is 6, how do I interpret coefficients, etc for dropped category?

EDIT: Here's some example code:

library(glmnet)
library(caret)

df1 <- data.frame(id = 1:210, var1 = rep(c('Mon','Tues','Wed','Thurs','Fri','Sat','Sun'),30))
df1$targetVar <- runif(210)
    df1$mktVol <- round(runif(210)*1000000,0)
df1$mktVol <- ifelse(df1$var1 %in% c('Sat','Sun'), 0, df1$mktVol)

df1
vtu <- c('mktVol','var1')
dv1 <- dummyVars( ~.,data = df1[,vtu])
df2 <- data.frame(predict(dv1,df1))

glmnet1 <- cv.glmnet(df2$targetVar, data.matrix(df2[,-c('targetVar')]), nfolds = 5)

glmnet1 <- cv.glmnet( data.matrix(df2[,-1]), df2[,"mktVol"] ,
      family="gaussian", alpha=.95, nfolds=5, standardize = FALSE,
      type.measure="mse")

Coefficients1 <- coef(glmnet1, s = glmnet1$lambda.min)
Active.Index <- which(Coefficients != 0)
Active.Coefficients <- Coefficients[Active.Index]
names(X1)[varsToUse[Active.Index]]

##############################

df1 <- data.frame(id = 1:210, var1 = rep(c('Mon','Tues','Wed','Thurs','Fri','Sat','Sun'),30))
df1$targetVar <- runif(210)
    df1$mktVol <- round(runif(210)*1000000,0)
df1$mktVol <- ifelse(df1$var1 %in% c('Sat','Sun'), 0, df1$mktVol)

df1
vtu <- c('mktVol','var1')
#dv1 <- dummyVars( ~.,data = df1[,vtu])
#df2 <- data.frame(predict(dv1,df1))
dv1 <- model.matrix(~.,data = df1[,vtu])

#glmnet1 <- cv.glmnet(df2$targetVar, data.matrix(df2[,-c('targetVar')]), nfolds = 5)

glmnet1 <- cv.glmnet( data.matrix(df2[,-1]), df2[,"mktVol"] ,
      family="gaussian", alpha=.95, nfolds=5, standardize = FALSE,
      type.measure="mse")

Coefficients2 <- coef(glmnet1, s = glmnet1$lambda.min)

##############################

df1 <- data.frame(id = 1:210, var1 = rep(c('Mon','Tues','Wed','Thurs','Fri','Sat','Sun'),30))
df1$targetVar <- runif(210)
    df1$mktVol <- round(runif(210)*1000000,0)
df1$mktVol <- ifelse(df1$var1 %in% c('Sat','Sun'), 0, df1$mktVol)

df1
vtu <- c('mktVol','var1')
#dv1 <- dummyVars( ~.,data = df1[,vtu])
#df2 <- data.frame(predict(dv1,df1))
dv1 <- model.matrix(~ 0+ .,data = df1[,vtu])

#glmnet1 <- cv.glmnet(df2$targetVar, data.matrix(df2[,-c('targetVar')]), nfolds = 5)

glmnet1 <- cv.glmnet( data.matrix(df2[,-1]), df2[,"mktVol"] ,
      family="gaussian", alpha=.95, nfolds=5, standardize = FALSE,
      type.measure="mse")

Coefficients3 <- coef(glmnet1, s = glmnet1$lambda.min)
Coefficients1 
Coefficients2 
Coefficients3
smci
  • 1,456
  • 1
  • 13
  • 20
screechOwl
  • 1,677
  • 3
  • 21
  • 32

2 Answers2

1

"In the extreme case of k identical predictors, they each get identical coefficients with 1=kth the size that any single one would get if t alone. From a Bayesian point of view, the ridge penalty is ideal if there are many predictors, and all have non-zero coefficients (drawn from a Gaussian distribution).

Lasso, on the other hand, is somewhat indifferent to very correlated predictors, and will tend to pick one and ignore the rest. In the extreme case above, the lasso problem breaks down. The Lasso penalty corresponds to a Laplace prior, which expects many coefficients to be close to zero, and a small subset to be larger and nonzero."

Page 4, Regularization Paths for Generalized Linear Models via Coordinate Descent, Jerome Friedman, Trevor Hastie, Rob Tibshirani.

So, you can leave all of them in--since Ridge guarantees that the (X'X) matrix is invertible---but I wouldn't recommend it.

Francisco Arceo
  • 582
  • 3
  • 7
  • Why wouldn't you recommend leaving all predictors in the model? Ridging assumes all variables are of similar size, so why choose 1 out of 7 coefficients which should be "roughly the same" to be 0? (in fact you can use the same argument in the paper you referenced - the likelihood is indifferent to such a choice, but the penalty is not) – probabilityislogic Jun 25 '13 at 22:04
  • 1
    I recommend leaving one out for interpretation, unless the constant is excluded. Consider the case where you have a single binary indicator representing two groups (one treated, one not treated). In OLS, the parameter estimate yields the interpretation that the effect of the binary on y is the point estimate when the indicator is set to unity. So, when the binary is set to zero the prediction would only be the intercept. To have a variable representing both groups, the model would have to remove the intercept. If OP does remove the intercept, then he/she can put all of them in. – Francisco Arceo Jun 27 '13 at 14:10
  • I disagree that omitting a level helps interpretability. We can interpret the model by comparing $E(Y | X=x)$ vs $E(Y | X = x^\prime)$, where $E(Y|X)$ is the fitted regression equation. If $X$ is continuous, then we can, for example, take a partial derivative. The challenge, as I see it, is that the Lasso is biased, so that $E(Y | X=x) \neq X \hat{\beta}_\text{lasso}$. – Andrew M Feb 07 '17 at 17:10
  • @AndrewM It's been a few years since I posted this and I somewhat agree with you....but in reality, it doesn't matter. Omitting one makes the coefficients relative to the base category and including it makes it a partial effect without a base, it just shifts things around. – Francisco Arceo Feb 07 '17 at 17:30
  • "Including it makes it a partial effect without a base, it just shifts things around." This is true insofar the fitted values in OLS are invariant to a change of basis. The $\ell_1$ penalty on the lasso means that the estimator has no such invariance. All of this complicated by the fact that most algorithms standardize the design matrix to have unit variance before fitting. – Andrew M Feb 07 '17 at 22:48
  • If you use `n`, then you know that one of the contrast levels will not get used; the smaller `n` is the more waste, and in the binary (n=2) case you're generating twice as many variables as you need. `n-1` dummies is best to avoid wasting unnecessary memory on the model-matrix, and CPU time in 'selecting' the unnecessary levels out. Also improves legibility of the output - no point in seeing pages of unnecessary excluded variables. – smci Feb 10 '17 at 09:24
0

The question seems have little thing to do with glmnet, but factor encoding and interpretation on coefficient in general.

I would suggest to look at logistic regression that uses categorical variables as a start. R Library: Contrast Coding Systems for categorical variables a great resource to learn different types of encoding and interpretation. The key is "comparing to base level". For example, if we want to encode days of weeks. We can use Sunday as a base level, and all other days will compare against it.

Haitao Du
  • 32,885
  • 17
  • 118
  • 213