1

I have a dataset with 56 columns, 4 numeric and the target variable which is also numeric. I am trying to eliminate some of the categorical variables from my model and wanted to get some understanding of the methodologies to do that.

I am currently working in R, which is somewhat new to me.

Any help would be great!

Edit: I would like to eliminate some variables to increase my models performance. While the model is not performing poorly, I believe the model can perform better than it is now. Some of the variables I can take a look at and know right off the bat that this is something that I can remove. But, I would like to show proof that this is not an informative predictors.

Edit2: Here is information about my dataset.

Out of the columns that are categorical, there are roughly 5 variables that have levels greater than 2. My goal is essentially to remove variables that are the least informative to my model. My model is a regression model to predict the sale price of something. Using accuracy as a metric, I want to have the most accurate model and remove predictors (both numeric and categorical) that are not conducive to the predicting accuracy of my model.

In removing categorical variables, I would be removing the entire variable, not particular levels.

rmahesh
  • 115
  • 6

2 Answers2

1

The group LASSO was designed specifically to allow groups of predictors to be selected or omitted together, so that is what you would want to use in your case to keep all levels of a categorical predictor together. There are several implementations in R. This, unlike stepwise selection, provides a principled way to decrease the number of predictors; you can choose the LASSO penalty (and thus the number of maintained predictors) by minimizing cross-validation error to get a good tradeoff between parsimony of predictors and performance while providing reasonable application to new data samples. It seems that you do not want to omit any of your numeric predictors, so make sure that the implementation you use allows for forced inclusion of chosen predictors.

There is one unavoidable difficulty with LASSO or ridge regression with categorical predictors: whether/how to normalize those predictors before doing the analysis, which applies a single penalty to all predictors. Normalization is clearly necessary with numeric predictor values: you don't want the penalization to depend on the scale of measurement (e.g., millimeters versus miles). So programs may default to scaling all predictors to 0 mean and unit standard deviation for calculations, then back-transforming to the original scales for reporting coefficient values.

But is that what you want with a binary categorical predictor, where the mean and SD necessarily depend on the class prevalence? Does such normalization even make sense with a multi-level categorical variable? Depending on the implementation, your results with a multi-level categorical predictor might even differ depending on your choice of reference level. So spend some time thinking about what you want to accomplish, the details of how your chosen LASSO implementation works, and whether or not to specify normalization for the categorical predictors.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • Thank you for the elaborate explanation, that makes sense to me. I would like to follow up by saying that I have no problem removing numeric predictors if they are in fact uninformative in the model. If both numeric and categorical variables can be used in this, I think this is the solution I was looking for. Any idea on where to look to implement this in R? – rmahesh Sep 08 '18 at 21:20
  • A quick web search for `group lasso r` shows 3 packages with this functionality: `grpreg`, `grplasso`, and `gglasso`. I don't have much experience with such packages, so read the documentation carefully. – EdM Sep 09 '18 at 00:56
0

Here are some ways to do this:

1) Lasso: Perform regularized regression with the lasso penalty. This will force most of your predictors to be 0.

2) Forward selection: At any point, greedily add the feature that adds most predictive value.

3) Backward selection: Start with all predictors and greedily remove the feature that if removed affects the prediction quality the least.

elexhobby
  • 775
  • 1
  • 5
  • 16
  • Note that forward and backward selection are typically [very poor choices](https://stats.stackexchange.com/q/20836/28500) in constructing a model. They tend to overfit, producing models that do not perform well on new data samples. The regularization provided by LASSO tends to avoid these difficulties, but there might be complications if some of the categorical variables have more than 2 levels. – EdM Sep 08 '18 at 16:55
  • @EdM Thanks for the response. I am interested in using the LASSO method. My data set does have levels greater than 2 although the majority of the categorical variables only have 2 levels. – rmahesh Sep 08 '18 at 17:34
  • Thank you for the post. I have used backward selection in the past and had some success but in the case of this data set, that does not seem to be working. Can the lasso method be used for categorical variables with levels greater than 2? – rmahesh Sep 08 '18 at 17:36