4

Is it is useful to use dummy variables even though it implies a big increase in the number of parameters?

In my case, I have a training set of about 13000 observations, and I am using 6 parameters. One of these is composed of 150 different elements; therefore whether I transform it in a dummy variable I'll have, in total, more than 150 parameters.

Is this transformation sensible?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Simone
  • 199
  • 8
  • 2
    If you can calculate information criteria for your models, you should look at them. If their values decrease, the transformation was worth it. – Richard Hardy May 16 '17 at 08:22
  • 1
    A possibility is using regularization, what you describes is the use case for "fused lasso", see https://stats.stackexchange.com/questions/227125/preprocess-categorical-variables-with-many-values/277302#277302 and links therein – kjetil b halvorsen May 16 '17 at 10:03

1 Answers1

4

Whether to use dummy variables or not is a choice often forced by the library implementation. Most of the model fitting/ML libraries need numerically represented data.

In such a case, you can try any of the following strategies, depending on size of data.

  1. If distribution of those 150 unique values is not uniform (that is not all of them are equally likely), you can only create dummy variables for more frequently occurring values (say, top 10). Replace all rarely occurring values with catch-all "other" bucket and create a single dummy variable for "other". Real world categorical features often have highly skewed distributions.
  2. (Variant of 1) replace the column value of the categorical variable by its frequency of occurrence in training set. Though note that you'll have to store this mapping from value to its frequency in the training data for re-use during prediction phase.
  3. Create as many dummy variables, but aggressively filter them using feature selection methods. This ensures that all variables are not used in the modeling phase. You can see example of this approach used in a pipeline fashion here in scikit-learn documentation.

Hope this helps.

hssay
  • 564
  • 4
  • 9