how to deal with categorical features (with distinct 10000+ values) other than conversion to one-hot encode and ordinal

Question

Machine Learning Problem : I have a doubt in one of my feature which has an categorical value 1. One way of dealing with it would be like converting those values into numbers means in ordinal form. But the problem with this would be my model could have learned the relation as 1<2<3 which is not true in terms of categorical values 2. Other way would be converting values into one hot encoding but it could increase the dementionality and can make my data very sparse So how to deal with that type of problem?

Some of people said to use model like xgboost which handles the categorical values but how they internally tackle that?

Tell us, please, why this is a prob!em: memory? Use sparse matrices. To many parameters to learn? Maybe some variant of lasso. Other?? And: most importantly: what do you want to do with your data?? — kjetil b halvorsen, Sep 07 '18 at 05:07
Yes memory is a problem and i also want to use it as its an important parameter for judgement — Ajay, Sep 07 '18 at 05:37
Then you can look into sparse matrix representation of your data. Tosay more we really need to know your goal. Is this a regression problem? — kjetil b halvorsen, Sep 07 '18 at 05:41

kjetil b halvorsen · Answer 1 · 2018-12-04T23:01:42.797

After you have given some extra information in comments this is approaching an answerable question. You should really edit that extra information into your question (many will not read comments). And, read How to ask a statistics question . Context is essential in statistics, so what do your two classes represent? How many examples? ...

A classification problem with two classes can often be better represented as a logistic regression. Then you get predicted probability for the two classes, and this can later be used to obtain a classification. Frank Harrell has written about that, Choosing between logistic and discriminant, Why isn't Logistic Regression called Logistic Classification?

Then the memory problem (but it could well be a fitting/inference problem too: to estimate 10000+ parameters you need a really big sample). Look at the post Principled way of collapsing categorical variables with many levels? and search this site for the tag many-categories

Software: glmnet implements lasso for logistic regression (among other things), and uses sparse matrices out of the box. Try it!

how to deal with categorical features (with distinct 10000+ values) other than conversion to one-hot encode and ordinal

1 Answers1