Does collinearity of one-hot encoded features matter for SVM and LogReg?

Question

Sometimes I encode categorical features as binary values - one feature per possible category value indicating whether that feature name matches the original category value (i.e. one-of-K scheme).

Now these values are linearly dependent, since obviously their total sum is 1.

Does this linear dependence matter for linear SVM, kernel SVM, logistic regression, etc.? Where does it matter so that I need to remove one of the features? Does it cause problems for normal linear regression? For which methods does it not make a difference?

score 1 · Answer 1 · answered Aug 23 '16 at 14:57

1

Based on my understanding, collinearity will impact the estimation of the weights. It leads to multiple solutions. So if your goal is to see the weights of the features and calculate the significance, you probably have to remove one dummy value and use only K-1 values. In this case, the intercept is the weight of the dummy values that you removed. Or you can use K values without intercept.

But if your goal is to have a prediction model with high prediction performance (i.e., what you really care is the outcome), it does not matter which encoding schemes you use. If using K values, you can either disable the intercept term or add a regularization term to eliminate the impact of the collinearity.

answered Aug 23 '16 at 14:57

Munichong

1,645
3
15
26

Hi, is there some example, how I can disable the intercept term? – Mutatos Dec 30 '19 at 16:09
@Mutatos What package are you using? Sklearn? – Munichong Jan 14 '20 at 21:47
yes I am using Sklearn. – Mutatos Feb 24 '20 at 07:39
@Mutatos You can set fit_intercept to FALSE, as https://stackoverflow.com/questions/46779605/in-the-linearregression-method-in-sklearn-what-exactly-is-the-fit-intercept-par – Munichong Feb 24 '20 at 17:56

score 0 · Answer 2 · answered Aug 23 '16 at 14:43

The linear dependence won't matter for any method that somehow excludes the intercept column from the design matrix. E.g. in SAS and R in linear regression you can request that the intercept is not included. If there is linear dependence, then what happens depends on the implementation. I believe in one of SAS regression procedures (probably PROC REG) they automatically drop one or more predictors to ensure linear independence.

Does collinearity of one-hot encoded features matter for SVM and LogReg?

2 Answers2

Linked