1

I'm performing a Logistic regression for a binary classification task.

As a preprocessing technique I use a transformation with WOE and Information value(IV), but I found something counterintuitive since when I check the regression's coefficients there are some that have the value of zero though those variables have a "high" IV

From the table below I see that for variable CANT_ACT_ECONOMICAS has a coefficient of 0 but is has the highest IV

Model coefficients:
***** Fature Impportance *****
                        importance
COMPANY_DOMAIN_CLASS      0.027879
GENDER                    0.000000
CANT_ACT_ECONOMICAS       0.000000
ACT_ECONOMIC_GROUP       -0.065918
PRODUCTIVE_ZONE          -0.108427
DOMAIN                   -0.284159
COMPANY_AGE              -0.321768
WEEKDAY                  -0.474630
ACT_ECONOMIC_RISK        -0.595959
USE_OF_PROCEEDS_INTENT   -0.805546
AGE                      -0.895112
USE_OF_PROCEEDS_RISK     -0.898074
START_HOUR               -1.030490

Information Value:

AVG_IV
CANT_ACT_ECONOMICAS 0.058573
COMPANY_AGE 0.034645
PRODUCTIVE_ZONE 0.032958
ACT_ECONOMIC_RISK   0.030869
COMPANY_DOMAIN_CLASS    0.019605
AGE 0.004793
USE_OF_PROCEEDS_RISK    0.004689
USE_OF_PROCEEDS_INTENT  0.003318
START_HOUR  0.001870
WEEKDAY 0.001217
DOMAIN  0.001032
GENDER  0.000568
ACT_ECONOMIC_GROUP  NaN

My question is:

If a variable have a high information value, does it necessarily have to have high importance on logistic regression model (high coefficient)?

EdM
  • 57,766
  • 7
  • 66
  • 187

1 Answers1

2

No, because variables can be correlated. For an exemple you can take a degenerate problem where you have the same variable twice. Their IV will be the same by construction, however their coefficient could vary wildly as long as their sum is constant and equal to the coefficient you would get with one variable. So even with high IV you can get pretty much anything in term of coefficients.

Practically, the regression may not work with a duplicated variable, but in general you will get similar problems with highly correlated variables : high IV, but you don't know what variable the model will take for the main effect. This can result in one variable having a big weight and the other playing a role of correction.

One popular exemple is about house pricing : both squared meters and number of windows would have a high IV for predicting the price. Used individually they would result in a high coefficient. However if you use both, the model can take squared meters as the main variable, and use number of windows as a negative correction. (for a given house size, having too many windows is not good). As you can see, a high IV, doesn't mean a high (or even a positive) coefficient. This is mostly due to IV being univariate measures, while coefficients usually depends on correlation.

In your specific case, the coefficient is exactly zero. I supsect some form of penalisation, which has removed some variables, probably because they are correlated to other variables. In my previous exemple this would amount to removing the number of windows variable from the model as the squared meter variable is enough to take into account the size effect on the price.

lcrmorin
  • 1,380
  • 16
  • 31