1

I'm fitting a logistic regression (binary) using Python's statsmodels, and here's a snippet of summary from the model:

enter image description here

I have noticed that the large coefficients only occurred on two variables and it seems like it's due to not converging (though I set max to 500).

Warning: Maximum number of iterations has been exceeded. Current function value: 0.094121 Iterations: 500

I'm wondering what's the reason behind it and what are some possible ways of fixing this.

Just as extra information, I did:

  1. drop one of the levels from binning
  2. add a constant to the design matrix

Any help is appreciated! And please let me know what other information might be useful to identify the problem.

TYZ
  • 755
  • 4
  • 17
  • 3
    Did you look into separation? See https://stats.stackexchange.com/questions/180447/is-this-really-perfect-separation-in-logistic-regression-or-is-something-else-g and search this site. – kjetil b halvorsen May 15 '19 at 10:52

3 Answers3

1

This is not a good application for binning. To have an adequate fit for an underlying smooth relationship that is steep in places, binning requires a large number of bins resulting in a losing battle in the bias-variance war because of high variance. For continuous variables use fewer parameters and still get a better fit using things like restricted cubic splines and other cubic spline bases.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
0

Your factors may are nested within each other, check the model variance inflation factor (VIF). This correlation leads to high errors, since you have many factors there is a considerable chance of nesting.

It is equivalent of collinearity between continuous variables, but in your case correlation between factors.

Good luck!

Best, VMaia

maiava
  • 109
  • 5
  • I checked VIF for variables before binning and all variables have VIF of ~1, should I do that for all the binary bins after splitting as well? – TYZ Jan 22 '19 at 20:56
  • I think it is worth to try, but maybe nesting is not the problem. – maiava Jan 22 '19 at 21:28
  • Interestingly, I checked correlation between all the bins from the two variables with huge coefficients but there's no obvious high correlations. I'm wondering if there are any other possible causes. – TYZ Jan 22 '19 at 21:50
  • 1
    Did you run a VIF for the whole model, containing all variables? – maiava Jan 22 '19 at 21:52
0

Apart from endorsing Frank's comment that binning is seldom a good idea and certainly not to this extent the most likely explanation, as mentioned by Kjetil in a comment is that the model suffers from separation. For some combination of levels the outcome variable is all zero or all one. A typical symptom of this is the coefficients trying got go off to infinity until the software calls a halt and very large standard errors.

Assuming it is separation then advice is available How to deal with perfect separation in logistic regression? where in my opinion the answer by scortchi is the best advice.

mdewey
  • 16,541
  • 22
  • 30
  • 57