7

I have a step-wise derived binary logistic regression model. I have used the calibrate(, bw=200, bw=TRUE) function in the rms package in R to estimate its future calibration. The output is given below and it shows the bootstrap overfitting-corrected calibration curve estimate for the backward step-down logistic model. However, I am not sure how to interpret it.

enter image description here

I understand that calibration refers to whether the future predicted probabilities agree with the observed probabilities. Prediction models suffer that predictions for new subjects are too extreme (i.e, that the observed probability of the outcome is higher than predicted for low-risk subjects and lower than predicted for high-risk subjects). This is seen by tracing the dotted curve which is higher than the ideal (dashed) for low-risk group, and lower than the ideal for high-risk group.

Using the same reasoning, the bias-corrected curve seems to be worse, in the sense that it produces even more extreme probabilities. Is my interpretation correct?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Abbas
  • 338
  • 3
  • 11

1 Answers1

2

The curve labeled bias-corrected appears to be "over confident": its predictions for Predicted P(Class=1)<0.5 are too low and its predictions for Predicted P(Class=1)>0.5 are too high relative to Actual probability.

This is also the case for the curve labeled apparent, except at the extremes (roughly: x<=0.28 or x>=0.9) it appears to actually be $less$ confident.

I am unsure of the details of the bias correcting method in rms, but I don't think the result is necessarily "worse"; With the correction, the probability estimates are parallel to the ideal. In other words, although it's known that the model is slightly overconfident, we can say that the difference between its mean prediction for a population with P(class)=0.2 is half of its mean prediction for a population with P(class)=0.4, which was not the case before and probably what one would hope for.

sjw
  • 5,091
  • 1
  • 21
  • 45
  • 1
    I wouldn't emphasize the last paragraph, otherwise nice answer @user99889. The bootstrap overfitting-corrected calibration curve shows that there is overfitting and that the apparent fit is optimistically good. The bootstrap-corrected curves are almost always farther away from the 45 degree line than the apparent curves. Overall I'd say that moderate overfitting is present and the results are probably not acceptable for field use, if risk estimates are needed. – Frank Harrell Sep 02 '17 at 13:15