2

I am conducting a binary regression using different sorts of parametric links (logistic, Pregibon, Aranda-Ordaz, ... see) and I would like to compare their predictive and classification perfomance in a particular data set using the c-index. However, I wonder if this is correct given that this measure does not depend on the additional shape parameters in the link function, only on the estimated regression parameters.

Barlett
  • 23
  • 3

1 Answers1

2

The $c$-index (concordance probability or ROC area; simple function of Somers' $D_{xy}$ rank correlation) only uses the ranks of predictions so is not sensitive enough for model comparisons, especially of the type you outline. I suggest using proper scoring rules such as the Brier (quadratic) score and deviance-based measures such as generalized $R^2$. Even those are not ideal. Bootstrap bias-corrected smooth (loess) calibration curves may be a better approach. An indirect approach would be to find the link that minimized the likelihood ratio $\chi^2$ due to all two-way interaction terms, i.e., the link that maximizes additivity.

Sometimes I see analysts worrying about getting the link function right when the more important assumption may be the linearity assumption for the predictors. It is often a mistake to assume linearity. Expanding predictors using regression splines is an important part of the fitting process no matter what the link.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
  • Many thanks for your answer (I am also glad to get an answer from such an expert in this area). Just to clarify, do these measures depend on the choice of the link function? Could you also please provide a reference? What would you recommend for assessing the predictive and classification performance of the different links? – Barlett Oct 23 '14 at 13:19
  • 1
    Classification performance will not be relevant. For prediction performance, deviance (log-likelihood)-based measures are customized to the link being used but are not automatically perfectly comparable across links. The Brier score is model-agnostic so may be preferred. But more goal-directed model accuracy assessment may be warranted (calibration, functional form, freedom from need for interactions). – Frank Harrell Oct 23 '14 at 13:32