3

I would like to compare a risk prediction test (model) with a single predictor (continuous variable):

Let's say I have a risk prediction testA (e.g. a logistic regression model) that gives a risk probability for cancer (yes/no) between 0-1 when it is applied on a patient with the characteristics B, C, D, E (test A uses all of these characteristics B-E).

Now there has been a new characteristic X found (a continuous variable such as systolic blood pressure) and we think that it could be a good predictor of the cancer state (yes/no) of a patient.

If I want to compare the testA and the characteristic X now to decide which is the better discriminator for the cancer state (yes/no), how could I do that?

I read through

  1. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures. Epidemiology. 2010;21(1):128–138.

and thought about performing a ROC analysis and compare the AUC of the testA with the AUC of the characteristic X using a DeLong test.

However, since characteristic X ist not really model, I am not sure if my approach makes any sense.

Thank you for your help.

captcoma
  • 155
  • 13

2 Answers2

2

The $c$-index (concordance probability; AUROC; see also Somers' $D_{xy}$ rank correlation coefficient) is not sensitive enough for comparing two models. See http://fharrell.com/post/addvalue for the most sensitive measures.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
1

In theory ROC analysis doesn't care if your test is a model or not. In fact, ROC analysis doesn't even see the model, only the model predictions. Therefore whether you have a model or not shouldn't matter.

In practice you should take into consideration the shape of the ROC curve. Especially if either testA or characteristicX has very few or even a single threshold, the ROC curve will be under-estimated, and the comparison might be misleading.

I have done similar analysis myself in a 2013 paper of mine, if you want to check it out, where I compare several measured biomarkers (characteristics XYZ) with several types of models that includes them (test ABC), in a cross-validation setting.

Calimo
  • 2,829
  • 17
  • 26
  • Dear Xavier. Thank you very much for sharing your paper. Do I understand it correctly that the test (i.e. the panel) would be rated as positive only if all parameters reached the treshold? An implementation of your method in R (probably with a multivariate discovery approach) would be of great interest for medical research. Have you done any net benefit analysis to compare your method to an approach using log. regression? Ps: I am a huge fan of your pROC package. – captcoma Jul 16 '19 at 10:50
  • 1
    @captcoma no the final positivity is a parameter as well. See https://github.com/xrobin/PanelomiX – Calimo Jul 16 '19 at 17:05