2

My question is about any possible relationship between the significance of a comparison test (Mann-Whitney U) and the ROC curve.

If the comparison test is strongly significant, should we expect a better ROC curve, and consequently better NPV, PPV etc.?

In details: I'm comparing the blood results between 2 groups for a predictive model. My pilot data, from 50 samples, showed statistical significance (Mann-Whitney U test) and the ROC curve showed good specificity, sensitivity, PPV and NPV.

Afterwards, I ran the same tests with 200 samples. The Mann-Whitney U test showed an even better statistical significance (far better). However, from the ROC curve only the specificity was of value, but not as good as the one from the pilot results. The sensitivity, PPV and NPV have worsened tragically!

I was expecting with bigger numbers and better statistical significance to have a better ROC.

Is there an explanation for that?

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
Panos
  • 21
  • 1

2 Answers2

2

I would accuse overfitting. With N=50, there is a risk of overfitting if your model has too many degree of freedom. It's easier to fit 50 subjects than 500 subjects.

Concerning your feeling of contradiction with the better p-value of your Mann-Whitney U, it's perfectly normal to have a better level of evidence for the difference between groups when N increases. Let's say that in reality, there is constant difference of 5% of blood pressure between your two groups, if you are not unlucky, increasing N should increase your p-value. That's why a better p-value does not mean that the difference between groups or that it's easier to discriminate subjects.

brumar
  • 2,092
  • 11
  • 14
2

Because you have a continuous test, sensitivity, specificity, NPV, PPV just get in the way. So does the ROC curve. It provides no insights here, although it is traditional. Just stick to the idea of separation, which the concordance probability (c-index), Wilcoxon-Mann-Whitney, and Somers $D_{xy}$ rank correlation equivalently quantify. The c-index (which happens to be the ROC area although in some ways this detracts from the c-index) like $D_{xy}$ is a unitless measure of separation. You are right that significance will generally be greater for larger $N$. But I would do the following:

  1. Use a nonparametric smoother such as loess to estimate the relationship between X and the probability of a positive diagnosis
  2. Describe the degree of relationship between the X and the diagnostic outcome using c or $D_{xy}$
  3. De-emphasize P-values

For more details see the link to Biostatistics for Biomedical Research at the top of http://biostat.mc.vanderbilt.edu/ClinStat - read chapter 18.

If you really want to test $H_{0}$: ROC area = 0.5 (absence of diagnostic information), the Wilcoxon-Mann-Whitney 2-sample rank-sum test is definitely the test to use.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322