Is McNemar's Test Appropriate Here?

Question

I am applying two diagnostic tests with dichotomous outcomes (condition present yes/no) to a sample of patient data. Both take into account a range of vital parameters and produce a test result, which is then compared to the gold standard, such that the usual statistics (sensitivity, specificity, etc.) can be calculated. One test seems to perform somewhat better than the other in terms of the area under the curve.

Is there any way to test if this modest improvement in diagnostic quality is statistically significant? I have come across McNemar's test for paired nominal data, which is applicable to the 2x2 contingency table, but I am wondering if it is also appropriate and meaningful in this case. Most explanations and examples seem to pertain to pre- and post treatment comparisons, which is not the case here.

Answer 1

You could use McNemar's test. You do (potentially) have a 2x2 table. For each patient, you have the output from the two diagnostic tests, and the knowledge of the 'correct answer' from the gold standard. Thus, for each patient, you have test1 correct/incorrect, and test2 correct/incorrect. From those, you can construct a 2x2 contingency table:

           test2
test2       correct  incorrect
  correct        35         14
  incorrect      23          7

Note that McNemar's test is really just a binomial test of the equiprobability of the off-diagonal events (see here and here).

binom.test(14, 37)
#   Exact binomial test
#  
# data:  14 and 37
# number of successes = 14, number of trials = 37, p-value = 0.1877
# alternative hypothesis: true probability of success is not equal to 0.5
# 95 percent confidence interval:
#  0.2245761 0.5524320
# sample estimates:
# probability of success 
#              0.3783784 

That said, if your tests yield AUC's, they presumably output more information than just condition present yes/no. Most of that information is being thrown away. McNemar's test only uses whether the classification was correct.