2

I'm trying to compare the performance of two classification algorithms. Thanks to CrossValidated, I already know that the best way to do this is via McNemar's test. I performed this, and sure enough, the models are significantly different.

My question is, does that mean the sensitivity, pecificity and accuracy of the models are all significantly different? Am I supposed to execute separate Mcnemar's tests for each measure?

I followed slide 22 of http://web.engr.oregonstate.edu/~tgd/classes/534/slides/part13.pdf to calculate my McNemar's test. Apparently, this compares error rates.

su kas
  • 23
  • 3

1 Answers1

3

Slide 22 is a bit confusing. The error rates on that slide are usually better described as positive and negative test results. Before talking about false positives (FP), true negatives (TN), etc. let us consider what the test actually is. The McNemar test is a test for difference of proportions. The Wikipedia entry for the McNemar test relates that for 

                 | Test 2 positive | Test 2 negative | Row total
_________________|_________________|_________________|__________
 Test 1 positive |       a         |        b        |  a + b
                 |                 |                 |
 Test 1 negative |       c         |        d        |  c + d
_________________|_________________|_________________|__________ 
 Column total    |      a + c      |       b + d     |    n

The null hypothesis, $H_0$, of marginal homogeneity states that the two marginal probabilities for each outcome are the same, i.e., $p_a+ p_b=p_a+ p_c$ and $p_c+ p_d=p_b+ p_d$. Thus the null and alternative hypotheses are

\begin{align} H_0 & :~p_b=p_c \\ H_1 & :~p_b \ne p_c \end{align}

Thus, the alternative hypothesis means that $p_b$ ≠ $p_c$; the marginal proportions are significantly different from each other. To adapt this for TP, TN, etc. see similar answer at https://stats.stackexchange.com/a/241844/99274. This requires that one test is a "truth value" which is not necessarily native to the McNemar test, which test actually only requires that we have two tests, not that one of them is a "standard."

Carl
  • 11,532
  • 7
  • 45
  • 102