Comparing two classifier accuracy results for statistical significance with t-test

Question

I want to compare the accuracy of two classifiers for statistical significance. Both classifiers are run on the same data set. This leads me to believe I should be using a one sample t-test from what I have been reading.

For example:

Classifier 1: 51% accuracy
Classifier 2: 64% accuracy
Dataset size: 78,000

Is this the right test to be using? If so how do I calculate if the difference in accuracy between classifier is significant?

Or should I be using another test?

Answer 1

I would probably opt for McNemar's test if you only train the classifiers once. David Barber also suggests a rather neat Bayesian test that seems rather elegant to me, but isn't widely used (it is also mentioned in his book).

Just to add, as Peter Flom says, the answer is almost certainly "yes" just by looking at the difference in performance and the size of the sample (I take the figures quoted are test set performance rather than training set performance).

Incidentally Japkowicz and Shah have a recent book out on "Evaluating Learning Algorithms: A Classification Perspective", I haven't read it, but it looks like a useful reference for these sorts of issues.

Answer 2

I can tell you, without even running anything, that the difference will be highly statistically significant. It passes the IOTT (interocular trauma test - it hits you between the eyes).

If you do want to do a test, though, you could do it as a test of two proportions - this can be done with a two sample t-test.

You might want to break "accuracy" down into its components, though; sensitivity and specificity, or false-positive and false-negative. In many applications, the cost of the different errors are quite different.

Answer 3

Since accuracy, in this case, is the proportion of samples correctly classified, we can apply the test of hypothesis concerning a system of two proportions.

Let $\hat p_1$ and $\hat p_2$ be the accuracies obtained from classifiers 1 and 2 respectively, and $n$ be the number of samples. The number of samples correctly classified in classifiers 1 and 2 are $x_1$ and $x_2$ respectively.

$ \hat p_1 = x_1/n,\quad \hat p_2 = x_2/n$

The test statistic is given by

$\displaystyle Z = \frac{\hat p_1 - \hat p_2}{\sqrt{2\hat p(1 -\hat p)/n}}\qquad$ where $\quad\hat p= (x_1+x_2)/2n$

Our intention is to prove that the global accuracy of classifier 2, i.e., $p_2$, is better than that of classifier 1, which is $p_1$. This frames our hypothesis as

• $H_0: p_1 = p_2\quad$ (null hypothesis stating both are equal)
• $H_a: p_1 < p_2\quad$ (alternative hypotyesis claiming the newer one is better than the existing)

The rejection region is given by

$Z < -z_\alpha \quad$ (if true reject $H_0$ and accept $H_a$)

where $z_\alpha$ is obtained from a standard normal distribition that pertains to a level of significance, $\alpha$. For instance $z_{0.5} = 1.645$ for 5% level of significance. This means that if the relation $Z < -1.645$ is true, then we could say with 95% confidence level ($1-\alpha$) that classifier 2 is more accurate than classifier 1.

References:

1. R. Johnson and J. Freund, Miller and Freund’s Probability and Statistics for Engineers, 8th Ed. Prentice Hall International, 2011. (Primary source)
2. Test of Hypothesis-Concise Formula Summary. (Adopted from [1])