2

I am applying kNN and SVM classifiers to my classification problem. Both of the classifiers get over 95% cross-validation accuracy (leave one out cross validation used). Not sure how to tell if the results of two classifiers are statistically significant different?

I did a little bit search that

Discussion1 Discussion2

these posts discussed this topic; however, I am still confused how they use hypothetical test for this purpose.

For example, I have two cross-validation accuracies and how to form the 2 × 2 contingency tables of McNemar's test?

If my performance measure is AUC of ROC curve, then how to perform the McNemar's test? Thanks a lot. A.

Community
  • 1
Samo Jerom
  • 1,439
  • 2
  • 19
  • 31
  • 3
    McNemars test is used when you have cell counts of correctly and incorrectly classified cases for two classifiers such as page 22 of http://web.engr.oregonstate.edu/~tgd/classes/534/slides/part13.pdf – B_Miner Apr 12 '14 at 00:33
  • 1
    In your case you essentially have 1 value of AUC for both models. Not much can be done with that except compare them on face value. You are better off I think using a bootstrap or k-fold or other type of resampling and then use a paired t-test or Wilcoxon signed-rank test – B_Miner Apr 12 '14 at 00:38
  • Thanks a lot for your comments. As the size of my data set is relatively small. Therefore, I use leave-one-out cross validation. As you mentioned using paired t-test or Wilcoxon signed-rank test, I am still confused. Do you mean that I need to change to use k-fold cross validation in order to get the paired t-test result? – Samo Jerom Apr 14 '14 at 09:38
  • I think so Samo (needing to pick a different method). It seems to me you need to have variation in your estimate of AUC to be able to do any statistical testing. You are using the test to make inference about the "population" which you mimic by repeated resampling. Perhaps someone else knows how to use a single run of LOOCV to draw this inference, but I do not. Can you perhaps use a bootstrap on your data - that will be suited to "smaller" data and will allow you to create a distribution of AUC. Another option is to run a permutation test where you shuffle the lables and fit a new model.... – B_Miner Apr 16 '14 at 18:46
  • And calculate AUC. Then after doing this many times, compare the AUC you originally achieved with this distribution of AUC under the null hypothesis (good AUC on shuffled labels should only be a fluke). Hope this helps a little. You can wait for others to make suggestions or start a bounty as well to draw attention). – B_Miner Apr 16 '14 at 18:49

0 Answers0