2

I used a SVM to classify my features. With cross validation, on my testing set, I obtained an accuracy of 81%. I want to measure how good/bad this is. Is the technical term for good/bad statistical significance?

With my SVM we predict that an 8 dimensional feature vector is in 1 out of the 2 classes. I wanted to use a permutation test to find the p-value, but I do not know what to use for the test statistic.

  1. What can I use for the test statistic?
  2. Is this the wrong line of thought? What else can I do to show that 81% is a good/bad baseline?

Thanks for all the help!

CodeKingPlusPlus
  • 403
  • 1
  • 6
  • 14
  • 2
    Please be aware that p-values and statistical significance differ greatly from classification accuracy. See, for instance, http://stats.stackexchange.com/questions/31. So, your cross-validation accuracy is 81%. Whether that's "good" or "bad" depends on your application and your objectives, about which you have told us little. Your question therefore does not seem to be answerable--and even if it were, it would likely come down to a matter of opinion. Could you edit it to clarify these points? – whuber Apr 30 '14 at 18:12
  • Is the size of your classes the same? If one class has 81 instances and the other has 19, then 81% accuracy is not very good... – Bitwise Apr 30 '14 at 20:09
  • The size of my classes are about the same, the training set is about 1000, and the classes are about 500 each. – CodeKingPlusPlus Apr 30 '14 at 21:05
  • @Bitwise do you have any suggestions for me? – CodeKingPlusPlus Apr 30 '14 at 21:13
  • As whuber says, I don't see any use of doing a statistical test here. Imagine that you had a p-value - it would be significant, but what would it tell you? nothing in my opinion. You tested in cross-validation, so you have an estimation of how good you are doing (the baseline should be 50% if the classes are equal). This means that on future data you expect to be right in 81% of your predictions. Is that good enough? Depends on the application. – Bitwise May 01 '14 at 00:11
  • I feel like you are asking if there is an upper and lower bound on actual accuracy, given that with your training set you got an accuracy of 81%, something like, the upper bound is 83% and the lower bound is 79% on 99% of arbitrary training sets/test sets. – fileyfood500 Jun 13 '18 at 18:32

0 Answers0