1

I am working on a movie review sentiment analysis project, and comparing various classifiers on the same dataset. The data for the two classes is balanced, so I'm using accuracy on 3-fold cross validation as the basic measure of performance.

How can I check whether one classifier is better than another, with statistical significance? Is this a test I can do directly on the accuracy values, or do I need multiple accuracy values (from the multiple folds), or even individual classifications for each instance of data? Is the pair test applicable here?

Details: Dataset: 1000 positive, 1000 negative reviews. Bag of unigrams (words). Classifiers: Naive Bayes, SMO and LogReg Evaluation: Single accuracy percentage at the end of 3-fold stratified cross validation for each of the classifiers.

Neil
  • 111
  • 4

1 Answers1

1

This question on CV How to statistically compare the performance of machine learning classifiers? might help you. I recommend the literature pointed out by in the second (accepted) answer.

Community
  • 1
Jacques Wainer
  • 5,032
  • 1
  • 20
  • 32