3

I'm using libsvm in C-SVC mode (-s= 0) with linear kernel (-t= 0), and I'm required to train multiple SVMs( I have four classes).

My training and test sets have the same number of instances and features, they have 32768 instances (rows) and 128 features(columns).

I used five different C values which are: 1, 8, 64, 256, 2048.

During the training, I got this warning message:

"Warning: reaching max number of iterations"
"Optimisation finished, tier = ******".

The problem is: logically, I know that each time I increase C value, the accuracy of classification should be getting better than before or at least it should stay as before "nothing change". But in my case: after the C value = 256, the accuracy of classification is getting worse, and I don't know why I got this result and how to solve this problem.

By the way, I scaled my dataset, but I got the same result.

lennon310
  • 2,582
  • 2
  • 21
  • 30
Weam
  • 43
  • 3
  • "the accuracy of classification is getting worse" do you mean the training or testing accuracy? – lennon310 Feb 18 '14 at 01:43
  • Sorry, I want to add a note to my question. I trained my training set with my training labels using (svm-train), then I got the "model file" which I used in (svm-predict) to predict my test labels which were stored in the "output file", then I compared the predicted labels with the actual test labels in order to calculate the accuracy of the prediction [ This what I meant by the accuracy of classification] – Weam Feb 18 '14 at 23:11

1 Answers1

4

The warning indicates that the libsvm (thinks it) fails to find the maximum margin hyperplane. It may be caused by numerical stability issues. Some possible approaches that you may want to have a try:

  1. Try shrinking heuristics - h. It is aimed at speed up the optimization, thus reducing the running time instead of make it convergence if it is previous not.

  2. Use -w to set different weights to each class if your data is imbalanced.

  3. You may treat your problem as a binary classification, and use an one-vs-all strategy to achieve the multi-classification. In this post I provided the core code on this part.

The warning in your case implies that your trained model may be or may be not the optimal, so you may try to eliminate this warning first. Regarding the C value, as this post said, it controls the trade-off between wide margin and more training points mis-classified, and a narrow margin which fits the training points better but may be overfitted to the training data. As a result, larger C may tend to overfitting, and the test accuracy may be low. Yet I've no clue if you are talking about training accuracy in your OP, since I faced a similar problem.

Last but not least, if you are trying to find out an optimal C value, use grid search and cross validation to reduce the variance in testing.

Community
  • 1
lennon310
  • 2,582
  • 2
  • 21
  • 30
  • Actually I was talking about test accuracy, and also I've tried using -h but nothing changed. Yes, I'm trying to find out an optimal C value. So, your answer makes sense for me, and I'll try grid search, and I'll tell you if I faced any problem. Thank you so much for your answer. – Weam Feb 18 '14 at 22:40
  • Please, could you answer my question? Are these solutions will work with my problem if my training set is the same of the test set? – Weam Feb 19 '14 at 13:00
  • It's ok since you already have so many training samples. You data size may be a little bit large for a multiclass svm. Try to randomly sample a smaller set from the training set (say, 500 instances) and see whether the warning can be removed first. – lennon310 Feb 19 '14 at 13:32
  • If you are using large training samples, I suggest use one-vs-all as shown in my answer (3). If you are matlab, I provided the code in the link. Thanks. – lennon310 Feb 19 '14 at 13:33