0

I would like to verify the following methodology for using SVMs for three way classification. That is, the response $Y$ can be either $\{-1, 0, 1\}$:

First train an SVM to distinguish between $\{-1,1\}$ and $\{0\}$. Then train an SVM to distinguish between $\{-1,1\}$. For each instance $X$, first use the first SVM, and if it is not $\{0\}$, classify it using the second SVM.

My main questions are:

  1. Whether this type of thing is common
  2. Whether I can possibly get better results by using other combinations, e.g., first distinguishing between $\{-1, 0\}$ and then $\{0, 1\}$?
Gilles
  • 1,022
  • 1
  • 10
  • 21
Tommy
  • 205
  • 2
  • 8

1 Answers1

2

The most common approach is one-vs-all classification, which involves making $n$ models for $n$ classes. In your case that would be 1 vs (0, 1), 0 vs (-1, 1) and -1 vs (0, 1). Another common approach is all-vs-all, which requires $n(n-1)$ models and is hence more computationally demanding.

Your approach can work, but it depends largely on the problem. If distinguishing zero from the others is particularly difficult, for instance, you may get worse results. I recommend sticking to the usual methods.

Marc Claesen
  • 17,399
  • 1
  • 49
  • 70
  • Marc, thanks! How do we combine the SVMs, by majority? E.g., what happens when an instance recieves "1" over (0,1) and "0" over (-1,1) by the two respective SVMs. How does this discrepancy get resolved? If you have good literature to point me to I would gladly read that; everything I read only talked about SVMs in the binary case. – Tommy Mar 18 '15 at 17:16
  • ah, ok, I see that for SVMs (where a probability of each class is not predicted), we choose the class with the *greatest margin* http://nlp.stanford.edu/IR-book/html/htmledition/multiclass-svms-1.html. I was thinking in terms of pure output labels rather than the margin. – Tommy Mar 18 '15 at 18:26
  • also I just found this post so sorry about that: http://stats.stackexchange.com/questions/21465/best-way-to-perform-multiclass-svm – Tommy Mar 18 '15 at 18:36