1

Why are there differences between the coefficients for logistic regressions and SVM even when they give more or less the same results? Is there any interpretation for the differences?

I am currently using both logistic regression and SVM (linear kernel) classifier to perform classification among picture of males and females. Both the classifiers are giving me the same results yet when I check their coefficients (.coef_) I find that SVM coefficients have a much shorter range (0.2 to -0.2) whereas logistic regression model has a wider range (0.4 to -0.4). What is the reason for this difference?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
CS1999
  • 11
  • 1

1 Answers1

0

If you are not using regularised logistic regression, the difference in average magnitude is probably caused by that. The SVM has a term in the cost function that aims to minimise the magnitude of the weights (which is where the maximum margin separation idea comes from).

Regularised logistic regression will also give somewhat different parameter estimates as it is trying to give accurate estimates of the probability of class membership everywhere, whereas the SVM is trying only to locate the decision boundary for this problem (in the default case, this is where the probability of class membership is 0.5). This is both a good thing and a bad thing. If accuracy genuinely is the quantity of interest, then the SVM may give better results as it is less likely to be affected by trying to minimise errors at high or low probabilities at the expense of errors near 0.5 (see my answer to a vaguely related question). However, this does mean that if misclassification costs or class frequencies change, you need to retrain the classifier.

There is a good discussion of the difference at the end (Appendix D.2) of Mike Tipping's paper on the Relevance Vector Machine (a sort of Bayesian regularised logistic regression model).

Dikran Marsupial
  • 46,962
  • 5
  • 121
  • 178