21

Hinge loss can be defined using $\text{max}(0, 1-y_i\mathbf{w}^T\mathbf{x}_i)$ and the log loss can be defined as $\text{log}(1 + \exp(-y_i\mathbf{w}^T\mathbf{x}_i))$

I have the following questions:

  1. Are there any disadvantages of hinge loss (e.g. sensitive to outliers as mentioned in http://www.unc.edu/~yfliu/papers/rsvm.pdf) ?

  2. What are the differences, advantages, disadvantages of one compared to the other?

DHW
  • 644
  • 3
  • 13
user570593
  • 1,099
  • 2
  • 13
  • 19

3 Answers3

31

Logarithmic loss minimization leads to well-behaved probabilistic outputs.

Hinge loss leads to some (not guaranteed) sparsity on the dual, but it doesn't help at probability estimation. Instead, it punishes misclassifications (that's why it's so useful to determine margins): diminishing hinge-loss comes with diminishing across margin misclassifications.

So, summarizing:

  • Logarithmic loss leads to better probability estimation at the cost of accuracy

  • Hinge loss leads to better accuracy and some sparsity at the cost of much less sensitivity regarding probabilities

Firebug
  • 15,262
  • 5
  • 60
  • 127
  • 1
    +1. Minimizing logistic loss corresponds to maximizing binomial likelihood. Minimizing squared-error loss corresponds to maximizing Gaussian likelihood (it's just OLS regression; for 2-class classification it's actually equivalent to LDA). Do you know if minimizing hinge loss corresponds to maximizing some other likelihood? I.e. is there any probabilistic model corresponding to the hinge loss? – amoeba Mar 28 '18 at 15:51
  • 1
    @amoeba It's an interesting question, but SVMs are inherently not-based on statistical modelling. Having said that, check [this answer](https://stats.stackexchange.com/a/313611/60613) by Glen_b. The whole thread is about it, but for the epsilon-insensitive hinge instead. – Firebug Mar 28 '18 at 16:03
5

@Firebug had a good answer (+1). In fact, I had a similar question here.

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

I just want to add more on another big advantages of logistic loss: probabilistic interpretation. An example, can be found here

Specifically, logistic regression is a classical model in statistics literature. (See, What does the name "Logistic Regression" mean? for the naming.) There are many important concept related to logistic loss, such as maximize log likelihood estimation, likelihood ratio tests, as well as assumptions on binomial. Here are some related discussions.

Likelihood ratio test in R

Why isn't Logistic Regression called Logistic Classification?

Is there i.i.d. assumption on logistic regression?

Difference between logit and probit models

Haitao Du
  • 32,885
  • 17
  • 118
  • 213
1

Since @hxd1011 added a advantage of cross entropy, I'll be adding one drawback of it.

Cross entropy error is one of many distance measures between probability distributions, but one drawback of it is that distributions with long tails can be modeled poorly with too much weight given to the unlikely events.

aerin
  • 744
  • 5
  • 14