3

For logistic regression, the loss function is convex or not? Andrew Ng of Coursera said it is convex but in NPTEL it is said is said it is non convex because there is no unique solution. (many possible classifying line)

Sycorax
  • 76,417
  • 20
  • 189
  • 313
  • 4
    The convexity of logistic regression is demonstrated in [What is happening here, when I use squared loss in logistic regression setting?](https://stats.stackexchange.com/questions/326350/what-is-happening-here-when-i-use-squared-loss-in-logistic-regression-setting) – Sycorax Jun 13 '19 at 00:38

1 Answers1

8

I don't think anybody claimed that it isn't convex, since it is convex (maybe they meant logistic function or neural networks). Let's check 1D version for simplicity

$$L = - t \log(p) + (1 - t) \log(1-p)$$

Where $p = \frac{1}{1 + \exp(-wx)}$

$t$ is target, $x$ is input, and $w$ denotes weights.

L is twice differentiable with respect to $w$ and $\frac{d}{dw^2} L = \frac{x^2 \exp(wx)}{(1 + \exp(wx))^2} > 0$, so the loss function is convex.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
Jakub Bartczuk
  • 5,526
  • 1
  • 14
  • 36