1

From the literature I read that

For a neural network, the cost function, J(W,b) is a non-convex function, gradient descent is susceptible to local optima; however, in practice gradient descent usually works fairly well.

But on the other hand, support vector machine which is another popular classifier involves optimizing convex functions.

How is that solving SVM is a convex optimization problem and for feed forward neural network it is non-convex optimization?

Edit: Cost function of neural network is non-convex?

This link provides some information regarding the intuition behind the optimization in neural networks. But it is not clear there. Also my question was in the reason behind the difference in the optimization approaches in ANNs and SVMs. Both involves a cost function involves a sum of squared errors and a regularization term. Why is one convex optimization problem (SVM) and the other optimizing non-convex functions.

Community
  • 1
prashanth
  • 3,747
  • 4
  • 21
  • 33
  • 3
    The answer really isn't **too** illuminating. The SVM problem is convex because the constraints make it a quadratic program. Indeed, one of the motivations for SVMs was to construct a regularized Percepton algorithm that has a unique solution. – Sycorax Feb 11 '16 at 15:19
  • 2
    @IndieAI I don't think this is a duplicate. OP's specific question is about the contrast between SVMs and ANNs. It appears that OP understands what nonconvexity means and what it implies for optimization. – Sycorax Feb 11 '16 at 15:20
  • @Indie AI. This question is not a duplicate. I am looking for the intuition behind one being convex and the other being non-convex optimization. Both of them involve a cost function which includes a sum of squared errors plus a regularization term. But the link was helpful. It sure provides some hints about the non convex optimization in neural networks. But it is not very clear although. – prashanth Feb 12 '16 at 06:45
  • @user777. You are right. I am looking for the contrast between the type of optimization in ANN and SVM. – prashanth Feb 12 '16 at 06:50
  • 1
    @prashanth are you familiar with constrained and and unconstrained optimization? That's what it boils down to. – Sycorax Feb 12 '16 at 12:51
  • @user777 ok. yes SVM involves constrained optimization and the other is unconstrained. But how it is related to convex and non-convex optimization? – prashanth Feb 13 '16 at 11:34

0 Answers0