2

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum.

Will adding more neural units solve the problem?

Silverfish
  • 20,678
  • 23
  • 92
  • 180
ramborambo
  • 205
  • 1
  • 9
  • the way to avoid local minima is by using momentum Read this for more information http://www.willamette.edu/~gorr/classes/cs449/momrate.html – Cyber Progs May 17 '15 at 12:44
  • 3
    You can see from [this answer](https://stats.stackexchange.com/a/203300/103153) that "recovering the global minimum becomes harder as the network size increases". – Lerner Zhang Jun 19 '19 at 23:27

1 Answers1

-1

Yes, you will definitely avoid local minima with more neurons. Adding more neurons increases the volume of your solution space exponentially, but at the same time increases the amount of equivalent solutions factorially.

Lugi
  • 586
  • 2
  • 11
  • 1
    I would expect the opposite. As an extreme counter-example, consider the case where you only have 1 neuron. Then, you can always find the global optimum. – George Jun 20 '19 at 18:19
  • 1
    What guarantee do you have that this 1-dimensional function is going to be convex? – Lugi Jun 21 '19 at 12:46
  • What I had in mind is a network that consists of a single neuron which can be optimized with the perceptron algorithm. – George Jun 21 '19 at 17:29
  • The question is about gradient descent. – Lugi Jun 22 '19 at 18:05
  • 1
    this is interesting. have you got any reference? – carlo Oct 20 '19 at 10:01