0

After having this exact same issue with caret, I arrived at this thread. However, I do not intuitively understand why this answer is correct.

Why can't there be more weights than the number of observations? Is this a bug/idiosyncrasy in this particular R package or is there a statistical reason for it?

Karolis Koncevičius
  • 4,282
  • 7
  • 30
  • 47
e4a
  • 11
  • 2

2 Answers2

5

Not only neural networks can have more weights, then samples, but there are some preliminary results showing that so called overparametrized neural networks (ones that have more parameters then samples) can overperform smaller networks. Below you can see figure by Belkin et al (2019) illustrate the phenomenon observed in some experiments, where the test error first falls with growing number of hidden units, then starts overfitting when the number of hidden units approaches the number of samples, but after surpassing the interpolation threshold (at this point the network is able to memorize the training dataset), but then it starts falling again with increasing complexity of the network.

enter image description here

Tim
  • 108,699
  • 20
  • 212
  • 390
  • That is a super interesting paper (+1) but I can see that result being abused to the world's end. This idea of "interpolation regime" reminds me of the later 90's, early 00's papers where "*Adaboost would never overfit*" but then we realised it does. – usεr11852 May 08 '20 at 10:33
  • @usεr11852 agree, it will be abused and misinterpreted like universal approximation theorem, while same as it, it has negligible practical relevance. But it is interesting. – Tim May 08 '20 at 10:45
4

It’s an issue with the particular software, maybe not a bug, but at least a matter of how the software performs, not neural networks themselves.

Consider training an MNIST network in Keras. You can have stellar out-of-sample accuracy when you have more than 60,000$^{\dagger}$ weights, so certainly a neural network model allows for more weights than observations.

$^{\dagger}$There are 60,000 training images in the MNIST data set.

Dave
  • 28,473
  • 4
  • 52
  • 104
  • Fully valid example (+1) but I think we need to make some distinction between memorisation and generalisable performance. – usεr11852 May 08 '20 at 10:31