Difference between deeper and wider neural networks

Question

I'm learning NN and trying to understand the implication that architecture selection has in the model. Am I inferring correctly that:

Wider neural networks can approximate more interactions between input variables
Deeper neural networks can model more complex nonlinearities

Yes exactly like that, but as the net gets deeper, the effect like you mention might not be as prevalent. — Yohanes Alfredo, Nov 20 '19 at 11:31
See also https://stats.stackexchange.com/questions/222883/why-are-neural-networks-becoming-deeper-but-not-wider — Jan Kukacka, Nov 21 '19 at 18:14

score 2 · Accepted Answer · answered Nov 20 '19 at 11:43

Roughly.

First, that's assuming all layers are created equal. Imagine you want to interleave two different activation functions for each layer, or add dropout - then deeper networks give you more legroom for those to work their magic.

Second, a very deep, moderately wide net may be able capture the same interactions as a shallow, absurdly wide one, but not vice versa.

Difference between deeper and wider neural networks

1 Answers1