0

I know how effective ReLU are for deep neural networks, however i wanted to know if they can be used for smaller networks as well ? ( like a network with 2 hidden layers and 15 units each ).

aditya ramesh
  • 13
  • 3
  • 1
    Why not? Activation layer is just a transformation. As long as your network itself is a good fit for your data set, everything should be ok. – SmallChess Sep 20 '16 at 07:16
  • [Same question](http://stats.stackexchange.com/questions/225748/does-relu-layer-work-well-for-a-shallow-network) – Asciiom Sep 20 '16 at 08:55
  • I found it not performing well in super small tiny networks, but maybe I'm making some error. Please refer: https://stats.stackexchange.com/questions/284203/why-relu-activation-cannot-fit-my-toy-example-sinus-function-keras – thedk Jun 08 '17 at 22:27

1 Answers1

0

Can be used, yes. Are effective depends on your network, on your problem, and compared to what other activation function. In my experience ReLUs are pretty good even in small networks on classical problems (MNIST, CIFAR10) compared to tanh or sigmoid. The difference is more obvious in deeper networks though.

Jenkar
  • 354
  • 2
  • 10
  • I think the answer is wrong. There's evidence that ReLU is also good for deep networks. In fact I believe Google trained a 17-layer network with ReLu for AlphaGo. – SmallChess Sep 20 '16 at 09:07
  • My formulation might be a little bit wonky, edited. I did mean to say that they perform even better in deeper network. – Jenkar Sep 20 '16 at 09:11