6

I want to understand the logic behind keeping ReLU as $max(0,x)$ and not $min(0,x)$?

Why do we prefer positive inputs over the negative ones?

jsdbt
  • 121
  • 1
  • 6

1 Answers1

5

The weights learned in a neural network can be both positive and negative. So in effect, either form would work. Negating the input and output weights with the $\min$ form gives the same function as with the $\max$ form. The max form is used purely by convention.

AaronDefazio
  • 1,551
  • 7
  • 11
  • Can I keep it as $x$ only? Sparsity can anyway be induced by dropout. (PS Ignoring the non-linearity that $max$ or $min$ form would introduce in the system) – jsdbt Apr 01 '17 at 07:23
  • 2
    Without non-linearity, your network will compute just some linear function. No need to make it deep or anything. – Yuval Filmus Apr 01 '17 at 12:54