11

Regarding dropout, we know that in the forward propagation some neurons are put to "zero" (i.e., turned off). How about back propagation ?

Are these dropped out neurons also zeros (turned off) during back-prop ?

Thank

Refer to this link, which seems to be not very clear ... : Dropout backpropagation implementation

Community
  • 1
Bill Ancalagon the black
  • 1,704
  • 2
  • 14
  • 16

1 Answers1

24

Yes, the neurons are considered zero during backpropagation as well. Otherwise dropout wouldn't do anything! Remember that forward propagation during training is only used to set up the network for backpropagation, where the network is actually modified (as well as for tracking training error and such).

In general, it's important to account for anything that you're doing in the forward step in the backward step as well – otherwise you're computing a gradient of a different function than you're evaluating.

The way it's implemented in Caffe, for example, is (as can be verified from the source):

  • In forward propagation, inputs are set to zero with probability $p$, and otherwise scaled up by $\frac{1}{1 - p}$.

  • In backward propagation, gradients for the same dropped units are zeroed out; other gradients are scaled up by the same $\frac{1}{1-p}$.

Danica
  • 21,852
  • 1
  • 59
  • 115
  • Thanks a lot, you are right. Just one note: I think "drop out " will still do something even if the neurons are only turned off during forward-prop while not during back-prop. That will change the other neurons' values in the forward-prop because this "drop out" changed the connection between these neurons. However, you are right that this kind of "drop out" would be "computing a gradient of a different function than you're evaluating." Thanks again for your clear and sound answer. – Bill Ancalagon the black Jun 17 '16 at 19:47
  • @Bill Yes, if you did it that way it would change things, but what you'd be doing would be inconsistent and strange. – Danica Jun 17 '16 at 20:15