0

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability p and not present with a probability 1-p. In the test phase, all units are present, but we multiply each of them with the probability.

Now, is it like, let we have 4 input units originally named a,b,c,d. In the training stage, after applying dropout, with a dropout rate of 0.5, we are left with units a and c. So, As in the test stage, all the units are present, so, is it like, we multiply each of the units with 0.5? Also, Is p defined for each of the units in the network, or for the entire Neural Network?

Also, In doing so, how is the result same for training and test stage?

Akash Dubey
  • 425
  • 4
  • 15

1 Answers1

0

Your description is more or less correct. First thing to note though is that dropout is usually applied on an entire layer, rather than a neural network. Of course, you could apply dropout to all layers in your neural network.

Let's assume we are dealing with a fully connected network. If dropout is applied during training with a factor of 0.5, that means that only half of all activation functions will deliver an output. The idea is that by training a different mix of units during each step, you get a more robust model that is able to deal well with noise/variability. (You can also compare it to the effect of ensemble networks.)

Now during testing however, we would like to use the full knowledge of the network, and thus not drop any units. Doing so would mean that our network is no longer properly scaled: the expected value during testing no longer matches the expected value during training. A simple way to fix this is indeed to scale all the activation outputs of the layer by the value of dropout (p).

When p is 0.5, during testing, this means we have double the amount of outputs at the layer in question, so we rescale all those outputs by multiplying them with 0.5.

GR4
  • 632
  • 3
  • 7