1

DropOut is a good way to reduce over-fitting in Neural Network, and I found a article on DropOut.

My question is for each epoch, we should randomly pick half of the neurons of the network, and set the weights of them to zero, and update the weights of the networks? Does it right?

Show my process

for each epoch:
    randomly deactivate some neurons(in some percent) in each hidden layer
    learning with mini-batch data and update the weighs of the whole-nets 
get the output of weights
GoingMyWay
  • 1,111
  • 2
  • 13
  • 25

2 Answers2

2

for each epoch

it is typically done for each sample

we should randomly pick half of the neurons of the network

The dropout rate doesn't have to be 0.5, it could be anything from 0 to 1.

update the weights of the networks

yes, e.g. see Dropout: scaling the activation versus inverting the dropout

Community
  • 1
Franck Dernoncourt
  • 42,093
  • 30
  • 155
  • 271
  • 1
    Franck, thank you. Why `for each epoch`, it is typically done for each sample? In my view, for each epoch(trainning iteration), deactivating some neuron in each hidden layer, estimate the gradient for a different mini-batch, and updating the weights and biases in the network. – GoingMyWay Nov 28 '16 at 05:40
0

The answer provided by Franck is already a good correction on what happens at training time.

I would add that at test time you don't set any neurons to zero but you have to multiply the weights at test time by $p$, where $p$ is the fraction of neurons you put to 0 at training time. This is to ensure the magnitude is the same in expectation.

etal
  • 561
  • 2
  • 11