12

I understand that dropout is used to reduce over fitting in the network. This is a generalization technique.

In convolutional neural network how can I identify overfitting?

One situation that I can think of is when I get training accuracy too high compared to testing or validation accuracy. In that case the model tries to overfit to the training samples and performs poorly to the testing samples.

Is this the only way that indicates whether to apply dropout or should dropout be blindly added to the model hoping that it increases testing or validation accuracy

Community
  • 1
Kapes
  • 123
  • 1
  • 5
  • 1
    As far as I know dropout could never hurt. Another way to monitor the overfitting potential is to graph the magnitude of your weights. Higher weights lead to overfitting. – Frobot Mar 30 '16 at 03:43

1 Answers1

17

In convolutional neural network how can I identify overfitting?

Comparing the performance on training (e.g., accuracy) vs. the performance on testing or validation is the only way (this is the definition of overitting).

enter image description here

should dropout be blindly added to the model hoping that it increases testing or validation accuracy?

Dropout often helps, but the optimal dropout rate depends on the data set and model. Dropout may be applied to different layers in the network as well. Example from Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.:

enter image description here

You may also want to do some early stopping:

enter image description here

Franck Dernoncourt
  • 42,093
  • 30
  • 155
  • 271