What causes sudden drops in training/test errors when training a neural network?

Question

I've seen plots of test/training error suddenly dropping at certain epoch(s) a few times during the neural network training, and I wonder what causes these performance jumps:

This image is taken from Kaiming He's Github, but similar plots show up in many papers.

One related paper: Smith et al. 2018 Don't Decay the Learning Rate, Increase the Batch Size, https://openreview.net/forum?id=B1Yy1BxCZ — amoeba, Nov 20 '19 at 09:53

score 9 · Answer 1 · answered Nov 15 '18 at 08:39

9

They changed the learning rate. Note the drop is at exactly 30 and 60 epochs, obviously set manually by someone.

answered Nov 15 '18 at 08:39

cwhy

136
1
2

In section 3.4, it explains that the learning rate is divided by 10 when the error plateaus – xiawi Jan 08 '20 at 11:16

score 1 · Answer 2 · answered May 08 '20 at 07:40

1

If you refer to the ResNet (Deep Residual Learning for Image Recognition) paper it reads as follows, "The learning rate starts from 0.1 and is divided by 10 when the error plateaus". Hence, the reason fro drop is the update in the learning rate.

answered May 08 '20 at 07:40

Meysam Sadeghi

111
2

What causes sudden drops in training/test errors when training a neural network?

2 Answers2

Linked

Related