0

Can we say that if our learning is too small then vanishing gradient will occur.

And if our learning rate is too large then exploding gradient will occur?

very small learning rate == vanishing gradient,

very large learning rate == exploding gradient

Is this concept right or wrong?

zahid
  • 1
  • Exploding and vanishing gradients are about the **gradients**, not the learning rate. Setting the learning rate too small/large is also a problem, but it's a *different* problem. – Sycorax Dec 16 '21 at 18:59
  • Hi, Thank you for the answer but I am reading an article where a well known professor explained that "Small learning rates consume a lot of time to converge or will not be able to converge because of the vanishing gradient, i.e. the gradient goes to 0. Large learning rates puts the model at risk of overshooting the minima so it will not be able to converge: what is known as exploding gradient." – zahid Dec 16 '21 at 21:11
  • If the gradient goes to zero, then any finite learning rate $c$ won't update the weights: $c \cdot 0 = 0$. – Sycorax Dec 16 '21 at 21:12
  • Welcome to Cross Validated! Please give a full citation for the article so we can review it ourselves. – Dave Dec 16 '21 at 21:12
  • Here is the link to the article "https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484". – zahid Dec 16 '21 at 21:12
  • Yes if we define a small learning rate the it will vanish the gradient more faster. So is it a cause of vanishing gradient?? – zahid Dec 16 '21 at 21:18
  • This person is not a professor. Their bio says that they're a student and intern. https://medium.com/@reina.wang/about I suggest consulting a high-quality textbook like Ian Goodfellow's *Deep Learning*, not blog articles written by students. – Sycorax Dec 16 '21 at 21:18
  • Sorry I pasted wrong article link here is the link https://towardsdatascience.com/frequently-asked-questions-on-learning-rate-6defb4e45d2e#:~:text=Small%20learning%20rates%20consume%20a,is%20known%20as%20exploding%20gradient. – zahid Dec 16 '21 at 21:27
  • https://medium.com/@sebaidorsaf Link to her profile – zahid Dec 16 '21 at 21:28

0 Answers0