3

I am training a network to solve a regression problem using Keras. During training, the loss of my model goes directly from 7 to more than 300000 dramatically. Here is the training output: enter image description here

Here is the loss picture: enter image description here

I wonder what could be the cause of this problem? Thanks

Glen_b
  • 257,508
  • 32
  • 553
  • 939
SunshineAtNoon
  • 503
  • 1
  • 5
  • 9
  • What optimization algorithm are you using? And what are the hyper-parameters (step-size, momentum etc.) of the respective optimization algorithm? – Armen Aghajanyan Feb 29 '16 at 18:24
  • Sorry I forgot that, I use Adam as the optimization algorithm. The parameters are: learning rate=1e-4, beta_1=0.9, beta_2=0.999, epsilon=1e-08 – SunshineAtNoon Mar 03 '16 at 01:28

1 Answers1

-1

Based on my experience, just saying what i have seen, a few things could cause this: 1. learning rate. if it's too large, it could cause rising loss; just make it smaller, magnitudes smallers 2. in case of RNN, it could all blow up, weights, gradients, loss; there's paper published dealing with that.

Matian2049
  • 99
  • 1