0

I was training my NN when I found out something I CAN NOT understand. My net is a bilstmLayer and a softmaxLayer layer with 10 MaxEpochs and 150 MiniBatchSize. I want to classify 4 different type of signals, each one with about 10000 values. I have around 200 signals of each type.

Ok. Why are there 9 different patterns in the Accuray Graph? (It should be 10, but I cut the training on the 10th epoch before it finished) The graph shouldn't have any pattern, right? It should be random and shouldn't follow any pattern based on the epochs.

Please If anyone knows why this phenomenon happens I will be pleased to know.

enter image description here

Jan Kukacka
  • 10,121
  • 1
  • 36
  • 62
Henrial_14
  • 1

1 Answers1

7

I think if you plotted the average loss for each epoch, instead of the average loss for each minibatch, you'd see that your loss is basically flat, indicating that the network is not improving from one epoch to the next.

If the quality of model predictions doesn't change over time, and the inputs always appear in the same order, then you'd expect to see this kind of pattern.

So what's happening here is (1) the network isn't getting better at making predictions and (2) the inputs are always in the same order. This will give a periodic appearance to the plot, because the same predictions are being made at the same relative position, over and over again.

Some suggestions on how to improve the network can be found here: What should I do when my neural network doesn't learn?

Sycorax
  • 76,417
  • 20
  • 189
  • 313
  • Correct me if i'm wrong, but this would also affect other learning methods, not just LSTM, i think? The order being in the inputs (as in, as a "time" dimension to the input vector), not between them – Jenkar Apr 08 '19 at 15:43
  • I don't understand what you mean. Could you try to explain, and give pronouns antecedents? – Sycorax Apr 08 '19 at 15:45
  • As in, the fact that LSTMs operate on ordered inputs isn't really relevant for this question/answer. For example, if we have inputs A (1,2,3,4) and B (2,5,7,8), which we pass in a batch to an LSTM, the LSTM doesn't use the fact that the batch is in the order (A,B) or the order (B,A), but the order inside each input – Jenkar Apr 08 '19 at 16:01
  • 1
    Order is relevant if A, B are ordered. This can arise if you've taken a long time-series and parceled it out into smaller chunks, A, B, C,... For example, a natural language model trained on the text of *War and Peace.* – Sycorax Apr 08 '19 at 16:04