Is there something wrong in my neural network model?

Question

I designed my own neural network for solving the problem of text summarization. The number of documents in my training dataset is big (more than 100,000 documents) so it is hard to check it on the whole data. In order to verify that my model is good, I train it on a very small dataset (100 documents) in about 100 epochs to see how it behaves. I split this small dataset into 3 sets (6/2/2): training, validation and test. Here is the chart of the losses (red line is training loss, blue line is validation loss and green line is test loss)

Is my evaluation on this small dataset good enough to tell whether my model is performing well?
Does the above chart shows that my model is getting overfitting easily?
Do you have any recommendation for quickly evaluating a new proposed model and avoid overfitting?

Update:

I trained my network again on a bigger data set (4000 documents) and I got the following chart. It does seem my model is not well-designed.

Wow, 100 entries seems way too small to get valuable information. From my experience you can push up to 5000-6000 entries and it will not take too much time to train (around 15 minutes). Then you will have meaningful information to play with. — LoulouChameau, Aug 24 '16 at 09:20
You seem to be overfitting to your small training data-set. You should add more data, like suggested by @LoulouChameau. If you insist on keeping the small data set for training, then from your graph, you should only train for around 10 epochs, because after that the validation error starts going up, then model starts overfitting and it looses the power to generalize. — Gumeo, Aug 24 '16 at 10:57
Thank you for your comments. I have updated my post with a chart when training on a bigger dataset (4000 docs). It seems to behave similarly though — The Lazy Log, Aug 24 '16 at 11:47
Overfitting still happens at 10 epochs, though. So. Don't train it for more than 10 epochs. — Sycorax, Aug 24 '16 at 14:16
@GeneralAbrial. Yeah, but in that case the loss is still too big and the network does not learn anything useful in my problem. — The Lazy Log, Aug 25 '16 at 00:13
@TheLazyLog Wellllll you'll need to start looking into alternatives: using more data, changing the architecture, using regularization and other tools to get things where you want them. — Sycorax, Aug 25 '16 at 00:45

Franck Dernoncourt · Answer 1 · 2016-08-25T19:34:31.260

Is my evaluation on this small dataset good enough to tell whether my model is performing well?

It's difficult to tell when using a fraction of the available dataset. Also, it depends on the complexity of the task, among other things.

Does the above chart shows that my model is getting overfitting easily?

Yes since overfitting starts before epoch 5.

Do you have any recommendation for quickly evaluating a new proposed model and avoid overfitting?

Add some patience to avoid computing more epochs that needed, and some regularization technique to avoid overfitting too early

Is there something wrong in my neural network model?

1 Answers1