0

I am trying to classify a particular image.

My labels are PatternA and NotPatternA.

NotPatternA is any image that I have not classified as PatternA. There are very few PatternAs compared to NotPatternA. In my data I have 500 PatternA and 6000 NotPatternA.

I construct my training and testing sets, by choosing 400 from PatternA at random for training and the remaining 100 I put into Testing.

For NotPatternA I randomly draw a sample of 500 images and again split it 400/100 into training and testing.

After 10 epochs I start to get very low loss and high accuracy (near 100%). BUT testing gives me very high loss (2+). However, testing accuracy is also good (near 80%).

Why might my testing loss be so high?

ManInMoon
  • 183
  • 7

1 Answers1

1

Overfitting. Try incorporating cross-validation into your training schedule. Most ML libraries provide a way of cross-validating your model during the training process so that you can stop training once your model begins to overfit.

Edit: The reason why you are observing "high accuracy" in your test data set is because of the label imbalance. Your model is overfitted to the training set, but when it sees the new test data it can just guess NotPatterA and still be correct 90% of the time (this is also why I put "high accuracy" in quotes - in fact, 80% accuracy is not a good result given your label ratio).

timchap
  • 171
  • 4
  • BUT my training set and my testing set both have an equal number of the 2 labels... Surely 80% accuracy is therefore good? – ManInMoon Sep 09 '19 at 10:40
  • 1
    You need to think carefully about the definition of the metric "accuracy". Accuracy is the ratio of correct predictions to total number of predictions. In both your data sets, there are approximately 90% of samples with label NotPatternA. Therefore if I were to create an "algorithm" which predicted NotPatternA 100% of the time, I would achieve 90% accuracy on both test and train datasets. – timchap Sep 09 '19 at 10:48
  • 1
    No, I don't think so. I have specifically chosen to have 50% of PatternA and 50% of NotPatternA in both training and testing. – ManInMoon Sep 09 '19 at 10:50
  • Oh, you're quite right. I misread that part of your question. – timchap Sep 09 '19 at 10:51
  • (My suggestion to use cross-validation stands though) – timchap Sep 09 '19 at 10:52
  • I could try that. I will need to work out how! What I have already done is to rerun the training with freshly random draws - and I get the similar results. High test accuracy but very high loss too. Whereas in training the loss is close to 0... – ManInMoon Sep 09 '19 at 10:55
  • I am using Keras and Tensorflow in Python. Is the cross-validation an option in "model.fit"? – ManInMoon Sep 09 '19 at 11:02
  • Yep, the documentation lists both `validation_split` and `validation_data` as an argument of `model.fit` https://keras.io/models/model/ – timchap Sep 09 '19 at 12:44
  • Yes, got something working from sklearn. I get the same result for each fold. Training loss is low, but testing loss is high BUT with good accuracy! – ManInMoon Sep 09 '19 at 14:17