0

I'm training a CNN model for binary classification. The obtained confusion matrix and some corresponding performance measures on validation set are as follows:

ans =

       11046         177
         561        9306

accuracy:    0.965007
sensitivity: 0.943144
specificity: 0.984229
precision:   0.981335
recall:      0.943144
f_measure:   0.961860
gmean:       0.963467

It seems the results are good. But the training loss seems bad, as shown as follows:

training loss

Is this a good model?

Community
  • 1
mining
  • 789
  • 5
  • 11
  • 23
  • Are those performance measurements conducted on testing dataset (one that was not used for training the model, e.g. via cross-validation)..? If not, they I'd bet that you have a terribly overfitting model... – Tim Aug 03 '17 at 21:04
  • @Tim, thanks for comment. Yeah, the performance measures are based on validation set (not for training). – mining Aug 03 '17 at 21:14
  • Are spikes corresponding to new epochs? – tagoma Aug 03 '17 at 21:20
  • @edouard, thanks for comment. The x-axis is the `global_step` in tensorflow. In this case, each epoch contains `~2000` steps. – mining Aug 03 '17 at 21:23
  • doesn't it spike fairly constantly throughout the epochs? Were your data scaled ? Are data shuffled from one epoch to the next? – tagoma Aug 03 '17 at 21:28
  • 1
    Still this is unanswerable, e.g. if you classify 0's and 1's and your data consists almost only of 1's then if you predicted 1's for everything, you'd have almost 100% accuracy and your performance measures you'd be worse then such naive prediction. Those kind of measures are relative. Are those accuracy measures ok *for you*? E.g. if you have a killer robot that kills everything that is classified as non-human, then it'd be incorrectly killing 3499 out of 100 000 people -- I wouldn't let you make it work! – Tim Aug 03 '17 at 21:29
  • @edouard, yeah, training data is shuffled during training. For data scaling, I first convert the data into [0,255] using min-max normalization, then compute the mean value on training set, then minus this mean value when training or validating. I didn't divide the standard deviation. – mining Aug 03 '17 at 21:32
  • @Tim, yeah, you are right. But in my case, the number of `0` and `1` are nearly equal, the `1`s are slightly more than the `0`. – mining Aug 03 '17 at 21:33
  • 1
    @edouard, yeah, the spikes are constantly during each epoch. I doubt if the network are learning something. – mining Aug 03 '17 at 21:41

0 Answers0