2

I'm training convolutional neural network on imbalanced dataset, which has 9 classes. Number of classes in order is, 3000-500-500- ..... goes like this. Of course I'm not waiting %100 accuracy, but when I use class weight function from Scikit Learn and use it on Keras' Fit Function, it didn't get better than %60.80, even I change the weights, still same situation.

When I didn't do any class weight operation, I get %68 accuracy.

Also I also used smaller learning rates, still same.

I couldn't edit dataset because of the dimensions, (input: (5000,80,60,3), output: (5000,9)), it is hard to group this mixed dataset. Confused, why?

Mehmet Burak Sayıcı
  • 121
  • 3
  • I am assuming this is a classification problem. Please add more information about the question. It is unclear what your confusion is about? If you data is numeric look at SMOTE or other minority oversampling techniques. – knk Dec 28 '18 at 21:10
  • It's probably worth reviewing https://stats.stackexchange.com/questions/312780/why-is-accuracy-not-the-best-measure-for-assessing-classification-models/312783#312783 – Sycorax May 19 '19 at 16:49
  • I can't recall the source, but I remember the paper from last year showing that deep networks learn to ignore the weights when using weighed loss function. – Tim May 19 '19 at 17:04
  • @Tim That's a really interesting find. If you recall where you found it, I'd love to read the paper/whatever medium. – Sycorax May 20 '19 at 15:25
  • 1
    @Sycorax found it https://arxiv.org/abs/1812.03372 – Tim May 20 '19 at 16:11

3 Answers3

2

This is probably because your accuracy measures the accuracy across all of you classes equally. If you set the class weights of the most represented classes lower, this will cause those classes to be classified less accurately compared to others, and since you have more of those classes the overall accuracy goes down.

Dith
  • 21
  • 3
0

I don't know much about neural network, but if it involves a subsampling process for building a network, chances are even if you increase the weight of the less popular samples it still have small chance to be "seen" in each sampling process. I found it's better to replicate the unpopular cases to balance the training set and feed it into the algorithm.

Lily Long
  • 133
  • 6
0

If the accuracy you talked about is:

number of correct classifications / total samples

Then you are looking at the wrong metric. Indeed, one can construct a naive classifier that classifies all the samples in the most present class (without even looking at the samples) and such classifier would have very high accuracy.

If you have a class-imbalanced set, you should compute a confusion matrix (link) and measure precision, recall, f1 score by running

from sklearn.metrics import classification_report
print( classification_report(y_test_true, y_test_predicted) )
Petr Mičulek
  • 3
  • 3
Andrea Araldo
  • 101
  • 1