1

I have a unbalanced data set and use Cohen's kappa and AUC as performance measure.

Without down sampling the Kappa value is around 0.85, with random down sampling it is 0.95. and with a house-made focused down sampling it is approx 0.75. Which data would you use to train the classifier? The AUC for the three cases are 0.998 without downsampling, 0.998 with random downsampling and 0.921 with house-made downsampling

I'm suspicious that the random down sampling increases the kappa value.

The full data set looks like this enter image description here

When I randomly downsampl it looks like this

enter image description here

And with a house made sort of focused downsampling it looks like this

enter image description here

Matthias
  • 303
  • 1
  • 3
  • 7

1 Answers1

2

Cohen's kappa coefficient is a chance-adjusted index of agreement. It estimates the likelihood of chance agreement using Bayes' Rule: it assumes that the likelihood of raters randomly assigning an item to the same category is based on the product of raters' individual distributions for each category. In your case, the "raters" are the trusted labels and the classification predictions. By using downsampling, you are artificially modifying one of the rater's (i.e., the trusted labels') distributions. Unfortunately, this undermines the logic of the coefficient. So I would base my decision on other metrics (AUC or just accuracy) and use random or no downsampling.

Jeffrey Girard
  • 3,922
  • 1
  • 13
  • 36
  • 1
    Thank you for your answer! Yes I think random or no downsampling is the best option here. The focused undersampling is somehow biased. – Matthias Aug 19 '16 at 02:08