17

Kaggle's competition Porto Seguro's Safe Driver Prediction uses Normalized Gini Score as evaluation metric and this got me curious about the reasons for this choice. What are the advantages of using normalized gini score instead of the most usual metrics, like AUC, for evaluation?

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
xboard
  • 1,008
  • 11
  • 17
  • 1
    The Kaggle website used to have this answer: *"There is a maximum achievable area for a "perfect" model since not all of the positive examples occur immediately. We use the normalized Gini coefficient by dividing the Gini coefficient of your model by the Gini coefficient of the perfect model."* but it is not available anymore. https://webcache.googleusercontent.com/search?q=cache:6dKyKcZJS-0J:https://www.kaggle.com/wiki/Gini – Sextus Empiricus Oct 10 '17 at 01:01
  • 1
    So, gini is just auc on a different scale. Or are auc and gini applied to different curves? That is not clear to me as non-expert in machine-learning. The question is not very clear about this. – Sextus Empiricus Oct 10 '17 at 01:10

2 Answers2

5

I believe that the Gini score is merely a reformulation of the AUC: $$gini = 2 \times AUC -1$$ As for why use this instead of the commonly used AUC, the only reason I can think of is that a random prediction will yield a Gini score of 0 as opposed to the AUC which will be 0.5.

Miguel
  • 1,323
  • 8
  • 22
  • 7
    Besides that using the gini coefficient sets the performance of a random classifier to a score of 0... the normalization "improves" the other end of the scale and makes that the score of a perfect classifier is equal to 1 rather than a maximum achievable AUC<1. *The improvement being only relative depending on whether you think a more intuitive scale is good or not. Although beyond this easier interpretation you might argue that it (the normalization) also improves generalization and comparison of different data-sets.* – Sextus Empiricus Oct 10 '17 at 21:16
  • Why should macimum achievable AUC be less then 1, also I don't see how is gini setting it to 1? – rep_ho Oct 11 '17 at 14:21
  • It depends on what kind of curve they calculate the gini coefficient. Maybe they use something different than a ROC curve (whose max AUC would indeed be 1). Given the words on the kaggle website it seems plausible that the max AUC is not 1: > *" We then move from left to right, asking "In the leftmost x% of the data, how much of the cumulative response have you accumulated?"* – Sextus Empiricus Oct 12 '17 at 15:44
  • example: https://www.bayesserver.com/docs/charts/lift-chart – Sextus Empiricus Oct 12 '17 at 15:58
3

The relationship only holds for classification tasks because obviously, you cannot calculate AUC for a continuous target. However, we also use normalized Gini in regression tasks, like predicting insurance losses.

Nick Corona
  • 117
  • 6
  • When you do a grid search is the scoring metric still roc-auc then? – mathella Nov 10 '20 at 08:23
  • If it's a classification model, then yes. If its regression, AUC is not defined, so you must use Gini. Gini is usually not a defined metric in most machine learning packages, so you'll have to create your own code. – Nick Corona Nov 11 '20 at 18:58
  • 1
    It may not be terribly meaningful to refer to the tradeoff between sensitivity & specificity with a continuous outcome variable, but the AUC is equal to the concordance probability, which can certainly be calculated for a model predicting a continuous target (see: [How to calculate Area Under the Curve (AUC), or the c-statistic, by hand](https://stats.stackexchange.com/q/145566/)). – gung - Reinstate Monica Jan 08 '21 at 18:35