Biased prediction (overestimation) for xgboost

Question

I run xgboost and elastic-net on the same dataset for a classification problem, say we have

y_train, y_test
y_train_xgboost_prediction, y_test_xgboost_prediction
y_train_elastic_net_prediction, y_test_elastic_net_prediction

I find xgboost gives prediction with mean a lot higher than actual label, which is

average(y_train_xgboost_prediction) > average(y_train)
average(y_test_xgboost_prediction) > average(y_test)

While elastic-net gives prediction with average similar to average of actual label, which is

average(y_train_elastic_net_prediction) ~= average(y_train)
average(y_test_elastic_net_prediction) ~= average(y_test)

Actually xgboost has AUC higher than elastic-net, which model should we prefer in this case? How do we fix the bias in xgboost prediction?

Good question. Boosting methods usual do not give well calibrated probabilistic predictions (e.g. see Niculescu-Mizil & Caruana). That said, they can be calibrated on a follow up step (e.g. through isotonic regression). Choice of the model depends on what you want to do with it/use its estimates. Do we care more about the order or the marginal probabilities of our estimates? (Probably I should write this as an answer at some point.) — usεr11852, Sep 07 '19 at 01:57
I used AUC at first. But I also want to keep the original scale of probability prediction. So I'm thinking to do some post-processing based on xgboost prediction. — Salty Gold Fish, Sep 07 '19 at 15:45
XGBoost does not offer AUC as objective/loss. AUC is just an evaluation metric. Did you use "binary:logistic" as objective? — Michael M, Sep 07 '19 at 17:33
Yes, "binary:logistic" as objective and AUC as evaluation metric — Salty Gold Fish, Sep 07 '19 at 23:39

score 6 · Accepted Answer · answered Sep 07 '19 at 12:15

What is described is not very surprising. Boosting methods usually do not give well calibrated probabilistic predictions (e.g. see Caruana et al. (2004) "Ensemble Selection from Libraries of Models", Niculescu-Mizil & Caruana (2005) "Predicting good probabilities with supervised learning"). GLM-based methods provide more consistent marginal probabilities as they have a direct probabilistic connection.

That said, probabilistic estimates can be calibrated on a follow up step (e.g. through isotonic regression or beta calibration). Please note that these methods are usually applied on a separate hold-out sample. Check the paper from Kull et al. (2017) "Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers" irrespective of using beta calibration or not, it gives a very nice modern exposition of the matter.

The choice of the model depends on what we want to do with it/use its estimates. Do we care more about the order or the marginal probabilities of our estimates? If, for example, we care about the "fairness" of our prediction, the marginal estimates are more important. On the other hand, if we want to pick the "top X more probable" items for a particular treatment, AUC are more important. Probability calibration techniques try to bridge that gap to certain extent.

Biased prediction (overestimation) for xgboost

1 Answers1

Linked