How do I interpret improvement in AUC ROC from the business perspective?

Question

Let's say I'm training a logistic regression model to predict the click-through-rate (CTR) on online display ads. The training dataset consists of positive examples (ads were clicked) and negative ones (ads were not clicked). The binary classifier has achieved 0.7 AUC ROC on test dataset.

Now, instead of using logistic regression, I found that gradient boosting trees performs better, yielding 0.75 AUC ROC on test dataset.

How does the improvement in AUC ROC translate to potential improvement in CTR?

Answer 1

We have no idea. And the AUROC won't help you.

A statistical model alone does not generate value. (Therefore, you cannot quantify how much more value an improvement in your model yields.)

What generates value is decisions. Better decisions generate more value. If your improved model leads to better decisions, then it generates more value. But you might also be able to generate more value using the very same model, by changing the way you base decisions on the model.

Therefore, you will need to think about and possibly simulate how your model is translated into decisions, and you will need to assign a value to each pair (decision, actual outcome). Note that maximizing AUROC may not be the output from your model you need to maximize value. More info at my answer to the earlier "Classification probability threshold".