1

Suppose you have a binary random variable $Y$, and several other random variables $X_1,...,X_p$. Your goal is to "predict $Y$ using $X_1,...,X_p$." So, you go ahead and fit logistic regression, which is trying to estimate $P(Y=1 | X_1,...,X_p)$. However, when you evaluate the results of your logistic regression, you find that the AUC is terrible.

One possibility is that your model is "bad," i.e. your estimates of the coefficients for $X_1,...,X_p$ are incorrect. Perhaps you did not have enough samples, or the model was specified incorrectly etc. However, it also occurs to me that your model could be "perfect," but simply that the $X_1,...,X_p$ are not "very predictive."

By this, I mean that suppose $P(Y=1)=0.1$, but when you condition on the random variables $X_1,...,X_p$, the (true) probability does not change very much. For example, $$0.05\leq P(Y=1|X_1=x_1,...,X_p=x_p) \leq 0.2$$

for all $x_1,...,x_p$. In other words, $Y$ is almost independent of $X_1,...,X_p$. So, even if the logistic regression model perfectly learns the conditional expectation, the AUC could still be very bad!

Is there a name for this situation? And is there a way to detect it? How can I tell that my conditional expectations are "correct," even though my predictive ability is so poor? Or am I not even phrasing the problem correctly statistically?

AdamO
  • 52,330
  • 5
  • 104
  • 209
The_Anomaly
  • 420
  • 2
  • 10
  • 1
    It does not follow from $$0.05\leq P(Y=1|X_1=x_1,...,X_p=x_p) \leq 0.2$$ that the AUC must be bad. If all $Y=1$ have higher $P(Y=1|X)$ than all $Y=0$, the AUC is 1. Another way to state this is that if all the positives are ranked higher than all of the negatives, the AUC is 1. See: https://stats.stackexchange.com/questions/145566/how-to-calculate-area-under-the-curve-auc-or-the-c-statistic-by-hand – Sycorax Apr 30 '19 at 15:01
  • @Sycorax: do you want to post your comment(s) as an answer? [Better to have a short answer than no answer at all.](https://stats.meta.stackexchange.com/a/5326/1352) Anyone who has a better answer can post it. – Stephan Kolassa Apr 30 '19 at 15:08
  • 1
    It's not clear to me that my comment rises to the level of an answer because the main part of the question seems concerned with naming and resolving a certain kind of deficiency in logistic regression. If the reason is that "$Y$ can't be effectively modeled with $X$" perhaps this is a duplicate of https://stats.stackexchange.com/questions/222179/how-to-know-that-your-machine-learning-problem-is-hopeless but otherwise answers will turn on how to make a good predictive model using logistic regression (and whatever the name for this phenomenon is). – Sycorax Apr 30 '19 at 15:12
  • @Sycorax, I think you are right...my example is bad. Here is a better one, I think. Let $p=1,$ let $X$ be binary. Further, suppose that $P(Y=1|X=0) = 0.1$ and $P(Y=1|X=1)=0.2$. Now, suppose that after fitting on my training data, my logistic regression function $f(X)$ is "accurate," in that $f(0) = 0.1$ and $f(1) = 0.2$. I claim that you cannot have a perfect AUC on the test set. You would only get two possible "predictions," 0.1 and 0.2, and no threshold you set would be perfectly accurate. Clearly, not all $Y=1$ have higher $P(Y=1|X)$ than $Y=0$. – The_Anomaly Apr 30 '19 at 16:32
  • @The_Anomaly in that contrived example you are correct, an AUC is bounded by a certain theoretical value and analyses may "beat it" only by accident. It's an artifact of probabilistic teaching that we actually believe in "completely random" outcomes. Except in quantum physics, inherent randomness is not really believed to be a thing. You could predict the outcome of a coin flip knowing the torque, velocity, angle, air pressure, surface tension, etc, but instruments are too imperfect to catch all that. – AdamO Apr 30 '19 at 16:55
  • Since this Q is partially about terminology, it seems worth mentioning that the property of estimated probabilities matching true probabilities is **calibration**. So if you predict some event has probability 0.2 you'd expect the event to occur in about 20% of some series of such trials, etc if the model is well-calibrated. – Sycorax Apr 30 '19 at 18:24
  • @Sycorax, that is exactly the terminology I was looking for, thank you! – The_Anomaly May 01 '19 at 03:07
  • @AdamO, that is an interesting and helpful point, thank you. – The_Anomaly May 01 '19 at 03:08

1 Answers1

3

One must first ask if it is a "problem" at all. My viewpoint is that of a chaotician. I would look at the set of $X_1, \ldots, X_p$ and say that set of regressors is too small, and consequently my $n$ is too small too. If I needed to have an AUC of 1, I would get it by having an infinite set of regressors and of samples, and I could deterministically predict the outcome. Just because all of $X_1, \ldots, X_p$ predictors are statistically significant in the logistic model does not mean they comprise a complete set of predcitors (or even a good set at that). You likely (always) only have a subset of the ones you need, and there's always room for more.

Cancer is a good example, as there are lots of "risk models" that laudably try to predict it. Oversimplifying the background: cancers are caused by a random genetic mutation that destroys a cell's "self-destruct timer"; that mutation is caused by a complex sequence of signalling of transcriptase, reverse transcriptase, etc., moderated by the circulation, the functional status of the immune system, etc.. basically it is as close to a chaotic process as we can come. Practically, when we account for "easy to measure things" like age, smoking status, family history, etc. we are lucky to get an AUC of 0.65. Unless we are able to prize apart a strong genetic or phenotypic predictor, that is as good as we will ever do.

There are a number of scenarios to describe the above situation. One of the most commonly referred to "omitted variable bias" or, more precisely, "model misspecification" is quite accurate. There are variables, interactions, or alternative expressions that just aren't discovered yet, measured yet, or are practically able to fit with a finite sample size.

Prespecifying a target AUC is an excellent idea when setting out to make a risk prediction model. I think this is the only basis on which you can call an AUC "good" or "bad". Unrealistic expectations breed faulty data analyses. I've seen healthcare agencies go nearly bankrupt purchasing software which touts superior predictiveness to fellowship trained radiologists. When I read about the operating characteristics of these models, I found it just wasn't true. Retrospectively, the software only increased false positives and so cost more than just money, but stress, increased exposure of at-risk women to radiation via mammography, and unnecessary medical procedures (confirmational biopsy).

In prespecifying your target AUC, the reviewer of your research can check their expectations at the door: is this a clinically meaningful AUC? Is it a substantial improvement over existing technology? Is it in fact believable given other research I have encountered?

AdamO
  • 52,330
  • 5
  • 104
  • 209