Why does preponderance of a single outcome render binary logistic regression ineffective?

Question

This question was motivated, but is separate from, the question I posted here: How can I improve the predictive power of this logistic regression model?.

In that case the 'cancer' outcome was occurring with ~92% probability. It was commented to me that "these variables don't discriminate your data very well. Since most people have cancer in this data set you can do just as well at predicting whether they have cancer by just saying they all have it." In this instance the predictor variables were poorly chosen and it may not have mattered much what proportion of people had cancer.

Thinking more generally, at what point does the preponderance of one outcome become sufficiently great that logistic regression becomes a poor choice? Are there any rules of thumb to guide judgement in this area?

Answer 1

There's an excellent answer to this exact question here, based on King & Zeng (2001) (pdf).

The gist, from that article:

The problem is that maximum likelihood estimation of the logistic model is well-known to suffer from small-sample bias. And the degree of bias is strongly dependent on the number of cases in the less frequent of the two categories. So even with a sample size of 100,000, if there are only 20 events in the sample, you may have substantial bias.

Answer 2

A common way of evaluating classification models is the F-score, where $P$ is precision and $R$ is recall: $2\frac{PR}{P+R}$. Precision is the quotient of predicted true positive prediction and all predicted positive values; recall is the quotient of true positive values and all actual positive values. In more intuitive terms, precision is the percentage of your positive predictions that were correct; recall is the percentage of positive conditions that you correctly identified. A very intuitive explanation is given in this video from Andrew Ng's machine learning course. (Highly recommended!)

In the Coursera version of that video, a quiz suggests a reasonable approach to setting the logistic regression threshold for prediction: Calculate the F-score on a cross-validation set, and choose the threshold that maximizes F. (Note that an error-free, perfect classifier would have $F = 1$.) I would advise seeing if any threshold could outperform the all-positive model, in terms of F-score. Meaning, if guessing that everyone has cancer yields a higher F, then the model needs improvement. This could mean using polynomials of the features to a draw a non-linear decision boundary, or considering another method all together.