I am trying to understand the point of the second panel in the following xkcd comic:
Specifically, how can one be misled by controlling too many confounding variables in one's models?
Any pointers to what this criticism is called in the literature—so I can look into it further—will be appreciated.
There is no such thing as a "sweet spot" for the number of variables to control for in order to get an unbiased estimate of the causal effect. Since we are talking about confounding, we must have in mind the estimation of the causal effect of a particular variable. You use a graphic tool called the DAG to map out the causal relationships and then you condition on a set of variables that will yield you the causal effect. Conditioning on variables generally blocks the flow of association but conditioning on a collider (common effect) will induce association between variables that are not causally related. The more variables you condition on, the more likely you are to condition on a collider and thus induce association without causation; that said the more variables you condition on you are also blocking more backdoor paths, including those with colliders. The reasoning here should not revolve around "how many variables?" but around "which variables?" to condition on.
Below is an example where not conditioning on anything is what you want in order to estimate the direct causal effect of A on B. On the other hand, conditioning on the set {D} or {C,D} will bias the direct causal effect of A on B because it conditions on the collider D and opens backdoor path(s).
This post here can serve as a good introduction to causal reasoning with DAGs.
I would point out three things:
(1) Generally (related to the estimation of causal effects)
Usually you want to explain phenomena out there in the world with parsimonious models including variables deduced from some theory. You may just add any variable that comes to your mind to a regression model and end up with an almost perfect fit, but you did not learn anything about (or even fundamentally distorted) the relationship (aka causal/treatment effects) you actually interested in (also see the DAGs @ColorStatistics pointed to). (Literature e.g.: "Causal Inference in Statistics" by Judea Pearl).
(2) Specifically (more related to the overspecified model term)
You can perceive adding irrelevant variables to a regression model as estimating coefficients on irrelevant variables that are truly zero. Then if you do this the estimators of our regression coefficients are still unbiased, but also inefficient since we did not consider (true) zero restrictions on the coefficients of the irrelevant variables. Hence, inference stays valid but confidence intervals become broader. (Literature: basically any econometrics textbook, e.g. Wooldrige).
(3) Additionally (related to prediction)
If you are solely interested in prediction performance of a model based on your training data then adding 'irrelevant' variables to your model is less harmful (irrelevant in the sense of being not causal and not in the sense of having true zero restrictions on the coefficients). As the overspefication of your model only becomes problematic if you want to do inference (broader confidence intervals). (Have a look in the causal machine learning literature).
Well, it's related to the concept of p-hacking. Given a sufficient number of plausible confounding variables to be added in the study, it's possible to find a combination of them that would yield significant results (you just plug in or out until you get significant results, and you report those).
There's a very nice post in FiveThirtyEight about this so you can experience the idea, where you can even obtain contradictory results depending on which variables you wish to "correct for".
The term you are looking for is overfitting. Wikipedia has a good explanation.
There are some helpful mathsy explanations, but I thought perhaps this could use an intuitive example.
Suppose that you're investigating (perhaps for an insurance company) whether hair colour has an impact on crash risk. You look at the data, and at first pass you see that brunettes are 10% more likely to crash than blondes. But in the same data you see that brunettes are also more likely to get caught speeding. You do the controls to take out the effect of speeding, you'd find that the effect of hair on crash risk drops below signifance.
That would probably be an example of an inappropriate thing to control. It is likely that the fact that our brunettes speed more is the mechanism by which they're more likely to crash. As such, if you insist on zeroing out that mechanism, you're forcing yourself to see no effect even if it's obviously there. Intuitively, "Actually brunettes are very safe drivers considering how much they speed" is very obviously an unreasonable defence to make!
Conversely, suppose we look at the dataset again and find that people who are bald are 50% more likely to crash than those with red hair. But it also happens that the bald people in the dataset were typically older men, and younger women were underrepresented. Again you throw statistical controls at the situation and the effect disappears. This is probably a good thing to control, not least because your insurance company already asks about age and gender so you don't want to double count any effects. Again intuitively, saying "We already knew that age and gender are known to have an impact on road safety and on baldness prevalence. These data show bald young women are just as safe as hairy young women and bald old men are just as safe as hairy old men." seems like a very reasonable clarification.
(This example is entirely made up, and isn't an allegation that any particular hair type in the real world are dangerous drivers!)
