From a statistical perspective, can one infer causality using propensity scores with an observational study?

Question

Question: From the standpoint of statistician (or a practitioner), can one infer causality using propensity scores with an observational study (not an experiment)?

Please, do not want to start a flame war or a fanatical debate.

Background: Within our stat PhD program, we've only touched on causal inference through working groups and a few topic sessions. However, there are some very prominent researchers in other departments (e.g. HDFS, Sociology) who are actively using them.

I've already witnessed some pretty heated debate on this issue. It is not my intention to start one here. That said, what references have you encountered? What viewpoints do you have? For example, one argument I've heard against propensity scores as a causal inference technique is that one can never infer causality due omitted variable bias -- if you leave out something important, you break the causal chain. Is this an unresolvable problem?

Disclaimer: This question may not have a correct answer -- completely cool with clicking cw, but I'm personally very interested in the responses and would be happy with a few good references which include real-world examples.

Answer 1

At the beginning of an article aiming at promoting the use of PSs in epidemiology, Oakes and Church (1) cited Hernán and Robins's claims about confounding effect in epidemiology (2):

Can you guarantee that the results from your observational study are unaffected by unmeasured confounding? The only answer an epidemiologist can provide is ‘no’.

This is not just to say that we cannot ensure that results from observational studies are unbiased or useless (because, as @propofol said, their results can be useful for designing RCTs), but also that PSs do certainly not offer a complete solution to this problem, or at least do not necessarily yield better results than other matching or multivariate methods (see e.g. (10)).

Propensity scores (PS) are, by construction, probabilistic not causal indicators. The choice of the covariates that enter the propensity score function is a key element for ensuring its reliability, and their weakness, as has been said, mainly stands from not controlling for unobserved confounders (which is quite likely in retrospective or case-control studies). Others factors have to be considered: (a) model misspecification will impact direct effect estimates (not really more than in the OLS case, though), (b) there may be missing data at the level of the covariates, (c) PSs do not overcome synergistic effects which are know to affect causal interpretation (8,9).

As for references, I found Roger Newson's slides -- Causality, confounders, and propensity scores -- relatively well-balanced about the pros and cons of using propensity scores, with illustrations from real studies. There were also several good papers discussing the use of propensity scores in observational studies or environmental epidemiology two years ago in Statistics in Medicine, and I enclose a couple of them at the end (3-6). But I like Pearl's review (7) because it offers a larger perspective on causality issues (PSs are discussed p. 117 and 130). Obviously, you will find many more illustrations by looking at applied research. I would like to add two recent articles from William R Shadish that came across Andrew Gelman's website (11,12). The use of propensity scores is discussed, but the two papers more largely focus on causal inference in observational studies (and how it compare to randomized settings).

References

1. Oakes, J.M. and Church, T.R. (2007). Invited Commentary: Advancing Propensity Score Methods in Epidemiology. American Journal of Epidemiology, 165(10), 1119-1121.
2. Hernan M.A. and Robins J.M. (2006). Instruments for causal inference: an epidemiologist's dream? Epidemiology, 17, 360-72.
3. Rubin, D. (2007). The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine, 26, 20–36.
4. Shrier, I. (2008). Letter to the editor. Statistics in Medicine, 27, 2740–2741.
5. Pearl, J. (2009). Remarks on the method of propensity score. Statistics in Medicine, 28, 1415–1424.
6. Stuart, E.A. (2008). Developing practical recommendations for the use of propensity scores: Discussion of ‘A critical appraisal of propensity score matching in the medical literature between 1996 and 2003’ by Peter Austin. Statistics in Medicine, 27, 2062–2065.
7. Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys, 3, 96-146.
8. Oakes, J.M. and Johnson, P.J. (2006). Propensity score matching for social epidemiology. In Methods in Social Epidemiology, J.M. Oakes and S. Kaufman (Eds.), pp. 364-386. Jossez-Bass.
9. Höfler, M (2005). Causal inference based on counterfactuals. BMC Medical Research Methodology, 5, 28.
10. Winkelmayer, W.C. and Kurth, T. (2004). Propensity scores: help or hype? Nephrology Dialysis Transplantation, 19(7), 1671-1673.
11. Shadish, W.R., Clark, M.H., and Steiner, P.M. (2008). Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments. JASA, 103(484), 1334-1356.
12. Cook, T.D., Shadish, W.R., and Wong, V.C. (2008). Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons. Journal of Policy Analysis and Management, 27(4), 724–750.

Answer 2

Propensity scores are typically used in the matching literature. Propensity scores use pre-treatment covariates to estimate the probability of receiving treatment. Essentially, a regression (either just regular OLS or logit, probit, etc) is used to calculate the propensity score with treatment as your outcome and pre-treatment variables are your covariates. Once a good estimate of the propensity score is obtained, subjects with similar propensity scores, but different treatments received, are matched to one another. The treatment effect is the difference in means between these two groups.

Rosenbaum and Rubin (1983) show that matching treated and control subjects using just the propensity score is sufficient to remove all bias in the estimate of the treatment effect stemming from the observed pre-treatment covariates used to construct the score. Note that this proof requires the use of the true propensity score, rather than an estimate. The advantage of this approach is it turns a problem of matching in multiple dimensions (one for each pre-treatment covariate) into a univariate matching case---a great simplification.

Rosenbaum, Paul R. and Donald B. Rubin. 1983. "The Central Role of the Propensity Score in Observational Studies for Causal Effects." Biometrika. 70(1): 41--55.

Answer 3

Only a prospective randomized trial can determine causality. In observational studies, there will always be the chance of an unmeasured or unknown covariate which makes ascribing causality impossible.

However, observational trials can provide evidence of a strong association between x and y, and are therefore useful for hypothesis generation. These hypotheses then need to be confirmed with a randomized trial.

Answer 4

The question seems to involve two things that really ought to be considered separately. First is whether one can infer causality from an observational study, and on that you might contrast the views of, say, Pearl (2009), who argues yes so long as you can model the process properly, versus the view @propofol, who will find many allies in experimental disciplines and who may share some of the thoughts expressed in (a rather obscure but nonetheless good) essay by Gerber et al (2004). Second, assuming that you do think that causality can be inferred from observational data, you might wonder whether propensity score methods are useful in doing so. Propensity score methods include various conditioning strategies as well as inverse propensity weighting. A nice review is given by Lunceford and Davidian (2004). They have good properties but certain assumptions are required (most specifically, "conditional independence") for them to be consistent.

A little wrinkle though: propensity score matching and weighting are also used in the analysis of randomized experiments when, for example, there is an interest in computing "indirect effects" and also when there are problems of potentially non-random attrition or drop out (in which case what you have resembles an observational study).

References

Gerber A, et al. 2004. "The illusion of learning from observational research." In Shapiro I, et al, Problems and Methods in the Study of Politics, Cambridge University Press.

Lunceford JK, Davidian M. 2004. "Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study." Statistics in Medicine 23(19):2937–2960.

Pearl J. 2009. Causality (2nd Ed.), Cambridge University Press.

Answer 5

Conventional wisdom states that only randomized controlled trials ("real" experiments) can identify causality.

However, it is not as simple as that.

One reason that randomization may not be enough is that in "small" samples the law of large number is not "strong enough" to ensure that each and all differences are balanced. The question is: what is "too small" and when starts "big enough"? Saint-Mont (2015) argues here that "big enough" may well starts in the thousands (n>1000)!

After all, the point is to balance differences between groups, to control for differences. So, even in experiments, great care should be taken to balance differences between groups. According to the calculations of Saint-Mont (2015) it may well be that in smaller samples one can considerably be better off with matched (manually balanced) samples.

As to probability. Of course, probability is never able to give a conclusive answer - unless the probability is extreme (zero or one). However, in science, we found ourselves frequently confronted with situations were we are unable to provide a conclusive answer as stuff is difficult. Hence the need for probability. Probability is nothing more than a way to express our uncertainty in a statement. As such, it is similar to logic; see Briggs (2016) here.

So, probability will help us but will not give conclusive answers, no certainty. But it is of great use - to express uncertainty.

Note also that causality is not primarily a statistical question. Suppose two means differ "significantly". Does not mean the grouping variable is the cause of the difference in the measured variable? No (not necessarily). No matter which particular statistic one uses - propensity score, p-values, Bayes Factors and so on - such methods are (practically) never enough to backup causal claims.