52

I admit I'm relatively new to propensity scores and causal analysis.

One thing that's not obvious to me as a newcomer is how the "balancing" using propensity scores is mathematically different from what happens when we add covariates in a regression? What's different about the operation, and why is it (or is it) better than adding subpopulation covariates in a regression?

I've seen some studies that do an empirical comparison of the methods, but I haven't seen a good discussion relating the mathematical properties of the two methods and why PSM lends itself to causal interpretations while including regression covariates does not. There also seems to be a lot of confusion and controversy in this field, which makes things even more difficult to pick up.

Any thoughts on this or any pointers to good resources/papers to better understand the distinction? (I'm slowly making my way through Judea Pearl's causality book, so no need to point me to that)

Frank Barry
  • 671
  • 1
  • 7
  • 5
  • 4
    Recommend you read [Morgan and Winship, 2007](http://www.amazon.com/Counterfactuals-Causal-Inference-Principles-Analytical/dp/0521671930). Chapters 4 and 5 do an explicit compare and contrast of regression and matching for causal effect identification. – conjugateprior Apr 09 '12 at 19:36
  • 2
    When you check the balance statistics, you are ensuring that there is not extrapolation between the treatment groups you are comparing with respect to multidimensional covariate space. Regression simply extrapolates without checking for this, so extrapolations can give poor predictions. – StatsStudent Apr 21 '16 at 06:26

5 Answers5

22

One big difference is that regression "controls for" those characteristics in a linear fashion. Matching by propensity scores eliminates the linearity assumption, but, as some observations may not be matched, you may not be able to say anything about certain groups.

For example, if you are studying a worker training program, you may have all the enrollees be men, but the control, non-participant population be composed of men and women. Using regression, you could regress, income, say, on a participation indicator variable and a male indicator. You would use all your data and could estimate the income of a female had she participated in the program.

If you were doing matching, you could only match men to men. As a result, you wouldn't be using any women in your analysis and your results wouldn't pertain to them.

Regression can extrapolate using the linearity assumption, but matching cannot. All the other assumptions are essentially the same between regression and matching. The benefit of matching over regression is that it is non-parametric (except you do have to assume that you have the right propensity score, if that is how you are doing your matching).

For more discussion, see my page here for a course that was heavily focused on matching methods. See especially Causal Effects Estimation Strategy Assumptions.

Also, be sure to check out the Rosenbaum and Rubin (1983) article that outlines propensity score matching.

Lastly, matching has come a long way since 1983. Check out Jas Sekhon's webpage to learn about his genetic matching algorithm.

The_Anomaly
  • 420
  • 2
  • 10
Charlie
  • 13,124
  • 5
  • 38
  • 68
  • 3
    Maybe this is because I am not a statistician, but when it seems you assumed linear regression when the OP asked about regression in general. But I guess the gist is that adding covariates to any kind of regressor makes some assumptions about the input space so it can extrapolate to new examples, and matching is more cautious about what kind of things can be extrapolated. – rrenaud Apr 09 '12 at 19:37
  • 3
    You do make some assumptions about the functional form of the confounding variables when you estimate the propensity function. You also subsequently match on individuals who have "close" values of the propensity, so I wouldn't immediately assume that propensity matching solves the problem of nonlinear confounder effects. – AdamO Aug 07 '16 at 16:55
  • 3
    The links are broken. – Carlos Cinelli Aug 19 '17 at 06:39
18

The short answer is that propensity scores are not any better than the equivalent ANCOVA model, particularly with regard to causal interpretation.

Propensity scores are best understood as a data reduction method. They are an effective means to reduce many covariates into a single score that can be used to adjust an effect of interest for a set of variables. In doing so, you save degrees of freedom by adjusting for a single propensity score rather than multiple covariates. This presents a statistical advantage, certainly, but nothing more.

One question which may arise when using regression adjustment with propensity scores is whether there is any gain in using the propensity score rather than performing a regression adjustment with all of the covariates used to estimate the propensity score included in the model. Rosenbaum and Rubin showed that the "point estimate of the treatment effect from an analysis of covariance adjustment for multivariate X is equal to the estimate obtained from a univariate covariance adjustment for the sample linear discriminant based on X, whenever the same sample covariance matrix is used for both the covariance adjustment and the discriminant analysis". Thus, the results from both methods should lead to the same conclusions. However, one advantage to performing the two-step procedure is that one can fit a very complicated propensity score model with interactions and higher order terms first. Since the goal of this propensity score model is to obtain the best estimated probability of treatment assignment, one is not concerned with over-parameterizing this model.

From:

PROPENSITY SCORE METHODS FOR BIAS REDUCTION IN THE COMPARISON OF A TREATMENT TO A NON-RANDOMIZED CONTROL GROUP

D'Agostino (quoting Rosenbaum and Rubin)

D’agostino, R.B. 1998. Propensity score matching for bias reduction in the comparison of a treatment to a non-randomized control group. Statistical Medicine 17: 2265–2281.

Brett
  • 5,708
  • 3
  • 29
  • 41
  • 5
    (+1) There was also an interesting thread about the causality issue in this related question, [From a statistical perspective, can one infer causality using propensity scores with an observational study?](http://stats.stackexchange.com/questions/3400/from-a-statistical-perspective-can-one-infer-causality-using-propensity-scores-w). – chl Mar 22 '11 at 07:36
  • 3
    I agree with the general premise of this answer, but when one matches based on the propensity scores it is not the same as plopping all the covariates into the model (and hence is not just a dimension reduction technique). It is not the same if one weights by propensity scores either. – Andy W Mar 22 '11 at 12:46
  • @ Andy. Indeed. That is an important point and my answer referred to propensity score adjustment. Stratifying and matching will produce different results that the ANCOVA model and can have some undesirable properties (which you hint at in your reply below). – Brett Mar 22 '11 at 14:02
  • @Andy. I just noticed that last sentence. Adjusting for a propensity score and ANCOVA models produce equivalent estimates of effects. I've read results of simulations that show this clearly. I agree, matching and stratifying are different animals, but propensity score adjustment and the equivalent ANCOVA will get you the same effect estimate. You get more power via the reduction of covariates, but that's it. – Brett Mar 22 '11 at 14:05
  • I don't know the correct lingo, but if when you say adjustment you mean including the propensity score on the RHS of a regression equation then yes I agree that it is equivalent to the ANCOVA solution. I'm not quite sure what undesirable properties I hinted at, although they definitely exist (that may be getting off topic though for this question). – Andy W Mar 22 '11 at 14:31
  • 1
    I disagree with this answer. Estimated propensity scores are good when they balance covariates in treatment and control groups and bad when don't. Just the same as for a regression conditioning approach. Whether they are 'better' depends only on that property, which will vary from problem to problem. – conjugateprior Apr 09 '12 at 19:28
  • @Conjugate Prior I'm not certain what you are disagreeing with as I stated "The short answer is that propensity scores are not any better than the equivalent ANCOVA model". In other words, it is not different than covariate adjustment. The only advantage outside of that is the gain in degrees of freedom--instead of controlling for n covariates, you can control for 1 propensity score. – Brett Apr 09 '12 at 20:34
  • 1
    I'm disagreeing because while the criterion, balance, is the same the two strategies are different, as are their strengths and weaknesses. One may or may not be a better approach, depending on the problem. Indeed, the '*equivalent* ANCOVA model' seems to me to be not well-defined. (Equivalent how?) – conjugateprior Apr 09 '12 at 21:28
  • See updated quote. – Brett Apr 09 '12 at 21:35
  • 1
    Right. I now see what 'equivalent' meant, but the sentence that starts 'However' in your quote introduces the relevant difference: in practice prop. scores are estimated separately precisely so they can be *way* more gnarly than the analysis model. (And there's another difference in the article's following paragraph, not quoted.) – conjugateprior Apr 09 '12 at 23:10
7

A likely obtuse reference, but if you by chance have access to it I would recommend reading this book chapter (Apel and Sweeten, 2010). It is aimed at social scientists and so perhaps not as mathematically rigorous as you seem to want, but it should go into enough depth to be more than a satisfactory answer to your question.

There are a few different ways people treat propensity scores that can result in different conclusions from simply including covariates in a regression model. When one matches scores one does not necessarily have common support for all observations (i.e. one has some observations that appear to never have the chance to be in the treatment group, and some that are always in the treatment group). Also one can weight observations in various ways that can result in different conclusions.

In addition to the answers here I would also suggest you check out the answers to the question chl cited. There is more substance behind propensity scores than simply a statistical trick to achieve covariate balance. It you read and understand the highly cited articles by Rosenbaum and Rubin it will be more clear why the approach is different than simply adding in covariates in a regression model. I think a more satisfactory answer to your question is not necessarily in the math behind propensity scores but in their logic.

Community
  • 1
Andy W
  • 15,245
  • 8
  • 69
  • 191
  • @Andy W See the quote from Rosenbaum and Rubin on the equivalence of regression with covariates and propensity score adjustment in my updated post. – Brett Apr 09 '12 at 21:39
0

I like to think of PS as a design portion of the study which completely separated from the analysis. That is, you might want to think in terms of design (PS) and analysis (regression etc...). Also, PS porvides a mean of supporting exchangeability for binary treatment; maybe others can comment on whether including the covariates in the outcome model can actaully support exchangebility, or whether one assume exchangeability prior to including the covariates in the outcome model.

Galois Theory
  • 282
  • 1
  • 7
-3

Stat Methods Med Res. 2016 Apr 19.

An evaluation of bias in propensity score-adjusted non-linear regression models.

Propensity score methods are commonly used to adjust for observed confounding when estimating the conditional treatment effect in observational studies. One popular method, covariate adjustment of the propensity score in a regression model, has been empirically shown to be biased in non-linear models. However, no compelling underlying theoretical reason has been presented. We propose a new framework to investigate bias and consistency of propensity score-adjusted treatment effects in non-linear models that uses a simple geometric approach to forge a link between the consistency of the propensity score estimator and the collapsibility of non-linear models. Under this framework, we demonstrate that adjustment of the propensity score in an outcome model results in the decomposition of observed covariates into the propensity score and a remainder term. Omission of this remainder term from a non-collapsible regression model leads to biased estimates of the conditional odds ratio and conditional hazard ratio, but not for the conditional rate ratio. We further show, via simulation studies, that the bias in these propensity score-adjusted estimators increases with larger treatment effect size, larger covariate effects, and increasing dissimilarity between the coefficients of the covariates in the treatment model versus the outcome model.

dt2016
  • 1