2

I recently came across this paper by Sibley et al. in which propensity score matching (PSM) was applied to examine the effects of COVID-19 lockdowns on well-being and government attitudes in New Zealand. Specifically, they treat COVID lockdown as the treatment and use PSM "to match the 1,003 post-lockdown (March 26 to April 12, 2020) respondents with 1,003 from the pool of 23,351 pre-lockdown 'controls'".

Is this a justified use of propensity score matching? I've am perplexed by this specific application. I have not been able to determine exactly why this may (or may not) be an inappropriate application.

The primary issue I see with the approach is that everyone in the 'control' eventually received the treatment, with the treatment being not just "non-random" but entirely deterministic. The best predictor of being in the treatment (i.e., COVID lockdowns) group is simply being in the dataset at all. In other words, because COVID lockdown is exogenous, none of the individual differences captured in the study should be related in any way to the treatment. In turn, I struggle to see how the resulting propensity scores would comprise a valid matching criteria. Is this the only issue?

My assumption is that other matching methods may be far more appropriate here, but I am not entirely sure what these would be or mathematically why. In this example, I would have assumed a fixed-effects regression approach would have likely led to similar conclusions (albeit with likely increased model complexity).

References

Sibley, C. G., Greaves, L. M., Satherley, N., Wilson, M. S., Overall, N. C., Lee, C. H. J., Milojev, P., Bulbulia, J., Osborne, D., Milfont, T. L., Houkamau, C. A., Duck, I. M., Vickers-Jones, R., & Barlow, F. K. (2020). Effects of the COVID-19 pandemic and nationwide lockdown on trust, attitudes toward government, and well-being. The American Psychologist, 75(5), 618–630. https://doi.org/10.1037/amp0000662

Doctor David Anderson
  • 153
  • 1
  • 5

1 Answers1

3

The goal of propensity score matching (and any matching) is to create groups that are equated on a list of observed covariates to be matched on. That was their goal in this study, and matching achieved that goal, with very small imbalances remaining between the pre-lockdown and post-lockdown groups. We don't know how large the imbalances were between groups prior to the matching because the authors omitted this from the study, which is inappropriate in my opinion. If the initial imbalances were small, dropping tens of thousands of observations through the matching would have been a huge waste of data. If, for whatever reason, the initial imbalances were large, however, then the propensity score matching was beneficial. Propensity score matching is not necessarily any less appropriate than any other type of matching, since all matching methods serve the same goal. If propensity score matching had been ineffective, the authors might have tried a different method. But there generally are not scenarios where propensity score matching is inappropriate and other forms of matching are appropriate because they all attempt to do the same thing.

I'm not exactly sure what you mean by fixed effects regression. If you mean a regression of the outcome on the treatment with fixed effects for...something (i.e., a difference-in-differences analysis), this is not possible because each participant was only measured once. The groups are independent groups measured at two different times. The authors also considered a within-subjects analysis using a different survey as a supplementary analysis and used paired analyses, which are equivlanet to difference-in-differences. If you just mean a regression of the outcome on the treatment and covariates (i.e., not a multilevel model, i.e., with only fixed and not random effects), propensity score matching and covariate adjustment through regression do the same thing, which is to control for imbalances in the covariates. Regression also increases the precision of the resulting effect estimate and would have been a good idea here, but for the purposes of adjusting for pre-"treatment" differences in the covariates, they are equally valid methods and regression is not more appropriate than matching.

Noah
  • 20,638
  • 2
  • 20
  • 58
  • Thanks for the response. However, a PSM is defined as "[assigning] the probability of treatment assignment conditional on observed baseline characteristics". In practice, you regress your characteristics onto a logistic function of treatment/non-treatment. However, in this case, the treatment was an exogenous variable with literally zero association with any underlying individual characteristic in the study. On a pragmatic level, this is a severe violation of the PSM assumptions. However, I imagine there is a case to be made that it mathematically works out to be an appropriate matching method – Doctor David Anderson Jul 08 '21 at 02:18
  • Formally, Rosenbaum and Rubin (1983) specify that for treatment assignment to be ignorable, the 2nd condition is: `0 < P(Z = 1|X) < 1` (every subject has a nonzero probability to receive either treatment). This is clearly violated in this instance. Secondly, the propensity score is `ei = Pr(Zi = 1|Xi)` defines probability of treatment conditional on baseline characteristics. Due to the exogenous cause of lockdown, shouldn't the following be true `Pr(Zi = 1| Xi) = 0` ? – Doctor David Anderson Jul 08 '21 at 02:41
  • 1
    You can use propensity scores with randomized trials. See [Williamson et al. (2014)](https://doi.org/10.1002/sim.5991). If there is any imbalance in the covariates, then PSM can reduce that imbalance, regardless of whether treatment was caused specifically by those covariates. This is not at all a violation of any PSM assumptions. In a randomized trial, the $X$ needed to eliminate bias from confounding is empty. Any larger set (with more covariates) is also an allowable adjustment set. – Noah Jul 08 '21 at 03:49
  • 1
    The treatment is deterministic, but not on a variable that causes confounding (whether the survey was taken before or after the lockdown started) so that variable would not be in $X$. Positivity is only required on the set of $X$ that form a sufficient adjustment set, which in this case are the background covariates they matched on. It's not true that $P(Z=1|X)=0$ because obviously not all the propensity scores were zero and there are respondents with similar covariate profiles in both groups. – Noah Jul 08 '21 at 03:53
  • For example, let's say I flip a coin and heads means you're treated and tails means you aren't. You could say treatment is deterministic based on which side the coin lands on. All treated units will have heads, and all untreated units will have tails. Is that a violation of positivity? No, because the coin flip is not a confounder even though it's the cause of deterministic treatment assignment. There may be still random baseline differences between the two groups, and this is what PSM aims to correct. – Noah Jul 08 '21 at 03:55
  • Thanks again for response, that paper is helpful. I now see how I was confused: why a logistic model, which uses predictors $X$ that are entirely unrelated to treatment $Z$, numerically will result in appropriate propensity scores (i.e., relative to medical RCT contexts). This was caused by almost all papers referring to propensity scores as "controlling for covariates that predict treatment". But thinking through how the math works out in situations where no stable person-level factors, then I see that the similar logistic fitted values (per person) necessarily indicates similar $e_i$. – Doctor David Anderson Jul 08 '21 at 07:10