How to test a covariate adjustment for propensity score matching?

Question

How can I test whether a covariate is sufficiently balanced after propensity score matching is made? Some authors say that standardized difference less than 0.1 is enough. Some authors say that additional tests like t-test are needed.

Which of the approaches is thought to be more preferable now?

Answer 1

Here are the best recommendations for assessing balance after matching:

• Examine standardized mean differences of continuous covariates and raw differences in proportion for categorical covariates; these should be as close to 0 as possible, but values as great as .1 are acceptable.
• Examine the same on interactions among covariates and polynomial terms of continuous covariates.
• Use the standardized mean difference of an estimated prognostic score. This should also be as close to 0 as possible.
• Examine variance ratios for each covariate; these should be as close to 1 as possible. Values between .5 and 2 have been deemed acceptable.
• Examine Kolmogorov-Smirnov statistics for each continuous covariates. There is no numerical guidance on these, but they should be as close to 0 as possible.

Some no-nos:

• Do not use p-values in or any statistic that incorporates sample size into the balance calculation
• Do not rely on the propensity score in any balance summary

For a fairly thorough review of the topic, see the documentation for the cobalt package in R.

Answer 2

An answer to this will generally depend on the number of matched subjects as do most rules of thumb which are meant for approximately 100-1000 observations. If you have significantly less subjects, testing is questionable with or without matching as your power is expectedly low and the chance for type II errors in such test very high. For significantly larger numbers of observations the chance of being 'off' by a bit more than 0.1 increases, but the tests might now be overpowered resulting in significance of 'differences' even if these are not relevant.

Whether testing is sensible or not, will also depend on the methodology of matching, whether you choose exact matching by categorical variables for which testing is less relevant or matching based on a score derived e. g. by logistic regression where your concern applies.

For all tests you make you should consider a correction for multiple testing and control of family-wise error rate. Thus, I would recommend to 'explore' the differences mainly through descriptive means such as comparisons of boxplots or violin plots to get a better impression of the distribution for not too small sample sizes (>50) and only choose to test differences, if you have a good reason to believe that these are different, as it will affect all tests performed in your whole analysis due to the necessity to control family-wise error rates for multiple tests.