8

$\newcommand{\P}{\mathbb{P}}$I am concerned with observational data in which the treatment assignment can be explained exceedingly well. For example, a logistic regression of

$$\P(A =1 |X) = (1+ \exp(-(X\beta)))^{-1}$$

wehre $A$ treatment assignment and $X$ covariates has very good fit with very high test $AUC >.80$ or even $>.90$. This is good news for the accuracy of the propensity model, but it leads to propensity score estimates $$\hat{\pi} = (1+ \exp(-(X \hat{\beta})))^{-1}$$ close to $0$ or $1$. These in turn lead to large inverse probability weights $\hat{\pi}^{-1}$ and $(1-\hat{\pi})^{-1}$ used in estimators such as the inverse probability weighted estimator of expectation of outcome $Y_1$ (observation under treatment):

$$n^{-1} \sum_i \hat{\pi_i}^{-1} A_i Y_{1i}.$$

This, I suspect, turns the estimates' variances very large.

It seems like a vicious circle that very discriminative propensity score models lead to extreme weights.

My question: what are available option to make this analysis more robust? Are there alternatives to fit the propensity score model or how to deal with large weights after the model has been fit?

Chill2Macht
  • 5,639
  • 4
  • 25
  • 51
tomka
  • 5,874
  • 3
  • 30
  • 71
  • 3
    You might want to take a look the covariates carefully. You should include all of the variables that affect both (not either, but both) participation and outcomes. Including ones affected by the treatment, either ex post or ex ante in anticipation of treatment, is bad. In particular, Including instruments – variables that affect participation and not outcomes – is also a particularly bad idea. They will not help with selection bias and may worsen the support problem drastically. For example, if some people are encouraged to take up treatment, you don't want to condition on that. – dimitriy Nov 22 '16 at 18:01
  • @DimitriyV.Masterov Thanks; your last points seems interesting/relevant to my situation. So are you saying it is best not to find the best treatment assignment model (but rather the one including the predictors of outcome and assignment)? I thought the more precisely we can predict assignment, the better. – tomka Nov 22 '16 at 19:18
  • 2
    I think that is a common misconception. For example, see Battacharya and Vogt (2012) paper in *International Journal of Statistics and Economics* on the instruments point. – dimitriy Nov 22 '16 at 19:29
  • @DimitriyV.Masterov while your answer may solve the problem of small propensities in some situations, it may still be the case that the set of $X$ relating to both $Y$ and $A$ is very discriminative on $A$. I am still interested in options to deal with this problem. – tomka Nov 24 '16 at 15:28

1 Answers1

5

This is a good detect. You are referring to positivity assumption. It requires that there be both exposed and unexposed participants at every combination of the values of the observed confounder(s) in the population under study. Positivity violations occur when certain subgroups in a sample rarely or never receive some treatments of interest. There are many papers on this topic, such as Austin and Stuart (2015) and Peterson et al. (2012). You may search for more online.

user36400
  • 154
  • 5
  • 2
    Thanks, but are you sure? I am referring to very small or large weights. What you describe sounds more like overlap between the propensity distributions of treated vs. untreated units (which is apparently called positivity, I did not know). However, it seems there can be overlap (positivity) while still having extreme weights, no? – tomka Nov 22 '16 at 19:16
  • Also there can be no overlap (positivity) while having no extreme weights, I believe. – tomka Nov 22 '16 at 19:24
  • That Austin & Stuart paper discusses using stabilized weights, which may be helpful for your situation. – Noah Nov 25 '16 at 06:20
  • @Noah Saw that. It's a good starting point. Unfortunately they do not document this claim very well and the effect it has on estimates when propensities are extreme is not known. – tomka Nov 28 '16 at 11:06