10

I have a pre-post intervention study with four groups: 1) Pre-Intervention Control, 2) Pre-Intervention Treatment, 3) Post-Intervention Control, and 4) Post-Intervention Treatment. The outcome is a binary variable. There four other predictor variables. I am reanalyzing a previous study. In the previous study, they used a difference-in-differences estimator in a logistic regression, while controlling for the four predictors. With the indicators for Treatment and Time, the model is:

$$\mbox{logit}(Pr(y=1 | \mbox{Time}, \mbox{Treat}))=\alpha_0\mbox{Time}+\alpha_1\mbox{Treat}+\alpha_2\mbox{Time}\cdot\mbox{Treat}+\beta x$$

However, reviewing the literature surrounding DiD estimators, it appears that using the DiD estimators in a logit regression (any regression with a nonlinear link function), results in the common trend assumption be violated. You can think of how there can't possibly be a common trend on the predicted outcome because it has support between 0 and 1. In addition, depending on where baseline was, differences on the index value (the stuff inside the logit function), can result in different margins on the probability of y. If baseline was somewhere in the middle, small difference on the index value would drastically change predicted probability, while if baseline started high, the differences would be minimal.

So what is the practical solution here? How should I go about reanalyzing this data? What can I do, practically, to still be able to draw a causal conclusion based on the DiD estimator and the already established study design? Any ideas?

AdamO
  • 52,330
  • 5
  • 104
  • 209
Alec Y.
  • 201
  • 1
  • 2
  • 4
  • 1
    I haven't used it myself, but look into conditional logit. It's in the `survival` package in R. – generic_user Mar 10 '14 at 20:56
  • The other problem with fixed effects in nonlinear models, which is kind of what you said, is that the algebra of the ``within'' transformation breaks down. – generic_user Mar 10 '14 at 21:01

2 Answers2

9

Linear DiD Methods
You could stick with the linear probability model which you can easily estimate via least squares. Running a simple linear regression for your difference in differences analysis has several nice properties:

  • the DiD coefficient is readily interpretable (which is not necessarily true for interaction terms in nonlinear models - see Ai and Norton, 2003); non-linear methods can nonetheless identify the incremental effect of the DiD coefficient (see Puhani, 2012)
  • there are several options available for you to correct for serial correlation of the errors; Bertrand et al. (2004) discuss why this is important and offer several options on how to do it (I listed the available methods in an earlier answer)
  • the linear probability model is much faster which is particularly true if you have a large data set

Drawbacks of the linear probability model are that it is heteroscedastic by construction though this isn't much of an issue given that this is easily adjusted for. For instance, the block bootstrap adjusts for both hetereoscedasticity and autocorrelation as suggested in Bertrand et al. (2004). If you are interested in prediction, the predicted probabilities can lie outside the (0,1) range but as far as I read your question you want to know the treatment effect from the DiD estimation.

So if none of these problems are real issues for you, the linear probability model is an easy and quick solution for your estimation problem.

Non-Linear DiD Methods
There exist alternative models for non-linear DiD but none of them are straightforward. Blundell and Dias (2009) describe the popular index model under the assumption of linearity in the index. They note though that even with a very simple non-linear specification this type of DiD regression is difficult to implement. Another option is Athey and Imbens (2006) who develop a non-linear DiD estimator which allows for binary outcomes. Again the implementation is everything but easy, though for completeness I mention it here.

Intuition for Interaction Terms in Non-Linear Models
Karaca-Mandic et al. (2012) provide a discussion of the changing interpretation of interaction terms when moving from linear to non-linear models. They provide the mathematical background and support the reader's understanding with graphs and applied examples using publicly available Stata data sets. Thanks to Dimitry V. Masterov for pointing out this useful reference.

Community
  • 1
Andy
  • 18,070
  • 20
  • 77
  • 100
  • That's just a additive risk model. Nor is it estimated with ordinary least squares, as you claim, but iteratively reweighted least squares (this is just the Fisher scoring approach to estimating *any* GLM). The modeling approach doesn't address OP's question since there may be assumption violations with this model assessed the same way you would for any binary endpoint GLM: plot the fitted versus predicted risk over time. – AdamO Mar 10 '14 at 20:33
  • Yes, the content of the LPM link was not what it had claimed to be. I changed the reference to a lecture now which discusses what is actually understood as LPM model. You might want to edit your comment accordingly because the LPM model definitely addresses the OPs problem. – Andy Mar 10 '14 at 20:42
  • 1
    When you say "it is heteroscedastic by construction though this isn't much of an issue given that this is easily adjusted for", are you talking about using HAC standard errors? Does this get around the finite sample problems inherent in confidence intervals that happen when you violate distributional assumptions? – generic_user Mar 10 '14 at 20:58
  • HAC standard errors are one option, others are outlined in the Bertrand et al. paper. About the finite sample properties I would have to read it up but there are variance estimators which can deal with a small number of clusters whilst being consistent. One of them is mentioned in the previous answer I linked. – Andy Mar 10 '14 at 21:02
  • Forget clusters for a minute -- can you adjust standard errors on a linear probability model to account for the fact that your CI's are only correct asymptotically? Maybe some bootstrap? – generic_user Mar 10 '14 at 21:08
  • 1
    @Andy On the first bullet, have you seen this [Puhani (2008)](http://ftp.iza.org/dp3478.pdf) paper? – dimitriy Mar 10 '14 at 21:10
  • @ACD you can't just forget about the clusters because that's where the autocorrelation happens. I've read up on the Bertrand et al (2004) paper again and the block bootstrap seems suitable for dealing with autocorrelation and heteroscedasticity. They don't say anything about the sample size though, however I feel that this point deserves to be raised as a separate question. – Andy Mar 11 '14 at 23:14
  • @DimitriyV.Masterov Thanks for your point! I included it in the answer, the note by Puhani was quite insightful. – Andy Mar 11 '14 at 23:15
  • 1
    @Andy it is sort of a different question, in that it is really general and OP is talking about a specific application. But if I'm not mistaken, the sampling distribution of beta in a linear probability model will take a long time to converge to something roughly normal, so confidence intervals will be wrong unless sample size is really big. This should be the case if the vcv accounts for clusters or not. (Right?) – generic_user Mar 12 '14 at 01:44
  • I looked up the LPM in a few econometrics texts. None of them mention the bias of the CI and test statistics other than heteroscedasticity. The block bootstrap should alleviate the small sample problem to a certain degree but this is just my intuition, I would need to dig deeper into the material. Sorry that I couldn't provide a satisfactory reply to your comment but it surely got me thinking! – Andy Mar 12 '14 at 01:58
  • @Andy You may also want to add [Karaca-Mandic, Norton, & Dowd (2012)](http://dx.doi.org/10.1111/j.1475-6773.2011.01314.x). The graphical presentation is very clear and it covers a wider set of topics. – dimitriy Mar 14 '14 at 07:46
-1

It sounds like your concern is that of model misspecification. You're interested in determining if the intervention lead to an incremental improvement in the risk of outcome comparing treated to control over time. It sounds like, in particular, you are worried that the log-linear term for the relationship between odds of outcome comparing groups differing by 1 unit in time may not be adequate.

There are two solutions to this, but firstly note: "all models are wrong, some models are useful" - George Box. We ask: what's the risk of getting the time effect wrong? (say it's quadratic) Well, if both groups are measured consistently across time, there's actually no difference. This is the value of balanced design. Adjusting for time improves precision when there is imbalance and the specified model is correct. If you are willing to assume that the specified time effect is "close to correct" (maybe there is a weakly non-linear trend), then using robust standard errors ensures that the inference is correct on the actual intervention effect. The interpretation of the parameter is a "time averaged" effect as a consequence of that.

Another solution is to use a more granular effect of time. Rather than assuming a linear increase, you can test nested models with categorical effects of time. Assume, for instance, there are four time points: two pre-intervention, and two post-intervention. The categorical model would then be:

$$\mbox{logit} (Y | X, T) = \alpha + \beta_1 X + \gamma_1 T_2 + \gamma_2 T_3 + \gamma_3 T_4$$

against the full model $$\mbox{logit} (Y | X, T) = \alpha + \beta_1 X + \gamma_1 T_2 + \gamma_2 T_3 + \gamma_3 T_4 + \eta X T_4 $$

And the simultaneous test of all post-by-treatment parameters (one $\eta$ for each extra unit time after treatment... accounting for Hawthorne effect with $\beta_1$) will account for a categorical effect-by-time interaction.

AdamO
  • 52,330
  • 5
  • 104
  • 209