6

Suppose we want to explain an experiment with the following model:

$$Y \sim \beta_1 X + \beta_2R + \varepsilon$$

where $X$ are endogenous variables and $R$ is a randomized treatment level. To interpret the outcome of an OLS fit of this model as causal it seems like we want to verify different assumptions: (1) $ R \perp \varepsilon$, and (2) $ X \perp \varepsilon$. If (1) holds but (2) doesn't, can we still interpret the coefficient for $\beta_2$ as causal? What other options do we have?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
oDDsKooL
  • 1,202
  • 2
  • 14
  • 32
  • Is your equation meant to be a causal model or just a regression? Why do you worry about causality if R is actually randomized? And from where do you get the assumption that $R \perp \epsilon$? – Julian Schuessler Jun 06 '17 at 12:59
  • @JulianSchuessler it's meant to be a causal model - R is a randomization that the experimenter controls (assumed perfect as random numbers can be) – oDDsKooL Jun 12 '17 at 12:19
  • You can view it as analyzing data from a randomized controlled trial while adjusting for a (non-randomized) baseline covariate. This is a standard thing to do and one can find plenty of discussion thereof, e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2886040/ – swmo Jun 14 '17 at 20:39

1 Answers1

3

There are some things that need clarification here.

Is $Y = \beta_1X + \beta_2R + \epsilon$ a structural equation? That is, do you believe the structural relationship between the variables you listed and the outcome is truly linear?

If this is the case, that is, if you truly believe the regression represents the structural equation of the model, then the answer is trivial --- if $R\perp \epsilon$ then you can identify $\beta_2$ regardless of the relationship between $X$ and $\epsilon$ (since you randomized $R$, we also have that $R \perp X$, assuming $X$ is not a collider or mediator --- more on that below).

However, if $Y = \beta_1X + \beta_2R + \epsilon$ is not a structural equation then things are more nuanced.

First you have to define what it is that you want to estimate, since $\beta_2$ is not a structural parameter per se. Usually you want to estimate the average treatment effect (ATE).

The first thing to keep in mind is that, since you performed an experiment, you can obtain the ATE by a simple difference in means, with no need to perform a regression.

Sometimes you want to control for other factors outside the experiment though, in order to reduce the variance of your estimate. When doing regression with experiments, you can stil get a consistent estimate of ATE, even if the true relationship is not linear.

But you have to keep some things in mind. As Freedman (2008) has shown, using a finite sample potential outcomes model:

  • Regression estimates are biased (though the bias gets small with large samples);

  • The effect on asymptotic precision is not unambiguous: it may improve or make it worse, depending mainly on balance between treatment and control (if it’s not balanced, it depends on other things which are hard or impossible to measure);

  • Usual (homoskedastic) estimated standard errors can overstate precision.

However, as Lin (2013) points out, with sufficiently large samples, these problems can be fixed. OLS adjustment cannot hurt asymptotic precision when a full set of treatment-covariate interactions is used. Also, asymptotic valid confidence intervals can be obtained using heteroskedastic consistent estimators.

Another big problem is that once you start including covariates and different specifications to your model, you are doing specification searches. As soon as the researcher try different sets of covarites looking for a “preferred” specification, the nominal type I error does not hold anynore. So if you are using frequentist statistics (like p-values) to make judgements about your data you have to keep this in mind.

In short, you can perform multiple regression adjustment to your experiment but: (i) make sure you have enough samples and include full sets of interactions; (ii) use the appropriate standard errors; (iii) always show your reader the unadjusted simple difference in means, which is the more "trustable" "hands of the table" estimate.

One final note, even if your $R$ is randomized, you should be careful about what variables you are controlling for., You should not control for colliders and you should not control for mediators if you're interested in the total effect, for instance.

Carlos Cinelli
  • 10,500
  • 5
  • 42
  • 77
  • If the R variable is actually randomised , then there should not be any significant (meaningfull) difference in the estimate of $\beta_2$ regardless of whether X is included or not... – Repmat Aug 19 '17 at 19:49
  • @Repmat you can have "small sample bias" – Carlos Cinelli Aug 19 '17 at 20:07
  • Let me posed a related specific question about this model. We are interested on ATE. If we eliminate the controls ($X$) we achieve the easiest case where $\beta_2$ represent anyhow the average causal/treatment effect of $X$ on $Y$. This result holds even if the researcher know nothing about the difference between regression and structural model, or better, he never heard the concept of structural model. – markowitz Nov 16 '18 at 16:56
  • Also in this case the above result holds, or at least I think so. Considering that, until now, we say nothing about the meaning of $X$ and $Y$, we have a causal conclusion but where are the causal assumptions? – markowitz Nov 16 '18 at 16:56
  • More clearly: it is not clear to me if causal assumptions are necessary in experimental paradigm. If yes, why? – markowitz Apr 01 '19 at 12:22
  • @markowitz the technical definition of a causal assumption is a constraint on the causal model. In the case of randomization the constraint is $Y_r \perp R$, that is, the potential outcome is independent of treatment assignment. This is the assumption you make in an experiment, which amounts to saying you believe the experiment was successful. – Carlos Cinelli Apr 01 '19 at 15:21