11

Miller and Chapman (2001) argue that it is absolutely inappropriate to control for non-independent covariates that are related to both the independent and dependent variables in an observational (non-randomized) study - even though this is routinely done in the social sciences. How problematic is it to do so? How is the best way to deal with this problem? If you routinely control for non-independent covariates in an observational study in your own research, how do you justify it? Finally, is this a fight worth picking when arguing methodology with ones colleagues (i.e., does it really matter)?

Thanks

Miller, G. A., & Chapman, J. P. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110, 40-48. - http://mres.gmu.edu/pmwiki/uploads/Main/ancova.pdf

Patrick
  • 723
  • 1
  • 8
  • 12
  • The following question on ["intuitive explanation of multicollinearity"](http://stats.stackexchange.com/questions/1149/is-there-an-intuitive-explanation-why-multicollinearity-is-a-problem-in-linear-re) may be of some help in the above context. –  Aug 22 '10 at 01:30

6 Answers6

4

It is as problematic as the degree of correlation.

The irony is that you wouldn't bother controlling if there weren't some expected correlation with one of the variables. And, if you expect your independent variable to affect your dependent then it's necessarily somewhat correlated with both. However, if it's highly correlated them perhaps you shouldn't be controlling for it since it's tantamount to controlling out the actual independent or dependent variable.

John
  • 21,167
  • 9
  • 48
  • 84
  • I know this is an old answer, but do you have some references going into more detail wrt. your first line, especially ones that discuss this with explicit reference to Miller & Chapman? – jona Dec 21 '15 at 16:44
4

In the social sciences, we often call this issue "post treatment bias." If you are considering the effect of some treatment (your independent variable), including variables that arise after treatment (in a causal sense), then your estimate of the treatment effect can be biased. If you include these variables, then you are, in some sense, controlling for the impact of treatment. If treatment T causes outcome Y and other variable A and A causes Y, then controlling for A ignores the impact that T has on Y via A. This bias can be positive or negative.

In the social sciences, this can be especially difficult because A might cause T, which feeds back on A, and A and T both cause Y. For example, high GDP can lead to high levels of democratization (our treatment), which leads to higher GDP, and higher GDP and higher democratization both lead to less government corruption, say. Since GDP causes democratization, if we don't control for it, then we have an endogeneity issue or "omitted variables bias." But if we do control for GDP, we have post treatment bias. Other than use randomized trials when we can, there is little else that we can do to steer our ship between Scylla and Charybdis. Gary King talks about these issues as his nomination for Harvard's "Hardest Unsolved Problems in the Social Sciences" initiative here.

Charlie
  • 13,124
  • 5
  • 38
  • 68
3

As I see it, there are two basic problems with observational studies that "control for" a number of independent variables. 1) You have the problem of missing explanatory variables and thus model misspecification. 2) You have the problem of multiple correlated independent variables--a problem that does not exist in (well) designed experiments--and the fact that regression coefficients and ANCOVA tests of covariates are based on partials, making them difficult to interpret. The first is intrinsic to the nature of observational research and is addressed in scientific context and the process of competitive elaboration. The latter is an issue of education and relies on a clear understanding of regression and ANCOVA models and exactly what those coefficients represent.

With respect to the first issue, it is easy enough to demonstrate that if all of the influences on some dependent variable are known and included in a model, statistical methods of control are effective and produce good predictions and estimates of effects for individual variables. The problem in the "soft sciences" is that all of the relevant influences are rarely included or even known and thus the models are poorly specified and difficult to interpret. Yet, many worthwhile problems exist in these domains. The answeres simply lack certainty. The beauty of the scientific process is that it is self corrective and models are questioned, elaborated, and refined. The alternative is to suggest that we cannot investigate these issues scientifically when we can't design experiments.

The second issue is a technical issue in the nature of ANCOVA and regression models. Analysts need to be clear about what these coefficients and tests represent. Correlations among the independent variables influence regression coefficients and ANCOVA tests. They are tests of partials. These models take out the variance in a given independent variable and the dependent variable that are associated with all of the other variables in the model and then examine the relationship in those residuals. As a result, the individual coefficients and tests are very difficult to interpret outside of the context of a clear conceptual understanding of the entire set of variables included and their interrelationships. This, however, produces NO problems for prediction--just be cautious about interpreting specific tests and coefficients.

A side note: The latter issue is related to a problem discussed previously in this forum on the reversing of regression signs--e.g., from negative to positive--when other predictors are introduced into a model. In the presence of correlated predictors and without a clear understanding of the multiple and complex relationships among the entire set of predictors, there is no reason to EXPECT a (by nature partial) regression coefficient to have a particular sign. When there is strong theory and a clear understanding of those interrelationships, such sign "reversals" can be enlightening and theoretically useful. Though, given the complexity of many social science problems sufficient understanding would not be common, I would expect.

Disclaimer: I'm a sociologist and public policy analyst by training.

Brett
  • 5,708
  • 3
  • 29
  • 41
2

I read the first page of their paper and so I may have misunderstood their point but it seems to me that they are basically discussing the problem of including multi-collinear independent variables in the analysis. The example they take of age and grade illustrates this idea as they state that:

Age is so intimately associated with grade in school that removal of variance in basketball ability associated with age would remove considerable (perhaps nearly all) variance in basketball ability associated with grade

ANCOVA is linear regression with the levels represented as dummy variables and the covariates also appearing as independent variables in the regression equation. Thus, unless I have misunderstood their point (which is quite possible as I have not read their paper completely) it seems they are saying 'do not include dependent covariates' which is equivalent to stating avoid multi-collinear variables.

  • Their argument concerns not correlated variables per se, but variables that are virtually inseparable from one another. Variables for which one might almost say "this is meaningless without that." Rather than degree of correlation, which can be assessed statistically, the issue is one to be worked out conceptually. Can grade increase without an increase in age? Hardly. Can depression intensify without an increase in anxiety? That's a harder one. – rolando2 Dec 07 '15 at 02:22
1

The (biggest) problem is that because the group variable(s) and the covariate(s) are together on the predictor side of the equation, the group variable(s) is(are) no longer the group variable(s), they are those variables with the covariate partialed out, so are no longer recognizable or interpretable as the group variables that you thought you were studying. Huge problem.

The key line is on page 45 "ANCOVA removes meaningful variance from "Group", leaving an uncharacterized, vestigal residual Group variable with an uncertain relationship to the construct that Group represented".

My current solution is to partial the covariate out of the DV, and then submit the DV residual to a regular ANOVA, as an alternative to using ANCOVA.

Todd
  • 11
  • 1
0

Some of the matching tools developed by Gary King and colleagues look promising:

Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250