13

Question: What is a good method for conducting post hoc tests of differences between group means after adjusting for the effect of a covariate?

Prototypical example:

  • Four groups, 30 participants per group (e.g., four different clinical psychology populations)
  • Dependent Variable is numeric (e.g., intelligence scores)
  • Covariate is numeric (e.g., index of socioeconomic status)
  • Research questions concern whether any pair of groups are significantly different on the dependent variable after controlling for the covariate

Related Questions:

  • What is the preferred method?
  • What implementations are available in R?
  • Are there any general references on how a covariate changes procedures for conducting post hoc tests?
Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
  • You might want to read here as well. http://stackoverflow.com/questions/23628323/r-post-hoc-tests-in-ancova-with-factorial-design – James Holland Apr 08 '15 at 20:07

4 Answers4

13

Multiple testing following ANCOVA, or more generally any GLM, but the comparisons now focus on the adjusted group/treatment or marginal means (i.e. what the scores would be if groups did not differ on the covariate of interest). To my knowledge, Tukey HSD and Scheffé tests are used. Both are quite conservative and will tend to bound type I error rate. The latter is preferred in case of unequal sample size in each group. I seem to remember that some people also use Sidak correction on specific contrasts (when it is of interest of course) as it is less conservative than the Bonferroni correction.

Such tests are readily available in the R multcomp package (see ?glht). The accompagnying vignette include example of use in the case of a simple linear model (section 2), but it can be extended to any other model form. Other examples can be found in the HH packages (see ?MMC). Several MCP and resampling procedures (recommended for strong inferences, but it relies on a different approach to the correction for Type I error rate inflation) are also available in the multtest package, through Bioconductor, see refs (3–4). The definitive reference to multiple comparison is the book from the same authors: Dudoit, S. and van der Laan, M.J., Multiple Testing Procedures with Applications to Genomics (Springer, 2008).

Reference 2 explained the difference between MCP in the general case (ANOVA, working with unadjusted means) vs. ANCOVA. There are also several papers that I can't remember actually, but I will look at them.

Other useful references:

  1. Westfall, P.H. (1997). Multiple Testing of General Contrasts Using Logical Contraints and Correlations. JASA 92: 299-306.
  2. Westfall, P.H. and Young, S.S. (1993) Resampling Based Multiple Testing, Examples and Methods for p-Value Adjustment. John Wiley and Sons: New York.
  3. Pollard, K.S., Dudoit, S., and van der Laan, M.J. (2004). Multiple Testing Procedures: R multtest Package and Applications to Genomics.
  4. Taylor, S.L. Lang, D.T., and Pollard, K.S. (2007). Improvements to the multiple testing package multtest. R News 7(3): 52-55.
  5. Bretz, F., Genz, A., and Hothorn, L.A. (2001). On the numerical availability of multiple comparison procedures. Biometrical Journal, 43(5): 645–656.
  6. Hothorn, T., Bretz, F., and Westfall, P. (2008). Simultaneous Inference in General Parametric Models. Department of Statistics: Technical Reports, Nr. 19.

The first two are referenced in SAS PROC related to MCP.

chl
  • 50,972
  • 18
  • 205
  • 364
3

This is an interesting question. I think that one have to be very careful with this since most of the softwares that do post hoc comparison after ANCOVAs do it BUT on non-adjusted means.

Bryan Paulson Tukey (BPT) test is recommended for pairwise comparison on ADJUSTED means, another procedure could be the conditional Tukey Kramer test.

AnastD
  • 310
  • 2
  • 9
2

Combining simple methods that you can easily access from R and general principles you could use Tukey's HSD simply enough. The error term from the ANCOVA will provide the error term for the confidence intervals.

In R code that would be...

#set up some data for an ANCOVA
n <- 30; k <- 4
y <- rnorm(n*k)
a <- factor(rep(1:k, n))
cov <- y + rnorm(n*k)

#the model
m <- aov(y ~ cov + a)

#the test
TukeyHSD(m)

(ignore the error in the result, it just means the covariate wasn't assessed, which is what you want)

That gives narrower confidence intervals than you get if you run the model without the cov... as expected.

Any post hoc technique that relies on the residuals from the model for the error variance could easily be used.

John
  • 21,167
  • 9
  • 48
  • 84
0

Why are you giving yourself so much trouble and confusing yourself?

You can consult Andy Field's Discovering Statistics Using SPSS (3rd edition) pp. 401-404.

Using contrasts function or comparing main effects option, you can easily do the post hoc on adjusted means after taking into account the covariate.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
Masood Siyyari
  • 1
  • 1