5

I am examining three independent groups that were measured on a continuous outcome variable. I have a priori belief that the result should be Group 1 < Group 2 < Group 3. I've been told to do an ANOVA, and then to consider doing three t-tests to look at the specific differences between the groups.

Is there any reason to bother looking at the result of the ANOVA? It's been put to me that to keep the Type 1 error rate to 5% I should look at the ANOVA and if it's non-significant I should abandon the t-tests. However, someone else suggested to me that since t-tests could be statistically significant even when the omnibus ANOVA is not, and that I should therefore not bother looking at the ANOVA, and should do the individual t-tests come what may.

If the answer is "it depends", what does it depend on?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • If you perform more than one (t-) test on the same sample then you have a multiple testing problem. This inflates the type I error. You could take a look at this answer: http://stats.stackexchange.com/questions/164181/family-wise-error-boundary-does-re-using-data-sets-on-different-studies-of-inde/164232#164232 –  Aug 31 '15 at 05:05
  • @fcoppens is that really true when you have a specific prediction Group 1 < Group 2 < Group 3, like OP does? – jona Aug 31 '15 at 19:36
  • @jona: if you break that up into more than one test then you have a multiple testing problem. If you perform a ' joint' test for all inequalities at the same time with one test then not. –  Sep 01 '15 at 04:31

1 Answers1

4

When you have an apriori alternative hypothesis, you can look directly at doing a hypothesis test specifically for that, that is, an hypothesis test with increased power for the alternative you are interested in. You have an ordered alternative, so could use a test specifically constructed for that. Ordered alternatives also named monotone alternatives. There is a section about that in the old book by Miller: "Beyond ANOVA".

The likelihood ratio test for monotone alternatives was developed by Bartholomew (1959, 1961, refs in Miller), and is connected to isotone regression and the "pool adjacent violators" algorithm. The problem is that the null distribution is complicated, and needs special tables, Miller's book has references. And, there is a complete book:

There is an alternative, maybe easier to implement (following Miller). This is due to Abelson and Tukey (1963), and summarized by: If the monotone alternative is $\mu_1 \le \mu_2 \le \dots \le \mu_I$ then base the test on the contrast with coefficients $c_1, c_2, \dots, c_I$. Write $R$ for the region in parameter space satisfying $\mu_1 \le \mu_2 \le \dots \le \mu_I$ and choose a maximin approach by choosing the contrast by maximizing the minimum power over $R$. Detail in Miller, above. The paper by Abelson and Tukey: "Efficient Utilization of Non-Numerical Information in Quantitative Analysis: General Theory and the Case of Simple Order", the Annals of Mathematical Statistics, 1963, have a table of maximin contrasts, which we report a part of below: $$ \begin{array}{crrrrrrr} j & n=1 & n=2& n=3 & n=4 & n=5 & n=6& n=7&n=8 \\ \hline 1 &-.707&-.816&-.866&-.894&-.913&-.926&-.935 \\ 2 & .707&.000&-.134&-.201&-.242&-.269&-.289 \\ 3 & & .816& .134&.000&-.070&-.114&-.144& \\ 4 & & & .866&.201&.070& .000&-.045 \\ 5 & & & &.894&.242&.114&.045 \\ 6 & & & & &.913&.269&.144 \\ 7 & & & & & &.926&.289 \\ 8 & & & & & & &.935 \\ \hline \end{array} $$

By using such an approach, there is only one hypothesis test and obviously no multiplicity problems!

A formula for the contrasts in the simple order case $\mu_1 \le \dots \le \mu_n$ is given by: $$ c_j = \{(j-1)[1-((j-1)/n)]\}^{1/2} - \{j(1-j/n)\}^{1/2} $$ and then one can simply use a $t$-test with this contrast. This do not need any special code (or package) in R.

There do seem to exist some implementations in R for tests for ordered alternatives, not the ones I wrote about, but a nonparametric alternative, the Jonckheere-Terpstra test. See https://en.wikipedia.org/wiki/Jonckheere's_trend_test . I will show an implementation in CRAN package clinfun, code below:

    library(clinfun) ### From CRAN
    set.seed(7*11*13)    ### for reproducibility
    mu  <-  c(10,  11,  12)
    g  <-   length(mu)
    n  <-   10   ### number of obs in each group
    gg  <-  list()
    for (j in seq(along=mu)) gg[[j]] <- rnorm(n, mu[j],  sd=1.0)
    nn  <-  sapply(gg,  length)
    ind  <- rep(1:g,  nn)
    xx  <-  unlist(gg)
    test.ordered  <-  jonckheere.test(xx,  ind,  alternative="increasing")

Output when run is:

> test.ordered

        Jonckheere-Terpstra test
    
    data:  
    JT = 219, p-value = 0.00407
    alternative hypothesis: increasing

Other implementations in coin, npsm and the package lawstat has analogous tests for equality of variances against ordered alternatives. I will come back extending this when I get hold of the referenced papers!

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467