0

If you do a statistical test that compares groups and find a significant difference between groups, can you then conclude that there is a difference between the "lowest" group and the "highest" group, without doing post-hoc tests?

I read on this site that an Anova can give a significant result when none of the post-hoc tests do. This answer and its comments explained that:

the data can reveal that the true means likely differ but it cannot identify with sufficient confidence which pairs of means differ.

That gave me the impression that it's not that the fact that Anova finds a difference and post-hoc tests do not means that there is no difference between the groups, but that the post-hoc tests just cannot identify this difference.

A comment to that answer has an example in R with a significant Anova result and non-significant Tukey-HSD post-hoc results. That confirmed this impression.

The group means and post-hoc results from that example are:

Group   mean
1       0.14
2       3.04
3       3.02

Tukey HSD results
2-1  0.051
3-1  0.053 
3-2  0.999

It seemed to me that, if I had not done a post hoc test and concluded, based on the significant Anova and the group means, that at least group 1 and 2 differ, that would be a reasonable conclusion. The fact that the post-hoc test is not significant at a significance level of 0.05 would just mean that the test lacks power to detect the difference, not that the difference doesn't exist.

But then I read this answer that says that it's possible that there is variance between groups whithout any pair of groups being significantly different:

ANOVA tests for significant variance among the set of groups. Significant variance can be present without any pair of groups being significantly different.

The answers that I found about this are about Anova, but my question is not specific to Anova. My question is: if I have a significant test result for a test with three or more groups, can I then conclude that the lowest and highest scoring groups differ, without even doing post-hoc tests? If not: how is that possible? How can there be significant variance between groups when none of the groups differ from each other?

broccoli
  • 63
  • 5
  • Nope! Those groups can be sparsely populated and have very wide confidence intervals. Or there can even be a non-linear trend so that the lowest to highest difference is 0 but other groups have non-zero differences. – AdamO May 24 '18 at 12:58
  • You have done 3 post-hoc tests, and none were significant. But there are many other possible post-hoc tests. The ANOVA tests for variance among the set of groups. What is your result for testing group 1 vs (group2 and group 3)? – timbp May 24 '18 at 13:08
  • @AdamO Do you have an example, or a reference for that? If those confidence intervals are super wide, so that indeed there is no difference between groups, how can the Anova give a significant result? – broccoli May 24 '18 at 15:40
  • @timbp Is it allowed to do post-hoc tests on different groups than the Anova? How do you calculate those? For Tukey's HSD the MSE from the Anova is used and the dof you use to compare with the studentized range table depends on the number of groups. If you merge groups, those values change, so I thought you have to do the post-hoc tests with the groups that are in the Anova analysis. I also still don't understand how there can be no difference between 1 and 2, but a difference between 1 and (2 and 3). I do see how adding group 3 might add power to detect a difference. – broccoli May 24 '18 at 15:44
  • 1
    @broccoli does my answer conceptually not make sense? If so examples won't really help. You can read the help file for `?TukeyHSD`. Basically, the confidence intervals (in Tukey HSD) are based on the residual variance but also the sample size in each group. This is the idea of generating intervals whose coverage are valid for individual comparisons at the cost of increased family-wise coverage. If for any comparison the sample size is small, the resulting CI is wide. – AdamO May 24 '18 at 15:54
  • @AdamO Yes, sorry, I worded my comment wrong. I know how to calculate TukeyHSD values and understand why they might all be insignificant when the Anova is significant. My question is about the conclusions to draw from that. In the examples I have seen, it just seems lack of power of the post-hoc test to detect a difference that probably is there. If a significant Anova does not mean that the group means are significantly different, what does it mean? – broccoli May 24 '18 at 18:54
  • @broccoli [is this post possibly related](https://stats.stackexchange.com/questions/347009/differing-p-values-caranova-anova-post-hoc-in-logistic-regression/347879#347879)? In my answer, I tried to slough off the nuances of logistic regression and separation: it is essentially the question you ask here. – AdamO May 24 '18 at 20:11

0 Answers0