1

I understand that Fisher's LSD test is used as a post hoc analysis if the ANOVA test rejects the null hypothesis All the means $X_1, X_2,.. X_n$ taken from various samples are equal.

Also, given that if we reject null hypothesis via ANOVA, we can definitely find at least one pair of means $X_i \& X_j$ such that Fisher's LSD test is also violated. - This is what has been taught and proven in my stats course.

However, is it possible that we cannot reject the Null hypotheses via ANOVA and yet find of a pair $X_i , X_j$ such that Fisher's LSD will reject that both the samples have same mean. - I feel that this type of scenario is possible.

So, is Fisher's LSD ( for all the pairs) more stricter test compared to ANOVA for rejecting the null hypothesis?

edit: None of the post-hoc tests are guaranteed to find a significant pair even after rejecting main ANOVA. This is true with Fisher's test as described here, but also with a popular alternative like Turkey's test

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
honeybadger
  • 1,262
  • 8
  • 23
  • 1
    "ANOVA" is not failing when it doesn't reject and it's not succeding when it does; similarly with post hoc testing. I have edited your title to remove the value-judgement. Further, the hypotheses are not about sample means they're about population quantities (that *sample* means differ is a matter of inspection). There are a number of other questions on site that deal with the issue of omnibus tests and post hoc tests not always corresponding.(one rejecting where the other would not). – Glen_b Sep 02 '17 at 21:42
  • 1
    @Glen_b I don't think it's a value judgment so much as saying a significance test "succeeds" is short for saying it successfully rejects its null hypothesis, and likewise for "fails". – Kodiologist Sep 03 '17 at 00:56
  • 1
    @Kodiologist I see just such phrasing quite often and it's almost always directly intended in the ordinary literal sense as a value judgement - and then functions as quite direct impetus for p-hacking. I think such phrasing should be resisted at every turn, in order not to further encourage the ingrained focus on significance as *the* criterion for success of a piece of research, which even today is widespread across many journals - including many which quite explicitly state in their editorial policies that this is not to be the case (e.g. when reviewers ignore that and editors allow them to) – Glen_b Sep 03 '17 at 01:16
  • @Glen_b I don't like significance testing any better than you do (possibly worse: I've yet to find a problem in social science where I think a significance test is helpful at all). I guess I just don't read much into "failed" as short for "failed to reject the null hypothesis". – Kodiologist Sep 03 '17 at 05:49
  • @Kodiologist: I just wanted your opinion on this concept; *Given that if we reject null hypothesis via ANOVA, we can definitely find at least one pair of means Xi&Xj such that Fisher's LSD test is also violated.* Do you think this statement always holds true? – honeybadger Sep 03 '17 at 05:53
  • @kasa "Violated" is an odd choice of term. But yes, I think so. For a definitive answer, create a new question (or search for an existing one). – Kodiologist Sep 03 '17 at 06:50
  • @Kodiologist: I just got to discuss this with my stats professor. The answer is no. The proof is given here for LSD https://stats.stackexchange.com/questions/59910/relation-between-omnibus-test-and-multiple-comparison/301160#301160 Thank you for your time. It was really helpful. – honeybadger Sep 03 '17 at 07:17
  • @kasa You're welcome. If your question was answered to your satisfaction, you can accept an answer by clicking on the check mark under the voting arrows. – Kodiologist Sep 03 '17 at 14:26

1 Answers1

4

So, is Fisher's LSD ( for all the pairs) more stricter test compared to ANOVA for rejecting the null hypothesis?

If I understand correctly, you're asking:

Is it true that for fixed $α$, if all the population means are equal and you test every pair with Fisher's LSD (with no requirement to conduct an ANOVA first), the probability that every pairwise null will be simultaneously rejected is strictly less than $α$?

The answer is yes, provided that $n > 2$ (and assuming, as usual, that the model is correct). To see this, note that the probabilities of rejecting $μ_1 = μ_2$ and $μ_2 = μ_3$ are each $α$. Call the former event $A$ and the latter $B$. By the definition of the model, $X_1$ is sampled independently of $X_2$ and $X_3$, so $B$ doesn't guarantee $A$, so $P(A|B) < 1$. Hence $P(A \wedge B) = P(A|B)P(B) = αP(A|B) < α$.

Kodiologist
  • 19,063
  • 2
  • 36
  • 68