7

From Wikipedia

Given a sufficiently large sample size, a statistical comparison will always show a significant difference unless the population effect size is exactly zero.

For example, a sample Pearson correlation coefficient of 0.1 is strongly statistically significant if the sample size is 1000. Reporting only the significant p-value from this analysis could be misleading if a correlation of 0.1 is too small to be of interest in a particular application.

I was wondering why "given a sufficiently large sample size, a statistical comparison will always show a significant difference unless the population effect size is exactly zero"?

Thanks and regards!

Community
  • 1
Tim
  • 1
  • 29
  • 102
  • 189
  • 2
    An editor interested in clarity would remove the implicit double negative in the first statement ("difference ... unless") by restating it more positively, like "When the population effect is nonzero, it can be detected with a sufficiently large sample." The rest is confusing because although the first statement refers to a *population* effect, the example refers to an effect *observed in a sample.* A better example would suggest that a *population* correlation coefficient of 0.1 will show up as a significantly positive correlation in a sufficiently large sample (but maybe not in a small one). – whuber May 17 '13 at 22:09
  • see also [here](http://stats.stackexchange.com/questions/2516/are-large-data-sets-inappropriate-for-hypothesis-testing/) – Glen_b Oct 26 '15 at 02:57

2 Answers2

11

With increasing sample size, the statistical power (see below) to detect even the smallest effect size is also increasing and these tiny effect sizes are then found to be statistically significant, even though they bear no relevance at all. Just as a thought experiment to illustrate it further: What if you could include all people of interest in a study. All statistics calculated from that complete "sample" would reflect the true values in the population with no errors. So if the population effect sizes are exactly 0, then, and only then you would find them to be exactly 0. Otherwise you would find some tiny differences or correlations or whatever your effect size is.

This post might also be interesting in that context.

Addition

I found this wonderful analogy of statistical power in Harvey Motulsky's Book Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking (the analogy was originally developed by John Hartung):

Suppose you send your child into your basement to fetch a tool, say a hammer. The child comes back and says, "The hammer isn't there." What is your conclusion? Is the hammer in the basement or not? We cannot be 100% sure, so the answer must be a probability. The question that you really want to answer is, "What is the probability that the hammer is in the basement?" For this question to answer, we would need a prior probability and thus, Bayesian statistics. But we can ask a different question, "If the hammer really is in the basement, what is the chance that your child would have found it?" It is immediately clear that the answer depends:

  • How long did your child spent looking? This is analogous to sample size. The longer the child keeps looking, the more likely it is that it finds the hammer. And importantly: even if the hammer is really small, if the child spent hours looking, it is likely that it finds the hammer, despite its small size. This is also true for studies: the larger the sample size, the smaller effect sizes ("hammers") can be detected.
  • How big is the hammer? This is analogous to the effect size. A sledgehammer is easier (i.e. faster) to find than a tiny hammer. A study has more power if the effect size is large.
  • How messy is the basement? It is easier to find the hammer in an organized basement than in a messy one. This is analogous to experimental scatter (variation). A study has more power when the data show little variation.

Your child has a hard time if it has to find a tiny hammer within a short time in a messy basement. On the other hand, your child has a good chance of finding if it spends a long time searching a sledgehammer in a tidy basement (so clean up your basement before sending your child looking for something!).

Community
  • 1
COOLSerdash
  • 25,317
  • 8
  • 73
  • 123
  • 1
    It's wise to be cautious about issuing extreme or universal statements, like something is "always false," because such statements are usually (always? :-) incorrect. In this case, you seem implicitly to assume that $H_0$ is simple. Composite null hypotheses frequently are true and have abundant, convincing evidence in support. But how has the conversation turned into whether the null is true, when the question concerns when it is *false*? – whuber May 17 '13 at 22:14
  • @whuber you're right, of course. I haven't thought about composite null hypothesis. I'll edit my answer accordingly (delete the first part). – COOLSerdash May 17 '13 at 22:19
  • Yep, that's the problem with universal statements: there's always some blaggard who will come along with a counterexample just to ruin your fun :-). – whuber May 17 '13 at 22:32
  • @whuber, haha, seems that way :) On the other hand: I didn't invent the universal statement. Even some [mathematicians](http://www.dur.ac.uk/r.j.coe/resmeths/critsig.htm) have it on their website. Even some [statistic books](http://books.google.ch/books?id=VtU3-y7LaLYC&pg=PA53&lpg=PA53&dq=null+hypothesis+is+always+false&source=bl&ots=cyur4yimby&sig=iHN2n_8PWbepEQmHXuOArXJeX04&hl=en&sa=X&ei=8SaXUbuRKYXT7AaL44DABg&ved=0CDoQ6AEwAg). – COOLSerdash May 18 '13 at 07:01
3

For concreteness, imagine a one sample test of means (large sample, on something where the population mean and variance exists to make the argument a little simpler).

Let the difference between the true mean and the hypothesized sample mean be any nonzero $\delta$. Then the sampling distribution of the sample mean minus the hypothesized mean will itself have mean $\delta$ and a variance that shrinks proportionally to $1/n$.

As such, as $n$ becomes sufficiently large, the probability that the test statistic will be outside the rejection region goes down.

It might help, in fact, to think of it in terms of a test based off a confidence interval. The confidence interval for the population mean will shrink in width as $\frac{1}{\sqrt n}$. As $n$ becomes sufficiently large, the typical CI pulls closer and closer to the population mean (it's still a random variable, of course), but $\delta$ remains constant.

Eventually, the half-width of the confidence interval ("margin of error") is typically much smaller than $\delta$ - making the hypothesized mean 'far' - more and more half-widths of a typical CI - from the actual population mean (resulting in a rejection probability that approaches 1)

You can construct similar arguments for almost any hypothesis test of a point null, as long as you have a few basic conditions satisfied (if you don't have consistency, the argument will fail, for example).

Glen_b
  • 257,508
  • 32
  • 553
  • 939