2

What are reasons or references for a statement such as "When you have a lot of data the statistical problem you run into is that even a tiny difference will be statistically significant." I see this quite often and I need references as to why this is true. Any help is appreciated.

russellpierce
  • 17,079
  • 16
  • 67
  • 98
Walt
  • 31
  • 1
  • 1
    I edited your post to remove your signature. Your name appears to the right of your post... so you don't need to sign your questions (or answers). – russellpierce Jun 18 '14 at 20:13
  • 1
    See [Are large data sets inappropriate for hypothesis testing?](http://stats.stackexchange.com/q/2516/32036) – Nick Stauner Jun 18 '14 at 20:29

1 Answers1

1

The reason is that as sample size increases your ability to detect deviations from the null (no effect) increases [standard errors of the estimate shrink]. Therefore, at sufficiently large sample sizes, even tiny effects may become statistically significant (i.e. $p < \alpha$). It is always (or at least almost nearly always) an appropriate thing to do to look at the effect size of your statistic. The effect size will quantify how large your effect is, hopefully in terms you care about (e.g. proportion of variance accounted for or magnitude of average difference scaled by error in measurement). If you know how to interpret your effect size, then you will also know how to judge whether this is a tiny effect of no theoretical interest, or an effect large enough to be worthy of consideration.

russellpierce
  • 17,079
  • 16
  • 67
  • 98
  • Hi @rpierce by "standard errors of the estimate shrink", you mean standard error of the mean SEM? – Ladislav Naďo Jun 18 '14 at 20:14
  • 1
    @LadislavNado if the inference is on the mean then yes. But odds ratios, proportions, rates, etc. all have associated standard errors. As with mean estimation, test statistics for these tests will reach "highly improbably domains" according to the null hypothesis as sample size increases. – AdamO Jun 18 '14 at 20:17
  • 1
    @rpierce As a note, there is inconsistent usage of the terms "effect sizes" in the literature. Some take "effect size" to mean "association measure" which does not vary with sample size, and I think that was your intended usage. Others use effect size to mean test statistics which do vary with sample size. – AdamO Jun 18 '14 at 20:18
  • Oh no...who uses effect size to refer to test statistics? How could that possibly be valid usage? – Nick Stauner Jun 18 '14 at 20:24
  • @AdamO yes that was my intended usage. I didn't know there was terminology confusion there. Thanks for letting me know, egads. – russellpierce Jun 18 '14 at 22:10
  • @NickStauner I agree its an entirely incorrect usage, but I do frequently encounter it. You'd be surprised how often people confuse "precision" and "size" when it comes to estimation. Tends to be the same people who think "exact" tests mean "most powerful". For that reason, I actually use the term "association measure" for effect size which is kind of a *tabula rasa* as far as researcher predisposition. – AdamO Jun 18 '14 at 22:16