1

I am looking to understand the need and intuition for statistical power. I understand how to calculate it, but I cannot understand why we need it. Here is my current thinking:

Suppose we have a control group and an experimental group and we would like to perform a comparison of means. In this case the Null hypothesis is that the difference is 0, while the alternative is that it is non-zero. Assuming that we can invoke the central-limit theorem and that the Null hypothesis is true, we center at normal distribution with mean=0. Then given data, we can calculate the p-value using this normal distribution. Increasing the sample size (number of data points) decreases the variance of this normal distribution and thus the threshold before an effect size becomes significant decreases.

Now assuming I want a minimum effect size of 1%, why can I not choose the sample size in order to decrease the variance of the normal distribution of the Null hypothesis, such that effect sizes under 1% are not significant. Why do I need power analysis to choose the sample size? What am I misunderstanding and what does the Power analysis solve, which I cannot solve using the Null hypothesis and p-values?

phil
  • 319
  • 2
  • 11
  • 1
    How often do you want to catch that difference of interest? – Dave Mar 03 '22 at 11:10
  • 1
    Regarding your last paragraph, what you are describing at the start *is* power analysis, that is it's purpose. – user2974951 Mar 03 '22 at 11:16
  • 1
    Generally speaking the point of power analysis is to choose between *different* tests with the same size, see for example https://en.wikipedia.org/wiki/Uniformly_most_powerful_test – J. Delaney Mar 03 '22 at 11:43
  • 2
    Note that the central limit theorem is not relevant here. It is strictly a _limit_ theorem and may lead to very inaccurate calculations for finite sample sizes. Also, you would do well to study planning by precision which is a simpler idea than power but leads to not-too-different sample size requirements. See [BBR](https://hbiostat.org/bbr). – Frank Harrell Mar 03 '22 at 12:33
  • 1
    Doesn't https://stats.stackexchange.com/questions/436226 answer your question? What else are you seeking that might not be contained in the posts there? – whuber Mar 03 '22 at 13:53
  • I am not really looking for a mathematical answer on what power is or how to apply it, I am only focused on understanding why we need it? Can you offer an example of what would go wrong if I completely ignore power? – phil Mar 03 '22 at 15:04
  • @phil Think about what happens if the power if zero. – Dave Mar 03 '22 at 15:13

0 Answers0