Why do I need statistical power for AB testing if my results are significant?

Question

I have been told that I need both significance and power for my AB results to be valid. I researched a lot for this and the above statement is not making sense. I get that we need high enough power to not reject the null hypothesis and assuming that the new feature has bought no actual effect, but why do we need power to reject the null hypothesis when my confidence interval is already so high?

My confusion is as below:

1. Power is (1-Beta). So higher the power, lower the probability of type 2 error (not rejecting the null hypothesis when it is false). The thing is, I am rejecting the null hypothesis as my results are very significant and alpha is already low.

2. Lower the alpha, more the sample size required at the same power: This further adds to my belief that you don't need statistical power to reject the null hypothesis. I mean, are we really saying the more my confidence interval, the more data size i will need to validate the effect?

I am not sure if I am missing some key concept. Please help me out as I am pretty sure that the new feature has positive conversion and I have already reached 99.99% CI.

Answer 1

Power is generally something you calculate before you perform a study. For example, let's say you are trying to test whether medication A is more effective than medication B. Because of some cost, each new participant is really expensive. So you calculate the minimum effect size you want to be able to detect (e.g. it lowers blood pressure by 10 points) and then determine from that information what sample size you would need to detect a 10 point difference in treatment. Let's say the power analysis says you need 40 participants.

Now let's say that the actual difference between treatment A and B is much larger than you minimum--- say 30 points. You would be able to detect this difference with a much smaller sample size. The point of your power analysis is to set a minimum effect size you qualitatively feel you need to detect.

So, power analysis isn't something you really ever do after a study, especially if your results are significant. If your results are significant, they're significant. No strings attached (well, at least related to power).

Answer 2

You are absolutely, exactly, completely right. This precise argument has been published by Hoenig & Heisey, "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis" (2001, The American Statistician).

Actually, they frame it the other way around: people often use "post hoc power" after finding no significant effect, and this "power calculation" "shows" that their study was underpowered to find the effect size they did find. But of course, in a precisely analogous way to yours, that is just a reformulation of the fact that a p value larger than 0.05 is logically equivalent to power that is too low to detect the observed effect at $\alpha=0.05$.

Answer 3

In addition to its use in deciding on a required sample size before a study (described in Tanner Phillips' excellent answer), there's another reason to care about statistical power: low statistical power can be a sign of the file drawer problem.

It is true that if you run a single study and get a significant result, then the statistical power of your design is at this point irrelevant. It's a calculation of how likely something that already happened was to happen, which isn't really useful information to you after your study is done.

However, there's another way to end up with a significant result in a study despite low power: Run lots of trials (or use lots of different dependent variables, or analyze your data lots of different ways, use your imagination), each of which is poorly powered to detect an effect and probably won't work, and then publish whichever one turns out significant by chance.

Thus, when a reader of a paper notices that the study design described therein is not sufficiently powered to reliably detect typical effect sizes for its domain, they have to decide which is more likely:

• The study authors had a theoretical reason to expect the effect size to be larger than is typical for their domain, and they turned out to be right.
• The study authors are engaging in some p-hacking.

We would all like to live in a world where the former was more common, but many scientific fields that rely most heavily on inferential statistics are currently in the middle of reckoning with the frequency of the latter.

This argument has been made most notably by John Ioannidis in his paper Why Most Published Research Findings are False.

Answer 4

Let us consider a test of whether $\mu = 0$ or $\mu \neq 0$. Well, let's measure $\mu$! Alas, there is always statistical variation in the outcome of a measurement. Let's call the scale of the noise $\Delta\mu$.

If your measurement was low-powered, it means that the anticipated effect size, $\mu^\star$, wasn't much bigger than the level of noise $\Delta\mu$. Thus, we should be worried if we appeared to able to significantly distinguish a new effect of size $\mu^\star$ from noise.

Slightly more formally, if the study is low-powered, whilst a significant result is rare under $H_0$ (the rate given by definition by $\alpha$), it is also rare under the anticipated effect size under $H_1$ (the rate given by definition by power)! So what can we really conclude? These kinds of considerations led Birnbaum to propose a measure of evidence against the null of the ratio, $$ \frac{\text{power}}{\alpha} $$ such that low-power implies weaker evidence against the null.

More formally again, if you denote the odds that an effect is real by $R$, and consider simple hypotheses, the probability that an effect is real given a significant result is $$ P = \frac{\text{power} \cdot R}{\text{power} \cdot R + \alpha} $$ This follows simply by Bayes theorem. So truly, low-powered studies result in weaker evidence.

See e.g., this article for further discussion (I'm sure there are heaps more).

Answer 5

There ARE times where it is appropriate to determine statistical power after a result is generated. If you have a very large sample size, then even a small difference between group A and group B will be statistically significant because the power to detect this difference is very high. Power depends on three things: your alpha level, the difference between group A and group B ("effect size") you hope to be able to detect, and the sample size. Changing any one of them will change the power.
It is also useful to compute the power of a study to determine why a result is not statistically significant. Many small studies, for example, are underpowered to detect a difference as significant only because the sample size is too small, or because the effect size is too small for the results to be significant with the sample size that was used. Many results have been dismissed based on a p-value alone, when in fact the study was underpowered to detect a difference, or the difference is in line with larger studies which were statistically significant because they had a larger sample size. In that case the problem is often a sample size issue, not an effect size issue. Sometimes just adding ONE more subject to the same experiment can push the results into statistical significance. A post-hoc power analysis can also determine the "achieved alpha" for a result, which is often much lower (or higher) than .05.

I know that you should ideally state your hypothesis in advance, including alpha level and sample size and the effect size you desire to detect, but sometimes when you're exploring your data you stumble on a relationship that is significant and meaningful. It is not "data fishing" to report this. Indeed, the p-value has been over-relied upon in statistical results, when sample size, effect size, alpha, confidence intervals and statistical power are very important in explaining results, too. Journals are increasingly recognizing this. It's also important to do your homework and look at what other studies in your field have done in terms of sample sizes and effect sizes and alpha levels, and the resulting effect on power.