5

I'm a marketer trying to understand the math involved in A/B testing. I've come across a tool that estimates how many more samples are required to reach significance for a test that's already begun. When you enter your test results thus far, it tells you how many more samples are required to reach the 90% confidence level.

If I enter the following data into the form:

           Visitors    Goals
Group A:        200       25
Group B:        190       26

The tool tells me that I need an estimated 8405 more samples to reach significance at the 90% confidence level.

So far I've:

  • Read through the explanation of sample size in a couple of different statistics books.
  • Manipulation of power analysis equations on paper.
  • Experimenting with power.prop.test() in R.

I'm just not seeing how to calculate this. Can someone help?

COOLSerdash
  • 25,317
  • 8
  • 73
  • 123
Lenwood
  • 503
  • 5
  • 10

2 Answers2

3

I can't reproduce the exact numbers produced by this calculator but the output seems quite close to the result you might get from power.prop.test() for a power of 50% in a two-sided test, using the sample estimates for the two proportions, e.g.

power.prop.test(p1=26/190, p2=25/200, power=.5, sig.level=.1)

(The result is 4390 per group, so 8780 total or 8390 visitors more than you already have, the 15-observation difference could easily be accounted for by a rounding error somewhere – including on the website – or some other computational detail.)

If this is really what it is, it seems that there is a fundamental flaw in the approach, as it implicitly assumes that the proportions you got from your sample are indeed the true proportions, which seems to defeat the whole point of doing an experiment in the first place. In fact, there is no telling if the difference will “reach” significance, as you could very well discover that the results are very different after you have collected more data (perhaps the difference will be smaller or even go in the other direction).

Intuitively, this should be easy to realize if you consider that the recommended sample size is more than 20 times the size of the sample you already have. The data you have will be dwarfed by these new data and you will in effect have a completely new experiment that could go either way (especially since it's already clear that the effect is not very strong and the data you already have are not conclusive – otherwise, you could already reject the null hypothesis).

For a similar idea, expressed more carefully in a completely different context, you might refer to Kraemer H.C., Mintz J., Noda A., Tinklenberg J., & Yesavage J.A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63 (5), 484-489.

The information provided on the calculator's page is in any case very thin and insufficient to serve as a basis for a proper power analysis. Even if you were willing to disregard the point I just discussed, you would at the very least need the desired level of power (together with the effect size and error level) to compute a sample size and it does not seem to be mentioned anywhere.

Gala
  • 8,323
  • 2
  • 28
  • 42
  • That's a great tip on `power.prop.test()`, I'll take another look there. Thanks also for the article, I think that's exactly what I was looking for. Regarding your comment about the fundamental flaw, my understanding is that the right way to do this is to calculate the required sample size prior to the test beginning. In my experience most marketers either don't understand this or ignore it. So however incorrect, calculating significance as you go is the standard practice. That's no comment on the website I linked to, just my experience with marketers that run A/B tests. – Lenwood Jul 15 '13 at 17:42
  • 1
    @Lenwood If you absolutely must adjust the sample size as you go, it might be better to consider [accuracy in parameter estimation](http://stats.stackexchange.com/questions/16985/how-to-report-general-precision-in-estimating-correlations-within-a-context-of-j#30287) rather than power and repeated tests, [which can be very misleading if done improperly](http://allendowney.blogspot.nl/2011/10/repeated-tests-how-bad-can-it-be.html). – Gala Jul 15 '13 at 18:02
1

I just stumbled upon the equation that R is using in the power.prop.test() function.

$$ n=\dfrac{2(\bar{p})(1-\bar{p})(Z_{\beta}+Z_{\alpha/2})^{2}}{(p_{1}-p_{2})^{2}} $$

Note that the power & confidence intervals are expressed as Z-scores. I'm using a power of 80%, which is $Z_{\beta}=0.84$, and a confidence interval of 95%, which is $Z_{\alpha/2}=1.96$. These values give me the same answers as I'm getting from R. I won't mark this as the answer, but wanted to include it in case anyone lands here looking for an exact solution.

EDIT: I should correct my statement. This isn't the exact equation that R uses in their power.prop.test() function, but it does yield the same results. Their equation is more complex than this. I've tried them head to head dozens of times now, and the results are consistently +/- 2 of each other.

Lenwood
  • 503
  • 5
  • 10