1

I have a typical A/B test setup where I have a control and treatment sample of equal size with very small success proportions. An example could be the following data with a sample size of 505000, where the average of Y, $\bar{Y}$ is the variable of interest:

$$ \begin{array}{c|lcr} Y & \text{Control} & \text{Treatment} \\ \hline 0 & 500000 & 499000 \\ 1 & 5000 & 6000 \end{array} $$

So $\bar{Y}_{control}=0.01$ and $\bar{Y}_{treatment}= 0.01202405$.

What are the tests you can use for $\bar{Y}_{control}$ being significantly different from $\bar{Y}_{treatment}$?

I've read about the chi-square test for equality of two proportions and the Z-test, which are supposedly exactly the same (What is the relationship between a chi squared test and test of equal proportions?).

I've also had people suggest binomial tests to me as 'tests relying on normal distributions won't work due to the fact that the proportions are very close to zero, which means that the confidence intervals won't be symmetric'.

A. Tom
  • 55
  • 6

1 Answers1

3

With enough data, the tests you mention are all the same. At sample sizes this large, the binomial is very well approximated by a normal distribution with the same mean and variance. But more to your question, you could use any of those tests. Here are some results in R.

results = c(5000, 500000, 6000, 499000)
m = matrix(results, nrow = 2)


#Chisquare
chisq.test(m, correct = F)

#Binomial
prop.test(c(6000, 5000), c(49900+6000, 550000+5000))

You raise an interesting point about asymmetric CIs. When computing CIs for the underlying proportion, extremely large or small conversions can result in CIs which cover non-physical parameter values (e.g. 1.01 or -0.01). Many alternative intervals have been proposed to rectify this (see the binom package in R). However, for the difference of proportions between two groups, confidence intervals covering non-physical values is rarely a concern (esp because conversion between test in control in AB tests are very close in absolute value).

There is an entire chapter dedicated to the comparisons of two groups on a binomial outcome in this book. If you are interested in knowing more about the details of these sorts of tests, and don't mind a little math, I highly recommend Chapter 2.

Demetri Pananos
  • 24,380
  • 1
  • 36
  • 94
  • Thank you for your comprehensive answer. I understand that for very large sample sizes, most of these tests will give very similar results. The point you mention about CIs covering non-physical values as it relates to proportions between two groups seems similar in that this will only cease to be problem as sample size grows to a point where the confidence intervals are tight enough to exclude non-physical values. I expect this to have an ambiguous answer, but would a test that is correct across all sample sizes be 'more' correct than one which is correct once n exceeds a certain point? – A. Tom May 04 '20 at 12:08
  • And, if so, what would be a test such as this? – A. Tom May 04 '20 at 12:13
  • 1
    "more correct" is an ambiguous term to me. We can talk about false positive rates and power of tests if you like, but for most AB tests, a chi square or test of proportions is usually a fine method of analysis provided there are no problems in the design of the experiment. – Demetri Pananos May 04 '20 at 12:14
  • Yes you're actually wording this better than I did, are there tests with greater or equal power at all sample sizes for proportional data like I'm describing? Because if that is the case I think one could argue that this would be 'more correct' (excuse the ambiguity) – A. Tom May 04 '20 at 12:18
  • 1
    I'm sure there is a most powerful test for the test of two proportions. Which test that is is known to me. But again, provided that the experiment is designed properly, this really shouldn't be much of an issue. – Demetri Pananos May 04 '20 at 12:19