1

I am trying to find the confidence interval of the end result of two A/B tests.

As an example, lets assume that for test 1 I am looking at the click-through rate of a page and have a control and treatment cell.

I have found that the click-through rate increases from 0.387 in control to 0.393 in treatment by using the test of differences between proportions to find that the difference between control and treatment is significantly different from zero (p=0.01). The sample size for both control and treatment is about 100K.

A week later, I do a second A/B test, again with a control cell and a treatment cell. I record the average number of items bought after the user clicks on the page I tested in Test 1. I find that for this test, the number of items increases from 0.367 in control to 0.385 in treatment. This time I use a T-test for differences between means with sample sizes of 60K per cell. I find that the sd of control is 0.805 and the sd of treatment is 0.815. Again, the result is statistically significant with p=0.00.

Now, since I increased the click-through rate by 0.393/0.387 = 1.015504 and the number of items bought after clicking by 0.385/0.367= 1.049046, my point estimate for increase in sales (assuming everything else remains equal) is 1.015504*1.049046=1.06531. I can also compute 95% confidence intervals for both Test 1 and Test 2 separately. However, I would like to compute the confidence interval for this estimate of 1.06531. For simplicity's sake, let's assume test 1 and test 2 are completely independent.

What method can I use to estimate an accurate confidence interval here? I know multiplying the two separate confidence interval doesn't work, and I have tried the Delta method described here (Confidence interval for the product of two parameters), but this gives a much narrower confidence interval than the product of the two separate confidence intervals, which seems counterintuitive. (I can't rule out I made a mistake though). The calculations in R are:

varp2 <- (0.004104847/3)^2

varp1 <- (0.005604847)^2

#Means of differences

p2 <- 1.049046

p1 <- 1.015504

#Formula

(p1)*(p2)

#Delta method, assuming zero covariance

varX <- ((p2)^2)*varp1+((p1)^2)*varp2

ub <- (p1)*(p2)+sqrt(varX)*1.96

lb <- (p1)*(p2)-sqrt(varX)*1.96

Where the variances may be a bit different than the ones you'd calculate manually, since the data I gave above was a rough estimate.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
A. Tom
  • 55
  • 6
  • 3
    Could you perhaps show your delta method calculation? Because that is indeed the way to go – Knarpie Aug 11 '20 at 15:55
  • 3
    Your title mentions that you're taking a product of CI's but in fact that's not what you seek at all; you are nowhere multiplying the *intervals*. You'll need to make your title reflect the quantity you're actually seeking an interval for. – Glen_b Aug 12 '20 at 00:43
  • Okay so for the delta method calculation I added converted the difference to ratios. The variances are simply the variances of the estimated differences between Control and Treatment in their respective tests. I have: varp2 – A. Tom Aug 12 '20 at 10:11
  • As for your comment about a better title, do you have a suggestion? I could imagine it being a confidence interval of the product of two random variables with their own distributions. – A. Tom Aug 12 '20 at 10:23
  • Can you please add those additional details as an edit to the post? Not everybody red comments (and comments with formulas/code are unreadable ... – kjetil b halvorsen Aug 12 '20 at 15:27

0 Answers0