0

Here are some example data:

+----------------+-----------+---------------+--------+-----------------+
|    Version     | Converted | Not Converted | Total  | Conversion Rate |
+----------------+-----------+---------------+--------+-----------------+
| Original       |       300 | 10,000        | 10,300 | 2.9%            |
| Test Variation |       175 | 5,000         | 5,175  | 3.4%            |
| Total          |       475 | 15,000        | 15,475 | 3.1%            |
+----------------+-----------+---------------+--------+-----------------+

Context is a website where a test landing page was created diverting a percentage of traffic to the variation. Over the time period the variation showed a higher conversion rate. Is this real or is it just ebbs n flows?

To get the probability of seeing 475 conversions based on a null hypothesis I might use:

> dbinom(x = 475, size = 15475, p = 300/10300)
[1] 0.009591306

There's a 0.0096% chance of seeing 475 conversions assuming no difference in the test versions.

Or, would I do this:

> 1 - pbinom(q = 475, size = 15475, p = 300/10000)
[1] 0.2961731

In this case "There's a 29% chance we would have seen 475 or more conversions if the null is true.

I must be over thinking it because I cannot form my question correctly. I want to know if the test was really a success or not. So that my head doesn't explode I'd prefer an answer in terms of a binomial distribution approach that I've started on, if possible. But would appreciate pointers on any in built r functions for this kind of thing too.

Should I be looking for the probability of seeing exactly 475 conversions or should I be looking for the probability of seeing 475 or more conversions?

Doug Fir
  • 1,218
  • 1
  • 13
  • 27
  • 1
    You will benefit from careful reading of the posts at https://stats.stackexchange.com/questions/31. Note that your calculations are inapplicable because they do not account for sampling error in the "original" group, as they must. – whuber Jul 06 '17 at 14:49
  • Thanks for the link, this part especially helped clarify my understanding here "Therefore, a pp-value of 0.060.06 would mean that if we were to repeat our experiment many, many times (each time we select 100100 students at random and compute the sample mean) then 66 times out of 100100 we can expect to see a sample mean greater than or equal to 55 ft 99 inches.", especially use of the word "greater than" – Doug Fir Jul 07 '17 at 02:26

1 Answers1

1
  1. If you want to show that your test variation has a higher conversion rate, you have to test the hypothesis that rates are the same against the alternative that rate is higher for the variation. Hence, you should use the probability of 475 conversions or higher.
  2. I don't get why do you compare the conversion rate in both groups to the rate in the variation. Doesn't it make more sense to compare the rates for the original and the variation directly?

  3. You use one-sample test, but the conversion rate in the original group is not precisely known - it is estimated from the sample as well. I would use two-sample z-test and Wilson's confidence interval for independent proportions:

    y1 <- 175 n1 <- 5175 y2 <- 300 n2 <- 10300 P <- (y1+y2) / (n1 + n2) z <- (y1/n1 - y2/n2) / sqrt(P*(1-P)*(1/n1 + 1/n2)) 1-pnorm(z) library("PropCIs") diffscoreci(y1, n1, y2, n2, 0.95)

Both methods don't find sufficient evidence that conversion in the variation is higher at $\alpha=0.05$.

Evgeniy Riabenko
  • 578
  • 2
  • 11
  • Hi thanks for clarifying my understanding, point number 1 really answers my question. I don't follow on points 2 and 3 though, could you expand a little? For 2 how would I "compare the rates for the original and the variation directly"? Isn't that what I'm already doing? For 3, I don't follow. When does one choose a two sample test and is using `pbinom()` inappropriate here? Why? What does the two sided test do that the method I'm currently using does not? – Doug Fir Jul 07 '17 at 02:23
  • I was reading the free stats book "open intro statistics" (https://drive.google.com/file/d/0B-DHaDEbiOGkc1RycUtIcUtIelE/view). Under the hypothesis testing chapter there's a tip box with the following statement "When you are interested in checking for an increase or a decrease, but not both, use a one-sided test. When you are interested in any difference from the null value – an increase or decrease – then the test should be two-sided". This is confusing when reading against my example. Am I looking for an increase or either an increase or decrease? – Doug Fir Jul 07 '17 at 02:46
  • 2. In your calculations above you are comparing the proportions in original group with the proportion in both groups in total. – Evgeniy Riabenko Jul 08 '17 at 00:01
  • 1
    3. You need two-sample test, not two-sided test. It it two-sample because you have two samples, proportions in which you are comparing. If you had just one group and a hypothesis about the proportion in it, you could've used one-sample test. In reality, you are interested in testing the hypothesis $p_1=p_2$, not $p_1=300/10300$, because the real conversion rate in the original group is unknown, and $300/10300$ is just its sample estimate. – Evgeniy Riabenko Jul 08 '17 at 00:07
  • Thanks for the follow ups this helps. Number 3 makes sense, I understand it now. For number 2, I should have made my probability parameter 475 / 15,475? – Doug Fir Jul 08 '17 at 01:20
  • No, you should have x = 175, size = 5175 instead – Evgeniy Riabenko Jul 11 '17 at 08:25