3

I am running a multi-variate email campaign with the following results:

Version----------Users------Conversion

Control Group----150--------2%

Variation A--------1000-------2.1%

Variation B--------1000-------2.9%

Variation C--------1000-------1.4%

How do I statistically know which is the winning variation of this campaign? Should I be using chi-squared tests to determine the winner?

Furthermore, is there any standard way of selecting the size of the control group of a sample if I know the sample size? Should the size of the control group be the same as that of the variations or can I keep it as 5% to 20% of the sample regardless of the size of the sample? This is for marketing campaigns I run regularly.

mdewey
  • 16,541
  • 22
  • 30
  • 57
Arpit Rai
  • 131
  • 5
  • 1
    The standard meaning of a "multivariate" test or analysis is one that involves more than one *response.* Yours is a standard *univariate* response situation. See, for instance, material on Analysis of Variance (ANOVA). Incidentally, there is no clear winner here because the control group is too small. – whuber Oct 21 '16 at 12:56
  • Alright. 2 follow up questions in this case: (1) Is there a formula to determine the minimum size of a control group? (2) If the Control Group had 500 responses with the same conversion rate, would there still be no clear winner here? – Arpit Rai Oct 21 '16 at 13:03
  • 1
    There is an entire branch of statistics, *Experimental Design*, that (among other things) is concerned with determining optimal sizes of control groups and experimental groups. One insight it offers is that since you are comparing your treatment groups to a control, and the uncertainty in the comparison depends on the uncertainties in both the treatment *and* the control groups, then you should make the control group size *at least as large* as any of the treatment groups. If you have a budget for 3150 observations, then the control group should have at least 788 of them. – whuber Oct 21 '16 at 13:13
  • 1
    (Continued) However, suppose you obtained the same percentages with group sizes of 1000 each. The standard error of the difference between group B and the control would be 0.69%, making their difference of 0.9% only 1.3 standard errors. Although that is mild evidence that at least one variation exceeded the control, most people would not consider it significant. A simple logistic regression, a chi-squared test, or even an ANOVA would all help you carry out these calculations, but they will agree you don't have enough data to determine whether any variation is better than the control. – whuber Oct 21 '16 at 13:17
  • Great. Thanks a ton! This is really really useful! :) I have one last unrelated question if you don't mind. Is there any relation between the size of the sample and the control group I should create out of this sample? Should I always create a control group sized similar to the variations or can I just keep it to 5% to 20% of the sample regardless of sample size? I know you mentioned there is an entire branch dedicated to this but I thought I’d still ask. – Arpit Rai Oct 21 '16 at 13:37
  • That requires a calculation as well as a slightly more specific statement of your problem. Consider editing your post to ask how to optimally divide your sampling budget between control and treatment groups for your specified purpose. – whuber Oct 21 '16 at 13:44
  • Sure! I've edited the question – Arpit Rai Oct 21 '16 at 13:52

0 Answers0