1

I have an entire market for which I plan to run two offers.

I randomly split the market 50/50. All of Group A receives offer 1 and all of group B receives offer 2.

I want to compare proportions of converters and revenue generated from the two groups

Are the groups A & B a "sample" or a "population" now?

When comparing proportion of converters or revenue from the two groups do I need hypothesis testing or is it enough to state the obvious eg Group A 30% converted vs Group B 20% converted and therefore offer 1 performed better than offer 2

clyguy
  • 21
  • 4
  • How are you interpreting this data? historically or to forecast the future? if to forecast the future then its (likely) a sample – seanv507 Apr 11 '19 at 08:26
  • @seanv507 if I am just reporting on the results of the offer, I should just consider group A and group B as two populations and report the results as is? – clyguy Apr 12 '19 at 17:33
  • Yes that's right – seanv507 Apr 12 '19 at 20:26
  • @clyguy: I think you should *at least* make some point about sampling variability. In marketing it is next to impossible to have a stable/constant group of observations so I think we should refrain from *population*-wide claims. Please see my answer below for more details. (Reasonable question though, +1) – usεr11852 Apr 13 '19 at 18:12

1 Answers1

1

Unless we are dealing with a isolated set of individuals that is stable across time we are dealing with a sample. Here, as the problem statement particularly refers to "an entire market" and a market is not an inherently stable construct, it would be better to refer to sample or at least "accessible population". If anything this will give us flexibility if a new customer join the market or old customer depart and will also allows us to naturally refer to sampling error. This is important even in the case mentioned (Group $A$ has 30% conversion and group $B$ has 20% conversion) as the conversion estimates are still subjected to sampling variability. CV.SE has a really good thread on the matter aptly title: "What is the difference between a population and a sample?".

To summarise: unless we are in a highly idealised situation, where we expected little to no changes (either from external factors coming into play or just because of time evolution) I would refrain from using the term "population" without some further qualification like "target" or "accessible" population.

usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • Thanks for that @usεr11852 ! I like the point that you make about "allows us to naturally refer to sampling error" Initially, the entire market is split 'randomly' and I believe this introduces sampling variability. The right thing to do would then be a t-test to determine whether the difference in two conversion rates are statistically significant? – clyguy Apr 14 '19 at 11:45
  • I am glad I could help; if you find an answer helpful you could consider upvoting it. For the follow-up question: Yes, I would probably suggest a Welch's t-test, to ensure any potential issues of unequal variances are taken into account. As the samples mentioned are probably not too small, we will lose almost nothing when using Welch's t-test instead of the usual Student's t-test. – usεr11852 Apr 14 '19 at 21:03