1

I was reading about Simple Random Sampling (SRS) from this lecture notes. It says that SRS, which actually is a sampling without replacement makes $x_i$ not independent.

Then I thought about A/B testing. When doing A/B testing for a product feature, usually users are sampled randomly without replacement, put in one of two buckets, and only one group will see the feature. Also, usually I analyse the result using t-test, which assumes independence of samples.

Is this a problem and what should be the workaround?

hans-t
  • 519
  • 2
  • 5
  • 18

1 Answers1

2

The author is talking about a finite sample correction. When A/B testing, the population is rather large, unless you have some website that is only accessible to only a limited number of people, so the correcting factor:

$$\frac{N-n}{N-1}$$

tends to $1$ and can be ignored, as long as the sample you draw for A/B testing ($n$) is supposedly smaller than the population of potential visitors to your site ($N$). Even if the sample was large compared to the population, how would you go about determining the population size of potential visitors to your website?

If anything, the usual unbiased estimator of variance has a larger denominator ($\frac{1}{n-1}$) than the finite-sample corrected version. In other words, the worst case scenario is an overestimation of the variance, resulting in slightly less power.

Frans Rodenburg
  • 10,376
  • 2
  • 25
  • 58