3

The click-through rate (CTR) is defined as the total number of clicks divided by the total number of impressions. The diversion unit is usually a cookie, as it is the case in the following course by Google:

https://www.udacity.com/course/ab-testing--ud257

In this setting, the same cookie can generate multiple views and clicks. I am trying to wrap my head around how this is a valid setup given that the data points that go into the calculation of the CTR are not independent. Why is it a valid metric to calculate in the context of hypothesis testing and using, for instance, various classical proportion tests? In addition, how does one go about doing a power analysis in this setting with dependent observations?

Thank you!

Ivan
  • 458
  • 1
  • 4
  • 17
  • 1
    [This question](https://stats.stackexchange.com/questions/398436/a-b-testing-ratio-of-sums) and its answers probably address the second part of your question. For the record I don't think this is a duplicate of the linked question as it asks the why in addition to how. – B.Liu Jul 08 '21 at 13:41
  • 1
    Oh, I’ve missed this one. Thank you for the reference! – Ivan Jul 08 '21 at 18:02

1 Answers1

2

You are right that views and clicks from the same user are not independent events, and this violates the assumptions of classical tests like a t-test on proportions. This grouping behavior in the data could be addressed by, for example, using a mixed effects logistic regression with a random intercept for each cookie, or a bootstrap or permutation test stratified by cookie (so that the unit of re-sampling is all of the views and clicks for each cookie).

However, you can keep going in this direction virtually forever. For example, the cookies are also not independent samples -- cookies correspond to real people who know each other and share links, different people who use the same device, cookies from one geographic region may be correlated (perhaps it was rainy so users stayed inside and were more active), etc.. As George E P Box said, "all models are wrong, but some are useful", and so your goal should be to find a model that is useful for understanding the signal in your data.

jkpate
  • 1,183
  • 5
  • 8
  • Do you mean that everybody knows that there is a violation in this case, but a statistician might decide to analyze the metric using the classical tools still—such as a proportion test for two independent samples, each one with independent observations—simply because it is a useful approximation/model? If one wants to be particularly careful, one might decide to fit a more elaborate model instead, such a multilevel model. – Ivan Jul 08 '21 at 18:12
  • 1
    I can't speak to what everyone knows, but essentially you're right. Maybe you have tried both methods on real or realistic synthetic data and realized that they gave essentially identical results, or you found you did not have enough data to effectively estimate the more complex model, or you concluded the improvement from the more complex model was not worth the operational overhead or extra compute time. – jkpate Jul 08 '21 at 20:59