0

We did an A/B test where A is control and B is the same website with one feature variant that is available on all pages on the website. The feature variant is live chat. We wanted to see if live chat helps people convert or if it hinders.

In total A and B have similar conversion rates. Only 1% of users in group B used live chat. However, the those who did had a conversion rate 10 times higher.

One possibility could be that A and B have similar conversion rates because the percentage of users who use live-chat is so low. Another possibility is that the users who used live-chat were more engaged and so more likely to convert anyway.

Would it make sense to bootstrap the distribution of conversion rates for the population who used live chat and see how that distribution compared to the overall distribution? Or is there another way to tease out causality?

Statbie
  • 1
  • 1
  • 1
    What exactly differed in the conditions of the A group versus the B group? Bootstrapping doesn't seem to have anything to do with the problem. – AdamO May 05 '21 at 19:49
  • A is the control. B is the variant. The feature in B that is different is used only 1% of the time. Traditional hypothesis testing techniques find no significant difference. I was thinking bootstrapping could be used on the subpopulation and compare to the overall sample population. – Statbie May 05 '21 at 20:24
  • That is not what I'm asking. What is the "variant"? Is this "feature" the variant? A short, maybe one paragraph, context is always helpful. – AdamO May 05 '21 at 20:31
  • Thanks, I edited the question and hope that is clearer. – Statbie May 05 '21 at 20:56

1 Answers1

1

In a randomized controlled trial we often distinguish between the per-protocol versus the intent-to-treat effects of a drug. For instance, a highly effective anti-cancer drug may have low tolerability, so if you restrict analyses on patients who actually use the drug, you no longer have a randomized comparison, and the "compliers" in the active arm are generally a healthier or more resilient set.

Whether the actual use of the newly adopted chat feature in the B group led to a higher conversion rate is moreover concerned with the "per protocol" effect. But the fact that conversion was higher regardless of utilizing the feature may be part of an intent-to-treat story: the fact that the option was made available was reason enough to suggest the site was high enough quality to warrant a conversion. There would be no comparable "per protocol" analysis since there is no placebo or dummy condition in the "A" group by which to self-select the utilizers.

AdamO
  • 52,330
  • 5
  • 104
  • 209