2

For context: How to define prior for beta-binomial A/B test

For P(A > B), you can draw samples from A's posterior and B's posterior and then count the number of times the sample from A is greater than the sample from B. If choice C entered the picture, I don't intuitively see why:

mean(A_samp > B_samp & A_samp > C_samp)

wouldn't work, but I suspect that's a bit too simplistic. I'm not sure how to extend this to the n >= 3 case.

John
  • 21
  • 1

1 Answers1

1

Actually, this blog entry by Evan Miller (mentioned in the link you refer to) describes case when you want to test the hypothesis that $A > \max(B,C)$ for binary and count data. It basically gives closed-form solution for what you are suggesting, so in fact there is not need to conduct simulation.

Notice however that since people mean often different things by A/B testing and if you mean that your data is counts of choices in multiple-choices scenario, then Dirichlet-multinomial model would be more appropriate.

Tim
  • 108,699
  • 20
  • 212
  • 390