1

We have an app, where you post projects and we'll be running A/B test on new version of posting page. How do we ensure that the new version is not worse?

We have 14 days of time to decide that. And on average we get 708.73 projects posted in 14 days with standard deviation of 36.16.

We could just wait until we see the results and see if it's better or worse. But how to determine it's just not worse. Doesn't matter if it's nearly the same.

Matjaz Muhic
  • 59
  • 9

1 Answers1

3

Tests that have an alternative hypothesis of a lack of difference fall under a tests for "statistical equivalence" or "statistical noninferiority." The key difference between tests for statistical difference and tests for statistical equivalence lie in how the null hypothesis is formulated. In the former, the null hypothesis takes a form akin to: "There is no difference in the means of the two groups." In the latter, the null takes a form such as: "There is a meaningful difference in means of the two groups." This is a nice description of the method.

To become more specific for your problem, the main question you have to answer is what difference in posting projects would be considered sufficiently meaningful that your organization would find it unacceptable? Would a reduction in 50 projects be too much? 100? That's only a question you can answer. Further, this is the most important question. Once you determine this threshold, then you can conduct an appropriate one-sided test for non-inferiority where the null is whatever you decide.

As an illustrative example, lets say you define "worse" as 50 fewer projects, your data are normal, and you like the usual confidence of 95%. Then you'd test against $H_0 = 50$ using a $t$-test with one-sided $\alpha = 0.05$. If this test is significant, you can say that with 95% confidence that your new method is no worse than the old method (or that it is non-inferior).

To take it one step further and declare equivalence, you can pick an upper bound as well, and conduct a similar one-sided test. If you do this, it fits into a TOST, or a two one-sided test, methodology. Graphically, as in the link above, this is the same as plotting confidence intervals (of level $2\alpha$) with an appropriate zone of scientific indifference (the two limits you previously determined).

Ashe
  • 1,085
  • 10
  • 25
  • Splendid! Thanks so much! I really have a problem figuring stuff out mostly cause I don't know what to search for. This helps a lot. :) – Matjaz Muhic Oct 14 '16 at 17:15
  • Also, this means that a normal A/B test where we test that there IS a difference is not suitable for this right? – Matjaz Muhic Oct 14 '16 at 17:16
  • Yes, you are correct it would be unsuitable. In a typical test for difference, you start by assuming there is no difference (the null). If you don't find one, you didn't provide evidence there isn't a difference; rather you just don't have evidence to change your assumption. To provide that positive evidence for noninferiority, the trick is the inversion of the null/alternative paradigm. Further insight into these thoughts are here: http://stats.stackexchange.com/questions/85903/why-do-statisticians-say-a-non-significant-result-means-you-cant-reject-the-nu – Ashe Oct 14 '16 at 17:54
  • So, what if the new variant is better? Can I still run the A/B test, see if one is significantly better or not. Otherwise if we don't get any conclusive results I use equivalency test? – Matjaz Muhic Oct 14 '16 at 18:23
  • Oh, and one last thing. If I manually (without software) want to compare the two variants, how do I go about that? And, in case i only care that it's not worse, I can use only one one-sided test right? Testing that it's above the lower bound - which we still consider acceptable? – Matjaz Muhic Oct 14 '16 at 19:51
  • To test for superiority, you can switch to the traditional hypothesis testing framework, but I would communicate that it was a post-hoc observation that the new method is better and it wasn't a prior hypothesis. In that case, your null will be the usual lack of difference in means. To conduct the test by hand, you'd have to select what test you want to do first. If it's a $t$-test, that's readily accessible by hand calculation (eg by looking at the Wikipedia page). It's also a command in Excel, if you have access to that. – Ashe Oct 14 '16 at 20:17
  • Or (after a quick Google search) use this tool: http://www.graphpad.com/quickcalcs/ttest1.cfm – Ashe Oct 14 '16 at 20:18
  • Thanks a lot! I wish I could buy you a beer! Or a beverage of your choice. :) – Matjaz Muhic Oct 14 '16 at 20:23