9

I'm relatively new to statistics, and understand that my question may be completely misworded. I am testing my own algorithm versus another. While the outputs are not identical, I want to show that the differences are "statistically insignificant." How can I quantify this, to make my point?

chl
  • 50,972
  • 18
  • 205
  • 364
Adam_G
  • 238
  • 2
  • 13
  • That'll depend on what kinds of data you're dealing with, and how big your samples are. Can you edit your post to include a more detailed answer? – naught101 May 22 '12 at 02:32
  • The equivalence test is powwered to reject nonequivalence. It forces you to pick the sample size large enough to have power. Without reversing the null and alternative hypothesis in small samples you would have very little change of rejecting the null hypothesis of no difference. But not rejecting is not the same as accepting because of the lack of power, That is why Blackwelder makes non-equavalence the null hypothesis and showing equivalence requires rejecting the null. – Michael R. Chernick May 22 '12 at 03:23
  • Note that the null hypothesis is that the difference in means is greater that a specified delta (the window of equivalence). – Michael R. Chernick May 22 '12 at 03:23

1 Answers1

13

If you are comparing two groups and want to show no significant difference, this is called equivalence testing. It essentially reverses the null and alternative hypotheses. The idea is to define an interval of insignificance called the window of equivalence. This is used a lot when trying to show that a generic drug is a suitable replacement for a marketed drug. A good source to read about this is William Blackwelder's paper titled “Proving the null hypothesis” in clinical trials.

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • Thank you Michael. I guess I need to do some more work before I can declare the data "significant" or "insignificant." I will start looking into this, and let you guys know if I have any follow-up questions, which I'm sure I will. – Adam_G May 22 '12 at 11:40
  • 3
    Just one last comment. The one-sided version of equivalence is called noninferiority. Often drug manufacturers can bring a so-called "me too" drug to market that does not even have to be as good in efficacy as the marketed competitor as long as it is not much worse (defined by a delta that is a window for noninferiority. This can be done because the product offers some other advantage to the consumer, like fewer doses per day or as in the case of an anticoagulant not needing to have INR monitored. – Michael R. Chernick May 22 '12 at 13:28