2

The lab I work in uses equivalence. When we get a significant result from a two one-sided test, we conclude that true difference between mu1 and mu2 is likely inside the specified margin, which is taken to be the largest difference that is of no clinical significance. I'm interested in using Bayesian estimation instead of a Frequentist approach for this, but I'm having trouble parsing out what his objections with equivalence testing are.

Specifically, I have no idea what the bold part is trying to communicate:

ROPE method cannot be used to accept null value in NHST. Because an NHST confidence interval (CI) has some properties analogous to the Bayesian posterior HDI, it may be tempting to try to adopt the use of the ROPE in NHST. Thus, we might want to accept a null hypothesis in NHST if a 95% CI falls completely inside the ROPE. This approach goes by the name of equivalence testing in NHST (e.g., Rogers, Howard, & Vessey, 1993; Westlake, 1976, 1981). Unfortunately the approach fails because the meaning of the CI is not the same as the HDI. In a Bayesian approach, the 95% HDI actually includes the 95% of parameter values that are most credible. Therefore when the 95% HDI falls within the ROPE, we can conclude that 95% of the credible parameter values are practically equivalent to the null value. But a 95% CI from NHST says nothing directly about the credibility of parameter values. Crucially, even if a 95% CI falls within the ROPE, a change of intention will change the CI and the CI may no longer fall within the ROPE. For example, if the two groups being compared are intended to be compared to other groups, then the 95% CI is much wider and may no longer fall inside the ROPE.

Passage is from this article: http://www.indiana.edu/~kruschke/BEST/BEST.pdf

The way I see it, with equivalence testing I'm never actually accepting the null hypothesis of mu1 - mu2 = 0, I'm rejecting two separate null hypotheses show that the mean difference is less than some practically significant amount. I'm not sure how the BEST approach allows you to conclude that mu1 - mu2 = 0 any more than the Frequentist approach.

dmacfour
  • 273
  • 1
  • 6
  • Does https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval or https://stats.stackexchange.com/questions/26450/why-does-a-95-confidence-interval-ci-not-imply-a-95-chance-of-containing-the nswer your question? – Tim Feb 23 '18 at 22:54
  • 2
    This passage is a typical passage of Bayesians bashing Frequentist methods. P-values are misleading, confidence intervals are misleading, statistical tests are misleading, etc. This passage is not saying anything in addition. Equivalence testing is fine as long as frequentist methods are concerned. – amoeba Feb 23 '18 at 23:04
  • I guess the issue here is that I don't see how my conclusion will change with the Bayesian approach, despite the obvious differences between credible and confidence intervals. The alternate hypothesis in an equivalence study is that |mu1 -mu2| < margin. The author claims that you can accept the null hypothesis of a traditional t-test with the BEST method, but it seems to me like he's just showing that |mu1 -mu2| < margin with a different approach, not explicitly concluding that mu1 = mu2. – dmacfour Feb 23 '18 at 23:32
  • Using an HDI and ROPE to make a decision is not hypothesis testing. If the HDI falls inside the ROPE you don't conclude that a null hypothesis is true, you conclude that the magnitude of the effect is practically equivalent to the null. For a discussion of the differences between Bayesian hypothesis testing and the HDI+ROPE method, and a discussion of how they differ from frequentist methods, see the article available at https://psyarxiv.com/wntsa/ or https://link.springer.com/article/10.3758/s13423-016-1221-4 – John K. Kruschke Feb 24 '18 at 11:01
  • @JohnK.Kruschke - thanks for the reply. I guess I was confused because with the way we've been using equivalence testing, we're also deciding that the magnitude of the effect is practically equivalent to the null (on the basis of two separate hypothesis tests). If you don't mind me asking, is it possible (or advisable) to use multiple ROPEs? As in, one region that's narrower than another? – dmacfour Feb 24 '18 at 18:05
  • You can think of the HDI+ROPE procedure as the Bayesian analogue of frequentist equivalence testing. Set the ROPE the same way you'd set the ROPE in equivalence testing! A key advantage of using HDIs instead of confidence intervals is that HDIs don't change when you change your testing intentions, but confidence intervals can change dramatically when you change your testing intentions. – John K. Kruschke Feb 24 '18 at 21:25

0 Answers0