5

Sometimes researchers (especially in collaborations) have two opposite but sound theories: There is a difference between two groups or there is only negligible difference. Now they ask their statistician. How should he approach the situation?

If he does a point hypothesis test, he favours the difference party since only their theory can be confirmed in case of rejected point hypothesis whereas not rejecting the point hypothesis teaches nothing.

If he does only equivalence testing, he favours the negligible difference party for the analogous reason.

Should he do both, of course with multiplicity correction? So a TOST for equivalence and a point hypothesis test? Or a TOST and a respective relevance test?

This procedure would have three outcomes:

  1. Equivalence of both parameters up to negligible differences.
  2. Large enough differences.
  3. Nothing to learn since both hypotheses have not been rejected.

Is such an "et-et"-approach reasonable? Why do we hardly see such "et-et"-analyses in publications? This is a general question for reasoning. So multiple answers are encouraged and I do not restrict this question to particular models.

Horst Grünbusch
  • 5,020
  • 17
  • 22
  • It is not quite clear what you are asking. Are you asking about the importance (or not) of multiple corrections in a relevance testing situation (i.e. given that, when using TOST for equivalence, relevance testing entails three tests... albeit the TOSTs are non-overlapping in the nulls). Also, for the sake of clarity to one's audience I would encourage notationally differentiating the null hypotheses for a test for difference (H$^{+}_{0}$) from a test for equivalence (H$^{-}_{0}$) as you have seen me use in other questions and answers. – Alexis Aug 19 '14 at 14:29
  • I hope I made it clearer. In fact, hypothesis testing is often used with rather explorative than confirmative intent. So both, equivalence and difference, are of equal interest. However, only one of them is usually used. Why? – Horst Grünbusch Aug 20 '14 at 13:35
  • Not sure. I suspect because of historical contingencies, possibly having to do with equivalence testing coming largely out of pharmaceutical research and clinical trials. I know that the (frequentist) curriculum I teach now emphasizes that one simply ought conduct both. – Alexis Aug 20 '14 at 15:46
  • Are you restricted to frequentist tests, or open to Bayesian methods? – Sean Easter Aug 21 '14 at 17:28
  • Also Bayesian approaches to testing can help. What I'm not interested in is a dichotomous decision "either difference or equivalence". Statistics should address situations too bad for reliable inference (e.g. too small sample sizes) as outcome (3) does. – Horst Grünbusch Aug 22 '14 at 09:04

1 Answers1

2

These arguments from "Bayesian Estimation Supersedes the t-Test" seem relevant:

This article introduces an intuitive Bayesian approach to the analysis of data from two groups. In particular, the analysis reveals the relative credibility of every possible difference of means, every possible difference of standard deviations, and all possible effect sizes. From this explicit distribution of credible parameter values, inferences about null values can be made without ever referring to p values as in null hypothesis significance testing (NHST). Unlike NHST, the Bayesian method can accept the null value, not only reject it, when certainty in the estimate is high.

Meaning, were the statistician Bayesian, she would accept the posterior as the best possible inference given the available data, assumptions, and prior knowledge. She'd then interpret the posterior as the credibility of both opposed theories.

If the pair could agree ahead of time on a region of practical equivalence—a ROPE is basically a range of negligible differences—then only one of the three outcomes can be credible under the posterior.

  1. The entire ROPE falls within the 95% highest-density interval (HDI) of the posterior of difference of the parameters; the groups are practically equivalent.
  2. The ROPE and HDI do not overlap; the groups meaningfully differ.
  3. The two overlap; the pair must agree to disagree, or to gather more data.
Sean Easter
  • 8,359
  • 2
  • 29
  • 58