How to test if the difference between the means of two groups is more than a given value?

Question

The null hypothesis for the t-test is the means between two groups are the same (i.e. the difference between the means of two groups equals 0). However, my null hypothesis is the difference between the means of two groups is less than 5 (alternate hypothesis is > 5). What's the appropriate test/approach in this situation?

Answer 1

Non- inferiority testing, if your null hypothesis is $\mu_1-\mu_2>\delta$ and equivalence testing, if your null hypothesis is $|\mu_1-\mu_2|>\delta$.

There are usually no tests with fancy names for doing this, but you can usually look at confidence intervals (two-sided level $1-\alpha$ ones to get a level $\alpha$ test) and see whether they overlap with your null hypothesis. If so, you cannot reject, otherwise you reject. If asymptotics apply, it's usually trivial to construct a test, too.

Answer 2

Note: the following assumes that by "difference", you mean the unsigned difference, i.e. you take one value minus the other, then take the absolute value. You should be precise, however, and explicitly say which you mean.

The purpose of having a null hypothesis is so that you can calculate the probability of seeing your results, given the null hypothesis. "This statistic is comes from a student-t-distribution whose mean has an absolute value less than 5" is not specific enough to calculate probability; you have different values within that range, and each value will give a different probability.

What you can do is take your data and then take the "worst case" (note that you normally should decide what your null hypothesis is before collecting any data, but taking the worst case ensures that you aren't improperly detecting significant). By "worst case", I mean: what null hypothesis would give you the least significance? If your sample mean is greater than 5, then the worst case would be taking the null hypothesis to be that the true mean is 5. You should then calculate the probability, given that the true mean is 5, of getting a sample mean greater than or equal to what you observed. If your sample mean less than -5, then the worst case would be taking the null hypothesis to be that the true mean is -5, and finding the probability, given that the true mean is -5, of getting a sample mean less than or equal to what you observed. If the sample mean is between -5 and 5, then you should not reject the mean. Since you don't know ahead of time which direction you're going, you should consider this a two-tailed test.