0

How do I test my hypothesis that the populations are the same in two trials with binomial/Bernoulli results?

For testing the hypothesis that the populations are different, I would use straightforward statistical hypothesis testing using a Chi-squared or Fisher's exact test (e.g. using chisq.test() or prop.test() in R, or the equivalent calculation in Excel), as suggested in these answers:

However, if I have understood right, these test the hypothesis that the populations are different, and failing to reject the null hypothesis that they are the same is not what I am trying to show. My hypothesis (based on a priori information) is that they are the same, and to prove that with my data I need to fail to reject the null hypothesis that they are different. So I need to test the hypothesis that the populations are the same.

I have found a question that helps for this issue, which says that this is testing for equivalence, which is indeed different to standard hypothesis testing:

However, from the linked suggestions there, I could only find answers that address normally-distributed data (i.e. numerical data that has a mean and a standard deviation), and my data is categorical (yes/no, success/failure, Bernoulli trials, binomial distribution).

Further, the top answer there says that

Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent

Which seems much more subjective than standard hypothesis testing, and seems to beg the question I'm trying to answer. I want to use the data to show that the groups are effectively equivalent, not pull a figure out of thin air for how large a difference is acceptable and see whether my data fall within that difference. (I can see how this approach might make sense in a clinical/medical regulatory context, though.)

Is there no simple procedure analogous to the Chi-squared test to test the hypothesis that two groups of Bernoulli-trials are essentially the same?

(If it is relevant, in my data N is quite large - of the order of thousands of individuals in each of two trials. I'm looking at several outcomes within each trial, and for some of those outcomes, about half of the trials are successes, but for others, the figures are near to 100% or 0%, so the number of individuals in some of the cells of a contingency table may be quite low or even zero. Inspecting the percentages by eye, they sure look the same - most differ by less than a percentage point or three - but that's not a proper statistical test.)

Community
  • 1
Rilvabin
  • 1
  • 2
  • "*they are the same, and to prove that*" you - quite literally - *can't* prove they're the same. Equivalence testing certainly doesn't do that. And if you have no objective way to define "essentially the same" then you must, perforce, deal with the fact that it's necessarily subjective. You can make it objective, if *you* have objective criteria for what you mean by "essentially". Nobody else can define what you mean when you say it. – Glen_b Oct 25 '15 at 11:48
  • For standard hypothesis testing, we can say "if the groups were the same, you would only expect to see results this different p% of the time". Is there really no way of saying "if the groups were different, you would only expect to see results this similar q% of the time?". If I'm saying what counts as significantly different couldn't I just say "I'll call them the same if the success rates are within x %-points of each other"? Which doesn't really look like much of a statistical test! – Rilvabin Oct 25 '15 at 13:49
  • Consider that "different" can be different by infinitesimally small amounts. $\pi_1 = \pi_2 + \delta$ is still different, whether $\delta = 0.1$ or $\delta=10^{-100}$. For any given sample size, there's a size of difference that will be effectively indistinguishable from no difference at your sample size (even though they're clearly different at a far smaller sample size). Until you specify the smallest size of the difference that's different, the lower bound on your $q$ is the same as for "no difference at all". ... ctd – Glen_b Oct 25 '15 at 22:43
  • ctd ... Your last two sentences suggest you might misunderstand how equivalence tests work. One doesn't specify what counts as significantly different - indeed the word "significant" doesn't enter into that definition of equivalence. It might be worth looking at the logic of equivalence tests. – Glen_b Oct 25 '15 at 22:46
  • Thank you! I think I understand a bit better why there isn't a simple answer. – Rilvabin Oct 26 '15 at 17:51

0 Answers0