Which test for cross table analysis: Boschloo or Barnard?

Question

I am analyzing a 2x2 table from a small dataset of 30 patients. We are retrospectively trying to find some variables that give a hint about which treatment to choose. The variables (obs normal / strange) and treatment decision (A/B) are of special interest and therefore the data looks like this:

\begin{array} {|r|r|r|r|} \hline \text{Obs/Tr. Dec.} &\text{A} &\text{B}\\ \hline \text{normal} &12 &13 &25\\ \hline \text{strange} &0 &5 &5\\ \hline &12 &18 &30\\ \hline \hline \end{array}

Obviously one cell lacks on entries which excludes a chi-squared test and Fisher's exact test doesn't give a saturating p-value (but still <10%). So my first idea was to find a test with a greater power and I was reading in a blog and in this article about Barnard's and Boschloos test, that in general there are three scenarios which yield to a powerful test:

1. Column and Rowsums fixed $\rightarrow$ Fisher's exact test
2. Column or (xclusive) Rowsums fixed $\rightarrow$ Barnard's exact Test
3. None are fixed $\rightarrow$ Boschloos's exact Test

The article above pointed out that the sum of treatment A and treatment B are almost never known before, so we can exclude Fisher's exact test. But what about the other alternatives? In case control where we have healthy controls we can control the placebo and verum group which numbers we can control, so one would choose 2: Barnard. In my case I am not sure, because on one the hand we have a similar mathematical problem (sum of observation levels equivalent to sum of placebo / verum), which leads to Barnard but the design is different, because we can't control the nr. of observation normal/strange before taking the sample which leads to 3: Boschloo.

So which test should be used and why? Of course I want high power.

(Another question that I would like to know is, if in case of chisq.test in r it wouldn't be better to use prop.test(x, alternative = "greater")? The theoretical aspects are explained here.)

Answer 1

There may be some confusion about term "Barnard"s test or "Boschloo"s test. Barnard's exact test is an unconditional test in the sense that it does not condition on both margins. Therefore, both the second and third bullets are Barnard's test. We should instead write:

1. Both margins fixed (Hypergeometric Dist'n)→Fisher's exact test
2. One margin fixed (Double Binomial Dist'n)→Barnard's exact test
3. No margins fixed (Multinomial Dist'n)→Barnard's exact test

Barnard's exact test encompasses two types of tables, so we distinguish the two by saying "binomial" or "multinomial" model as appropriate.

Typically, Barnard's exact test either uses a Z-pooled (aka Score) statistic to determine the 'as or more extreme' tables. Note the original Barnard paper (1947) uses a more complicated approach to determine the more extreme tables (referred as "CSM"). Boschloo's exact test uses Fisher's p-value to determine the 'as or more extreme' tables. Boschloo's test is uniformly more powerful than Fisher's exact test.

For your dataset, it sounds like neither margins were fixed, so would recommend using Boschloo's exact test with a multinomial model. I found Boschloo's test slightly better for unbalanced margin ratios (although typically very similar to Barnard's exact test with Z-pooled statistic). However, since both Boschloo's test and multinomial models are much more computationally intensive, you can also use the binomial model (the reasoning for why this would still be appropriate is a little complicated; to briefly summarize, the margins are an approximately ancillary statistic, so it's alright to condition on margin). For more details on the exact tests and information on implementation, please use the Exact R package (https://cran.r-project.org/web/packages/Exact/Exact.pdf). I am the author of the package and it's a more updated version of the code on the blog.