2

I am running a series of binomial tests for different trials. I ran 319 trials, and no matter the number of successes, my p-value is still < 2.2e-16. This includes when my trial had 0 successes.

binom.test(0, 319, p=0.5, alternative = c ("two.sided"), conf.level = 0.95)

the results come out as...

Exact binomial test

data:  0 and 319
number of successes = 0, number of trials = 319, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.00000000 0.01149728
sample estimates:
probability of success 
                     0

A successful trial has the same p-value...

Exact binomial test

data:  21 and 319
number of successes = 21, number of trials = 319, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.04120874 0.09887112
sample estimates:
probability of success 
            0.06583072

Am I interpreting the results wrong? Please help!

Underminer
  • 3,723
  • 1
  • 20
  • 36
Christy Judd
  • 21
  • 1
  • 2
    You are testing the null that the true rate is 50%. Both 0 and 21 in 319 trials would be fairly unusual if the true rate was that high, which is why the p-value is small. Try 160 successes instead. – dimitriy Sep 09 '21 at 13:25
  • Take a look at this example: https://stats.stackexchange.com/a/72583. Hypothesis testing is very unintuitive. – dimitriy Sep 09 '21 at 13:30
  • I love the neologism "bionimal" in the title: it sounds like a mechanical dog ;-). – whuber Sep 09 '21 at 13:31

2 Answers2

3

To help you understand the results, your binomial example is effectively testing if a coin is fair or not ($H_0: p = 0.5$) by flipping the coin 319 times.

The p-value represents the probability of obtaining results at least as extreme as what was observed if the null hypothesis was true.

Forgetting statistics and thinking logically, if you flipped a fair coin 319 times, how likely is it you would get 319 tails and 0 heads? Virtually impossible, never going to happen. In statistics that means the p-value is practically zero (or 2e-16 for a computer).

Again, if you flipped 319 times, how likely is it to get only 21 heads (or less)? Still virtually impossible, never going to happen. Again the p-value is still practically zero (or 2e-16).

If you change the number of successes to something more realistic (you would expect around half the flips to be heads with a fair coin), you will start getting results that support the null hypothesis being more reasonable (or more correctly, less evidence the null hypothesis is false). Try the same test with 150 successes and you will get a different p-value.

Underminer
  • 3,723
  • 1
  • 20
  • 36
  • 1
    Thanks for your help. I'm looking at the number of animals which are scarred. In one of the 'trials' 21/319 had scars. What would I put instead of 0.5, then? I figured it was essentially either a 'yes' or 'no' question, hence the 0.5. – Christy Judd Sep 09 '21 at 13:51
  • The 0.5 is your hypothesis, which means you are testing if half the animals are scarred. In the population you are testing/sampling from, that null hypothesis proved false (because you found much less than half to be scarred). You are supposed to choose a null hypothesis before you get data, but nevertheless your 95% confidence interval gives you an idea of what the true population proportion/percentage is of scarred animals (4.1%-9.9% percent of the whole population), which seems to be more valuable in your case. – Underminer Sep 09 '21 at 13:57
  • 2
    May I suggest "as least as extreme as" rather "as extreme as" since s <= 21 or s >= 298 are also included? Strictly speaking, binomialp(319,21,0.5) is just 3.552e-64. – dimitriy Sep 09 '21 at 14:20
  • I think that the technical question is trying to solve a business problem. The business question seems to be about the prevalence or confidence interval given the observed scars vs. total observed. I don't know if they author is using the right tool for the job. Maybe something like this: (https://epitools.ausvet.com.au/ciproportion) I'm seeing 0.0658 as prevalence and 0.0930 to 0.0989 as upper 95% CL. – EngrStudent Sep 09 '21 at 16:59
  • "I'm looking at the number of animals which are scarred." You may want to look at confidence intervals for the percentage of scarred animals alone rather than p-values for a comparison against a hypothesized percentage of scarred amimals. – cbeleites unhappy with SX Sep 09 '21 at 18:08
  • @DimitriyV.Masterov I often simplify language when answering basic questions for the sake of brevity. Nonetheless, I've adopted your suggestion now that the original poster has an answer and the question has presumably entered 'legacy' status. – Underminer Sep 14 '21 at 04:19
1

The 319 trial binomial test is coming back close to zero as there's very little chance of getting that outcome. Probably what you really want is to test for that number of successes or more. Here's how to construct a table of probabilities for 100 tests, just change the 100 to 319 or whatever number is required. The table will include left tails, right tails, equal, and not equal, so just select as desired.

library(data.table)
DT=data.table(cbind(Heads=seq(100,0,by=-1), Tails=seq(0,100,by=1)))
DT[, BinomL := pbinom(Heads-1, Heads+Tails, 0.5), keyby=Heads]
DT[, BinomR := pbinom(Heads, Heads+Tails, 0.5, lower.tail=FALSE)]
DT[, BinomEQ := pbinom(Heads-1, Heads+Tails, 0.5, lower.tail=FALSE) - pbinom(Heads, Heads+Tails, 0.5, lower.tail=FALSE)]
DT[, BinomLR := BinomL + BinomR]
DT[, BinomTot := BinomL + BinomR + BinomEQ]
View(DT)

The C code underlying R binom/pbinom/etc. uses algorithm 708, so in theory you should have plenty of digits of precision.

C8H10N4O2
  • 176
  • 4