1

We have been having a discussion about the significance of 2 accidents. If I have 2 groups:

  • Group 1 is a baseline where an accident has occurred 70 times in 3,500,000 and
  • Group 2 where an accident has occurred 2 times in 14,000

How confident can I be that an accident is more likely in group 2?

Can I do a valid calculation with only 2 events in group 2?

Is the difference between the groups statistically significant?

Thanks for any suggestions.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
Andrew98787
  • 13
  • 2

1 Answers1

1

The standard approach for a question like this is to calculate a for the mean accident rate in your two groups and check whether they overlap. Typically, one would use 95% CIs.

Now, things that can happen or not in $n$ independent samples, like accidents, are binomially distributed. So we need to calculate CIs for the binomial parameter $p$.

You have a rather small "success rate", i.e., accident rate. In such cases, the standard normal distribution approximation for $p$ breaks down, and you need somewhat more elaborate ways of calculating CIs. Brown et al. (Statistical Science, 2001) recommend Agresti and Coull for large $n$ (which you certainly have, with your numbers of trials).

We can use binom.confint() in the binom package for R to calculate the intervals and plot them for good measure:

> library(binom)
> 
> (CIs <- binom.confint(x=c(70,2),n=c(3500000,14000),methods="agresti-coull"))
         method  x       n         mean        lower        upper
1 agresti-coull 70 3500000 0.0000200000 1.579976e-05 2.529776e-05
2 agresti-coull  2   14000 0.0001428571 2.883566e-06 5.570670e-04
> 
> plot(range(CIs[,c("upper","lower")]),c(0.7,2.3),type="n",xlab="",ylab="",yaxt="n")
> axis(2,2:1,paste("Group",1:2),las=2)
> lines(x=CIs[1,c("upper","lower")],y=rep(2,2),lwd=2)
> lines(x=CIs[2,c("upper","lower")],y=rep(1,2),lwd=2)
> points(CIs[,"mean"],2:1,pch="|",col="red")

CIs

Note how the CI for group 2 is much wider than that for group 1, because the sample size is much smaller. (But look at the horizontal axis: both CIs are still tiny. We can be reasonably sure that the accident rate in group 2 is still less than $6\times 10^{-4}=.0004$.) Importantly, the CIs overlap, so based on this analysis, we cannot reject the null hypothesis that the accident rates are identical in both groups.

See the following two earlier threads on binomial CIs:

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Thank you for the detailed answer. The practical side of the problem is a debate about whether group 2 is failing to meet the expected standard, and whether action is justified to try to improve it. "Cannot reject the null hypothesis" would be used to argue that no action should be taken - it's basically the same argument as 2 is too small a number to draw any conclusions. But I think the CI shows that it is more likely that there is a difference, and it could in fact be much larger than already observed? Is that a fair statement? – Andrew98787 Aug 28 '19 at 11:04
  • Both are fair statements: on the one hand, the observed mean in group 2 is quite a bit larger than in group 1. On the other hand, we simply can't tell whether this is systematic, or simply due to chance. Because we have a rather small sample and a tiny accident rate, it's simply hard to estimate the underlying rate. As a sensitivity check, you might re-run the analysis with 1 (or 3) accidents in group 2. If the number of accidents might easily have been 1 or 3, you could think about whether you are comfortable basing your decisions on a single accident happening or not, out of 14,000. – Stephan Kolassa Aug 29 '19 at 11:08