0

Consider tossing two unfair coins 100 times:

How can I know using the two sets of Heads or Tail results whether the two coins have an equal (or similar) bias with a 95% confidence?

I am currently thinking the way is to do this through a student's t-test, though the p-value is off when both coins have a probability 1 of flipping to heads. (Resulting in no variance between the sets of coins)

The student's t-test assumes that the data is normally distributed, is it ok anyway to use this data because of the central limit theorem? (or should I repeat these 100 flip experiments and use those for a t-test)

My current null hypothesis is that both coins are equally biased, so the alternate hypothesis would be that the coins are differently biased.

speedy_turtle
  • 1
  • 3
    This is just a two-sample proportion test, something like prop.test in R. – Dave Nov 26 '21 at 18:26
  • 1
    Thank you guys for the help, I don't know too much about R as I am mainly using scipy, but it seems like a [two-sample z-test](https://sonalake.com/latest/hypothesis-testing-of-proportion-based-samples/#:~:text=%C2%A0-,2%2Dsample%20z%2Dtest,-Compare%20the%20proportions) is the way to go! Much better than using the T-Test for this application – speedy_turtle Nov 26 '21 at 18:44
  • Specifically, if one coin gives 55 Heads in 100 tosses and the other gives 35 Heads in 100 tosses, then R code `prop.test(c(55, 35), c(100,100))$p.val` gives P-value $0.006922651, providing strong evidence of different biases. Also R code `prop.test(c(55, 35), c(100,100))$conf.int` gives 95% Ci for difference in Heads probabilities as $(0.0549188, 0.3450812).$ // Note that `prop.test` in R essentially uses a normal approximation, so you'd need sufficiently large sample sizes as in my example here. (You could use Fisher's exact test as in @EdM's link for small samples.) – BruceET Nov 26 '21 at 18:46

1 Answers1

1

For exact inference on each coin you can invert the CDF of a binomial distribution to construct a confidence interval. Here is a related thread that discusses this idea. For comparing the coins to each other to see if there is evidence that the long-run probability of heads differs between the two coins you can invert either the two-sample Wald, score, or likelihood ratio test to construct a confidence interval. You could consider using a log or logit link function to improve the operating characteristics of the test. This will have you investigating whether the rate ratio or the odds ratio equals 1 instead of the difference in rates equalling 0. Here is an example investigating a single proportion by inverting a Wald test with a log link or a logit link.

If you prefer ease of implementation and your sample size is not small then you can use an identity link function while inverting a one-sample and two-sample Wald test:

$100(1-\alpha)\%$ CI for $p_1$ and $p_2$: $$\hat{p}_j\pm z_{1-\alpha/2}\sqrt{\hat{p}_j(1-\hat{p}_j)/n_j}, \hspace{3mm}j=1,2 $$

$100(1-\alpha)\%$ CI for $p_2-p_1$: $$\hat{p}_2-\hat{p}_1 \pm z_{1-\alpha/2}\sqrt{\hat{p}_2(1-\hat{p}_2)/n_2 + \hat{p}_1(1-\hat{p}_1)/n_1} $$

where $z_{1-\alpha/2}$ is the $100(1-\alpha/2)^{th}$ percentile of the standard normal distribution and $j$ indexes the two coins. For a 95% CI you would use $z_{1-\alpha/2}=1.96$.

If the $100(1-\alpha)\%$ confidence interval for the difference in proportions contains 0 then the hypothesis $H_0: p_2-p_1=0$ is plausible at level $\alpha$, as are all of the other hypotheses that make up the confidence interval.

Geoffrey Johnson
  • 2,460
  • 3
  • 12
  • 3
    The question is specific. Offering a litany of possible procedures is unlikely to be perceived as a useful response. Which of all these methods would you recommend and exactly how would it apply? – whuber Nov 26 '21 at 19:56