5

I'm trying to test the difference in proportions using the z test method and chi-squared method, but am getting very different answers. Is that normal?

My data:

        CI     CII
Male    205    102
Female  83     39

Calculating the z score I get 0.25 which should correlate to a p-value of 0.4013. Calculating the chi-squared score I get 0.0626 correlating to a p-value of 0.8025.

I read that the z-score requires some assumptions (probability of success is ~0.5 and n is high). Is this violating those? Or is it just the nature of these different approaches that gives very different answers with the same meaning (no evidence of difference).

I'm certainly open to miscalculations, but I've re-checked. If this behaviour isn't normal I'll recheck again!

Here are my calculations in R.

> r1 <- 205
> r2 <- 102
> n1 <- 288
> n2 <- 141
> (p1 <- r1/n1)
[1] 0.7118056
> (p2 <- r2/n2)
[1] 0.7234043
> (common.proportion <- (r1+r2)/(n1+n2))
[1] 0.7156177
> (se.pooled <- sqrt(common.proportion*(1-common.proportion)*(1/n1+1/n2)))
[1] 0.0463676
> (zscore <- (p1-p2)/se.pooled)
[1] -0.2501466
> 
> # chi-squared
> prop.test(c(205,102), c(288,141), correct = FALSE)

    2-sample test for equality of proportions without continuity
    correction

data:  c(205, 102) out of c(288, 141)
X-squared = 0.0626, df = 1, p-value = 0.8025
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.10208385  0.07888645
sample estimates:
   prop 1    prop 2 
0.7118056 0.7234043
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Tom
  • 1,511
  • 1
  • 12
  • 17
  • A related issue. If I'm presenting confidence intervals around each proportion calculated using the binomial distribution, but then comparing them and presenting a p-value using chi-squared, that seems a bit wrong. I might have overlapping CIs, but then a p-value that's less than 0.05. Am I thinking correctly? – Tom Mar 13 '15 at 04:29
  • 3
    Tom, can you show your math? These two tests should give very similar results for your sample sizes (especially if making the same choice about continuity corrections). – Alexis Mar 13 '15 at 05:19

1 Answers1

7

Very simple: both the z test and the contingency table $\chi^{2}$ test are two tailed tests, but you have got the one-sided $p$-value for your z test statistic. That is for $H_{0}: p_{1} - p_{2} = 0$, the $p$-value = $P(|Z| \ge |z|)$, but your reported $p$-value is only $P(Z \le z)$.

Notice that $0.4013 \times 2 \approx 0.8025$. Easy!

Alexis
  • 26,219
  • 5
  • 78
  • 131
  • 5
    And the square of the *Z*-score is (−0.2501466)² = 0.06257, which equals the test statistic `X-squared` from the `prop.test()` output. – Karl Ove Hufthammer Mar 13 '15 at 18:55
  • 1
    Thank you for that "easy" answer! (Which prompted a good schooling on one-tailed tests. And now @KarlOveHufthammer you're sending me down another schooling. If `x-squared` is simply the z score squared, why do we even have it? And why isn't it called `z-squared`? (Obviously, not to be answered here. I have a lot to learn!) – Tom Mar 16 '15 at 06:29
  • 2
    @Tom The reason that it’s called `X-squared` in the R output, is that the X is really an ASCII interpretation of the greek capital letter chi (Χ) (the lowercase version of this letter looks like this: χ). And it’s a *chi-squared* test, which is used in lots of other situations. That said, the test(s) should never be used for comparing binomial proportions, as they have terrible statistical properties. See https://stats.stackexchange.com/questions/82720/confidence-interval-around-binomial-estimate-of-0-or-1/82724#82724 – Karl Ove Hufthammer Mar 16 '15 at 16:42