Highly significant Pearson's chi-squared test (goodness of fit) when observed & expected are very close

Question

I have two arrays that I would like to do a Pearson's chi-squared test (goodness of fit). I want to test whether or not there is a significant difference between the expected and observed results.

observed = [11294, 11830, 10820, 12875]
expected = [10749, 10940, 10271, 11937]

I want to compare 11294 with 10749, 11830 with 10940, 10820 with 10271, etc.

Here's what I have

>>> from scipy.stats import chisquare
>>> chisquare(f_obs=[11294, 11830, 10820, 12875],f_exp=[10749, 10940, 10271, 11937])
(203.08897607453906, 9.0718379533890424e-44)

where 203 is the chi square test statistic and 9.07e-44 is the p value. I'm confused by the results. The p-value = 9.07e-44 < 0.05 therefore we reject the null hypothesis and conclude that there is a significant difference between the observed and expected results. This isn't correct because the numbers are so close. How do I fix this?

Answer 1

@NickCox has done a good job pointing out that your expected values don't match your observed values, in that they don't sum to the same total. But I gather your question is motivated by surprise that the chi-squared test is significant (which holds even with the presumably correct expected values), given that the proportions seem so similar.

x = matrix(c(11294,11830,10820,12875, 11464.50625,11668.22015,10954.68822,12731.58537),
           nrow=2, byrow=T)
p = apply(x, 1, function(y){ y/(.5*sum(x)) })
t(p)
#           [,1]      [,2]      [,3]      [,4]
# [1,] 0.2412269 0.2526752 0.2311028 0.2749952
# [2,] 0.2448687 0.2492198 0.2339795 0.2719320
chisq.test(x[1,], p=p[,2])
#  Chi-squared test for given probabilities
# 
# data:  x[1, ]
# X-squared = 8.0504, df = 3, p-value = 0.04498
windows()
  barplot(p, horiz=T, xlab="cumulative proportion")
  axis(side=2, at=c(.66,2), labels=c("observed", "expected"))

This reflects a very common misunderstanding of the nature of hypothesis testing and $p$-values. It is true that the proportions are very similar, but they are not identical—they differ at the third decimal place. Moreover, you have an enormous amount of data ($N = 46819$). When you test a hypothesis (e.g., by running a chi-squared test), the $p$-value conflates how far your data are from the expected values with how uncertain you are that the difference exists. Because you have so much data, in your case we can be reasonably certain that data as far (not very!) from the null are somewhat unlikely. This is simply the way hypothesis testing works when you have very large $N$ (see this excellent CV thread: Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples?). In other words, there is nothing to "fix" in your results (other than the inconsistent values).

If you are interested in a measure of how far your observed values are from the expected values that is uncontaminated by $N$, you could use the Cramer's $V$ for a goodness of fit test (see @Alexis' answer here: Understanding $χ2$ and Cramér's $V$ results): $$ {\rm Cramer's}\ V = \sqrt{\frac{\chi^2}{N}} = \sqrt{\frac{8.0504}{46819}} = 0.013 $$

Answer 2

Here is a calculation using Stata. Note that the expected frequencies have been scaled so that they have the same sum as the observed.

. chitesti 11294  11830  10820  12875 \ 11464.50625   11668.22015   10954.68822   12731.58537

observed frequencies from keyboard; expected frequencies from keyboard

         Pearson chi2(3) =   8.0504   Pr =  0.045
likelihood-ratio chi2(3) =   8.0536   Pr =  0.045

  +--------------------------------------------+
  | observed    expected   obs - exp   Pearson |
  |--------------------------------------------|
  |    11294   11464.506    -170.506    -1.592 |
  |    11830   11668.220     161.780     1.498 |
  |    10820   10954.688    -134.688    -1.287 |
  |    12875   12731.585     143.415     1.271 |
  +--------------------------------------------+

The very high chi-square statistic and very low P-value look like just a side-effect of feeding the software with an inappropriate set of expected frequencies. That is, big discrepancies between observed and expected occur because the expected frequencies are much too low. But it is surprising, indeed shocking, that the Python software doesn't check that.

Detail for Stata users only: chitesti is downloadable via ssc inst tab_chi.

Answer 3

Finding simple differences may be done like so:

observed = [11294, 11830, 10820, 12875]
expected = [10749, 10940, 10271, 11937]

for ob, ex in zip(observed, expected):
    print ob, ex, ob-ex

(Adjust it to your particular needs)