Interpretation of chi-squared on proportion

Question

I have this array:

name	responsive	not responsive
x	16	95
y	5	83

I am trying to compare the proportion of X_responsive with Y_responsive but the classic z-test of proportions can't be used here because my data is not normally distributed. I have used a chi-square test of proportions instead but I'm not sure what the output means in terms of my table.

import statsmodels.stats.proportion as ssp
(chi2, p, arr) = ssp.proportions_chisquare(count = df_obs.non_response, nobs = df_obs.sum(axis = 1))
print((chi2, p))

Chi2 = 3.96557520785049
p = 0.046439653061788856

I know it is significant but I'm not sure what is this is interpreted as.

You have binary variables, so you will never have normal data. Why do you think the z-test is inappropriate? // The $\chi^2$ test is a fine test to use, whether you think th z-test is appropriate or not. What confuses you about the output? — Dave, Aug 04 '21 at 13:56
What exactly the chi-square is revealing here. Is it just that there is a relationship between name and responsiveness? - if so, how can i determine the direction? whether is more responsive or y is more responsive? — 1lapint3, Aug 04 '21 at 14:01
It will help if you go back to the z-test to explain why you think it does not apply and what you would get out of it if it did apply. — Dave, Aug 04 '21 at 14:07
Yes with the z-test of two proportions the null hypothesis that the two proportions are equally responsive is rejected. However the test statistic gives me a direction which tells me which is more responsive than the other. How do I determine which variable is most responsive in chi-square? or is this inferred from the raw data — 1lapint3, Aug 04 '21 at 14:21
Yes, you determine that from the raw data. Again, though, why do you think the z-test does not apply? — Dave, Aug 04 '21 at 14:31
I ran a shapiro wilks test on the X and Y values, for which the raw data is continuous and numerical. We used these values to categorise them into X and Y. Once they become binary, does it not matter about the normality? — 1lapint3, Aug 04 '21 at 14:40
Once they become binary, they *cannot* be normal. Normal distributions can take any real value, and binary distributions only take two values (quite a lot less than $\infty$). The bigger issue, though, is that you have destroyed information by binning your variables. This may warrant a separate question, but what were your starting $X$ and $Y$, and what question(s) do you want to use $X$ and $Y$ to answer? — Dave, Aug 04 '21 at 14:43
X and Y had many values from -1 to 1. Of which, any that were X>0.4 were classified as one variabile and any Y>0.5 classified as another variable. I did this to see if values that were classified as X and Y were different in terms of responsiveness. My question does not warrant any investigation into X<0.4. The z-test may be better because it looks at the proportions of x>0.4 / all x-values? — 1lapint3, Aug 04 '21 at 14:50
You should post a separate question about your binning of the raw data. Perhaps it makes sense for your task, but it usually destroys information. — Dave, Aug 04 '21 at 14:53
Because you used the $X$ and $Y$ values to determine the binning, the p-value no longer is correct. See https://stats.stackexchange.com/a/17148/919 for an example of what can go wrong. — whuber, Aug 04 '21 at 15:09
Thank you both! this was helpful! I assume I run into the same problems if i keep my dichotomised variables and run the z-test? — 1lapint3, Aug 04 '21 at 15:32
The two tailed z-test and the chi-squared are *exactly the same test*, with *exactly the same assumptions*. — Glen_b, Aug 05 '21 at 05:07

BruceET · Answer 1 · 2021-08-05T07:36:27.507

Comment: Illustration of @Glen-b's Comment about the equivalence of prop.test and chisq.test for your data.

The two procedures give exactly the same P-value:

prop.test(c(16,5), c(111,88))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(16, 5) out of c(111, 88)
X-squared = 3.0944, df = 1, p-value = 0.07856
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.004154855  0.178806780
sample estimates:
    prop 1     prop 2 
0.14414414 0.05681818 

TBL = rbind(c(16,5),c(95,83))
chisq.test(TBL)

        Pearson's Chi-squared test 
        with Yates' continuity correction

data:  TBL
X-squared = 3.0944, df = 1, p-value = 0.07856

They are also the same if the continuity correction is not used.

prop.test(c(16,5), c(111,88), cor=F)$p.val
[1] 0.04643965
chisq.test(TBL, cor=F)$p.val
[1] 0.04643965

However, there are several versions of the test for equality of two binomial proportions. Some versions use $H_0$ to argue for using a pooled sample proportion $\hat p = \frac{x_1+x_2}{n_1+n_2}$ to get the standard error for $\hat p_1 - \hat p_2$ and some use separate estimates $\hat p_i = x_i/n_i$ for this purpose. Also, various computer programs use different kinds of continuity corrections.

Also, if counts are too small for an accurate P-value in chisq.test, then R allows the option to simulate a more accurate P-value. (Simulation is not supported for prop.test.)

Finally, the Fisher Exact Test can give a different P-value than any of the above.

fisher.test(TBL)$p.val
[1] 0.06220786

Simulated p-values in chisq.test tend to be close to the Fisher p-value, especially if you use more than the default number of iterations to simulate.

Table with small counts:

TAB = rbind(c(40, 3), c(60, 7));  TAB
     [,1] [,2]
[1,]   40    3
[2,]   60    7

chisq.test(TAB)

        Pearson's Chi-squared test 
        with Yates' continuity correction

data:  TAB
X-squared = 0.077317, df = 1, p-value = 0.781

Warning message:
In chisq.test(TAB) : 
 Chi-squared approximation may be incorrect

chisq.test(TAB, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 0.38181, df = NA, p-value = 0.7276

More iterations:

chisq.test(TAB, sim=T, B = 5000)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 7000 replicates)

data:  TAB
X-squared = 0.38181, df = NA, p-value = 0.744

fisher.test(TAB)

        Fisher's Exact Test for Count Data

data:  TAB
p-value = 0.7374
alternative hypothesis: 
 true odds ratio is not equal to 1
95 percent confidence interval:
 0.3292961 9.8349592
sample estimates:
odds ratio 
  1.549608

Interpretation of chi-squared on proportion

1 Answers1