Understanding $\chi^{2}$ and Cramér's $V$ results

Question

I have two groups of buyers, A and B, and I want to test whether the difference between the percentage of them who would buy a product is significant.

• Group A: 271 out of 2520 bought the product (10.8%) and 2,249 didn't buy.
• Group B: 1,073,839 out of 41,873,457 bought the product (2.6%) and 40,799,618 didn't buy.

I have used chisq.test() to conduct a $\chi^{2}$ test to answer the question of whether the percentage of buyers across my groups has a significant difference (I would say 10.8% and 2.6% are different enough).

library(vcd)
data <- rbind(x=c(271,1073839), n=c(2249, 40799618))
chisq.test(data)
#  Pearson's Chi-squared test with Yates' continuity correction
# 
# data:  data
# X-squared = 672.9477, df = 1, p-value < 2.2e-16
assocstats(data)
#                     X^2 df P(> X^2)
# Likelihood Ratio 382.03  1        0
# Pearson          676.22  1        0
# 
# Phi-Coefficient   : 0.004 
# Contingency Coeff.: 0.004 
# Cramer's V        : 0.004

So the $p$-value says there is a significant difference, the Cramer's $V$ says there isn't. How is it possible that with such a big difference in proportions (group A has more than 4 times more in sales) there is no significant difference according to Cramér's $V$?

Answer 1

There is a fundamental confusion here:

• The $p$-value you got comes from the $\chi^2$ test. It tells you the probability of getting a $\chi^2$ statistic as extreme or more extreme than yours if the null hypothesis is true. It tells you nothing about how big the effect is.
• On the other hand, Cramer's $V$ is a measure of effect size. It tells you how big the effect is. It tells you nothing about whether or not the effect is 'significant'.

In addition, Cramer's $V$ isn't really the right measure of effect size for you, although it can be appropriate for many $\chi^2$ analyses. It assumes that neither the row variable nor the column variable is fixed. In your case, the columns are two groups that could be understood as an independent variable, whereas the rows are the number who bought or didn't (this can be thought of as a dependent variable). That is, you think of your rows and columns differently, so you want a measure of effect size that does so as well. There are generally three such measures: risk differences, risk ratios, and odds ratios. Risk differences are the easiest for people to understand, but odds ratios are arguably the best measure. For your data, those values are (notice the denominators vary between the risk ratio and the odds ratio):

## risk difference: 
(271/2520) - (1073839/41873457)
# [1] 0.08189482
## risk ratio: 
(271/2520) / (1073839/41873457)
# [1] 4.19342
## odds ratio: 
(271/2249) / (1073839/40799618)
# [1] 4.578221

Answer 2

You do not say how you obtained your Cramér's V statistic, and while you give the $p$-value for your $\chi^{2}$ statistic, you do not provide the value of $\chi^{2}$ itself. However, the math for Cramér's V is straightforward:

$$V = \sqrt{\frac{^{\chi^2}/_n}{\min\left(\text{rows-1, cols-1}\right)}}$$

which in your case simplifies to:

$$V=\sqrt{\frac{\chi^{2}}{n}}$$

By definition, the $p$-value of V is equal to the $p$-value of the corresponding $\chi^{2}$ (as you can plainly see in the edit to your question which now includes the output from the chisq.test command).

Also: given your humongous sample size, it is not at all surprising that you found a significant difference. All differences are significant with a large enough sample size.

Answer 3

When I check your work I also obtain .004 for Cramer's V. This statistic, like many in the "variance explained family," tends to be low when you have two groups of very different size. The fact that group membership (A vs. B) hardly ever varies from B means that group membership has little opportunity to explain much of the variance in the buying behaviour. For that reason, it does not seem like the most effective indicator in this situation--at least, not when used alone.

There are other approaches. For example, using the common terminology of risk, the risk ratio (A to B) is 4.2. Group B shows a 76% reduction in risk. The odds ratio is 4.6. You could look into other effect size indicators such as at this starting point. But you may need to be prepared to show how your choice of indicators does not involve cherry-picking the ones most favorable to your cause.