1

Someone I know told me that he had performed $\chi^2$ test of independence with Yates's correction on a $2\times n$ contingency table, seen that $p\lt10\%$ (good enough for his purposes), and calculated adjusted residuals as described by Sharpe, q.v., only to have none of the categories show up with sufficiently high residuals to stand out. He asked me what that means. I told him that I didn't think that that was possible but that he should probably look to the higher (absolute) values of the adjusted residuals.

  • Am I right that that's not possible (so there was some error in the calculation)?
  • If it is possible, what's the answer to his question?
msh210
  • 187
  • 11
  • 3
    This happens all the time, especially with such relaxed standards of significance. The underlying intuition is that a group of tiny deviations from independence can suffice, in the aggregate, as evidence of lack of independence while none of the deviations alone is adequate evidence. It is *exactly* the same phenomenon as having a [significant overall regression (F) statistic without any individual coefficient being significant](https://stats.stackexchange.com/questions/3549/why-is-it-possible-to-get-significant-f-statistic-p-001-but-non-significant-r). Indeed, that's a duplicate thread. – whuber Jan 16 '19 at 13:00
  • 1
    Minor point: Yate's correction makes Pearson's $\chi^2$ act more like Fisher's "exact" test, which is conservative. Don't use continuity correction here. And instead of residuals, look at the raw stratified proportions. More important: don't put such emphasis on statistical significance, as Peter stated below. – Frank Harrell Jan 16 '19 at 13:04
  • @whuber many thanks. Those answers all point to multicollinearity, which seems irrelevant here. – msh210 Jan 16 '19 at 13:16
  • [My answer](https://stats.stackexchange.com/a/14528/919) points out that multicollinearity indeed is irrelevant. It is, however, almost always relevant in ANOVA unless you have used a carefully designed experimental arrangement of factors (to achieve orthogonality). – whuber Jan 16 '19 at 13:28

1 Answers1

2

No, you are not right and yes, this is possible.

One way it is possible is that N is large. With a large enough N, even tiny differences are statistically significant.

There's no "answer" except to not rely so much on statistical significance.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276