17

I'm conducting a chi-square goodness-of-fit (GOF) test with three categories and specifically want to test the null that the population proportions in each category are equal (i.e., the proportion is 1/3 in each group):

                OBSERVED DATA
Group 1     Group 2     Group 3     Total
  686              928            1012        2626

Thus, for this GOF test, the expected counts are 2626(1/3) = 875.333 and the test yields a highly-significant p-value of < 0.0001.

Now, it's obvious Group 1 is significantly different from 2 and 3, and it's unlikely that 2 and 3 are significantly different. However, if I did want to test all of these formally and be able to provide a p-value for each case, what would be the appropriate method?

I've searched all over online and it seems there are differing opinions, but with no formal documentation. I'm wondering if there is a text or peer-reviewed paper that addresses this.

What seems reasonable to me is, in light of the significant overall test, to do z-tests for the difference in each pair of proportions, possibly with a correction to the $\alpha$ value (maybe Bonferroni, e.g.).

Meg
  • 1,633
  • 4
  • 18
  • 28
  • t-tests would not be suitable. You could do pairwise goodness of fit tests (proportions tests). What differing opinions did you find? – Glen_b Jan 27 '15 at 00:11
  • Sorry - I meant z-test (for difference in two proportions). I'll edit. – Meg Jan 27 '15 at 00:16
  • This link says to group all other groups vs. the one of interest (it's for the Fisher's exact test, but this link is redirected from another link about the chi-square, where the author says to apply the same method for the chi-square as for the Fisher's exact): http://www.biostathandbook.com/exactgof.html#posthoc But this isn't really what I want - I want pairwise, not one group against all others. – Meg Jan 27 '15 at 00:22
  • 1
    Most other sources I find talk about a contingency table setting, not a GOF test. – Meg Jan 27 '15 at 00:28
  • Yes, you could do proportions tests (whether done as one-sample z-test or binomial test, or chi-square test) of each pairwise comparison. You have no need to do one-vs-all comparisons. – Glen_b Jan 27 '15 at 00:34
  • @Glen_b: Thank you for your response. Is this suggestion from your experience doing this same thing, or do you know of a source that suggests that? Would you correct the alpha value at all for multiple comparisons? – Meg Jan 27 '15 at 12:55
  • Its the most obvious pairwise comparison. Personally, I'd typically avoid multiple comparisons - and indeed, formal hypothesis tests altogether in most cases I deal with (focusing more on confidence intervals and if I do test, usually specifying contrasts before hand). On the rare occasions I do multiple comparisons, I would usually use a smaller alpha level, but not in all situations -- it depends on what its for. – Glen_b Jan 27 '15 at 14:42
  • Just revisited this post after it got an up-vote, and thought about this re: your comment directly above, Glen_b: Wouldn't you have the same multiple comparisons problem with CIs (since they're another inferential method that can also be used to test a hypothesis, just as the *p*-value can)? – Meg Oct 14 '17 at 16:24
  • If you're using CI's merely as a test then yes, they have the same issues as doing the test any other way does -- so if you're doing multiple tests and want to control familywise error you'd still have that issue. [That's not why I was suggesting CI's there.] – Glen_b Oct 14 '17 at 22:20
  • Okay - thanks. I wanted to clarify in case anyone else was reading (because CIs would have to be corrected too). I see you say you would specify contrasts beforehand, so presumably this is how you'd deal with multiple comparisons. I also agree that, in general, CIs are a better way to conduct hypothesis tests, or that they should at least be reported along with a *p*-value so the reader has all of the information instead of the simple dichotomy of a *p*-value. – Meg Oct 24 '17 at 13:40
  • No, CI's would not in general have to be corrected, because people don't normally use them as formal hypothesis tests. If there's no significance test there's no type I errors to compute familywise error rates over. What I was saying there was *if* you use them as significance tests, then they're significance tests and you may choose to worry about familywise error rates in that case. However, if you're focused on estimating effect size and calculating an interval around your estimate, there wouldn't be any particular need to focus on properties of tests you're not trying to do. – Glen_b Oct 24 '17 at 13:55
  • I agree that you don't have to worry about it if you're not doing formal tests. What I was saying is that *if* you report corrected *p*-values and you're choosing to also report CIs, they *would* need to be corrected, or the two would be inconsistent. – Meg Oct 26 '17 at 15:27

2 Answers2

8

To my surprise a couple of searches didn't seem to turn up prior discussion of post hoc for goodness of fit; I expect there's probably one here somewhere, but since I can't locate it easily, I think it's reasonable to turn my comments into an answer, so that people can at least find this one using the same search terms I just used.

The pairwise comparisons you seek to do (conditional on only comparing the two groups involved) are sensible.

This amounts to taking group pairs and testing whether the proportion in one of the groups differs from 1/2 (a one-sample proportions test). This - as you suggest - can be done as a z-test (though binomial test and chi-square goodness of fit would also work).

Many of the usual approaches to dealing with the overall type I error rate should work here (including Bonferroni -- along with the usual issues that can come with it).

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • Thank you for your advice and for posting as an answer. I, too, was a bit surprised this issue seemingly hasn't come up for the GOF case. – Meg Jan 27 '15 at 15:12
  • 1
    I surprised too as this issue isn't discussed. I came up to the same solution as Glen but still have doubts. First, each pair is not independent from the "global" sample. For example, imagine we have 70,16,14 so you suggest compare 16 and 14 against 15/15. However, in other observation it could be 72,14,14. i.e. the source of "superiority' in the pair could be not a counterpart in the pair. Second, should we apply some group adjustment like Bonferroni if the choices weren't actually indepedent? Third, should we distinguish if the choice was mutually exclusive or it was multiple choice? – Niksr Oct 31 '15 at 02:03
  • I curious, could be possible to employ Cochran Q-test with McNemar post-hoc for this purpose? It seems all conditions for this test are met: 1) control stage - even distribution 2) event - reaction on stimuli 3) this is pair comparison (between hypothetical random choice and actual choice) 4) null - reaction on stimulus is different from random – Niksr Oct 31 '15 at 03:48
  • `so you suggest compare 16 and 14 against 15/15` @Niksr, no. Glen compares the two groups as `50/50` percent. The 3rd group is excluded from the comparison. – ttnphns Oct 31 '15 at 12:14
  • Yes, I meant 16 and 14 are cases, not percents. – Niksr Oct 31 '15 at 17:55
3

I've had the same issue (and was happy to find this post). I now also found a short note on the issue in Sheskin (2003: 225) that I just wanted to share:

"Another type of comparison that can be conducted is to contrast just two of the original six cells with one another. Specifically, let us assume we want to compare Cell l/Monday with Cell 2/Tuesday [...] Note that in the above example, since we employ only two cells, the probability for each cell will be π_i = 1/2. The expected frequency of each cell is obtained by multiplying π_i = 1/2 by the total number of observations in the two cells (which equals 34). As noted previously, in conducting a comparison such as the one above, a critical issue the researcher must address is what value of alpha to employ in evaluating the null hypothesis."

Sheskin, D.J. 2003. Handbook of Parametric and Nonparametric Statistical Procedures: Third Edition. CRC Press.

Karen
  • 31
  • 2