1

I have the following data:

There are three conditions, A, B and C (presented in random order). Each condition has two difficulties (i.e., six total). Each respondent is scored as 1 (if correct), or 0 (if incorrect). Each respondent did each condition once. With this we get the following:

Condition          Correct
A Easy             19/29
A Hard             16/29
B Easy             10/29
B Hard              0/29
C Easy             13/29
C Hard              6/29

I want to statistically prove that respondents are more correct in one condition compared with another. I stumbled upon the McNemar's test, but since B Hard is a constant, that will not work. Any other suggestions?

Swifting
  • 11
  • 3
  • So each respondent provides 6 responses, is that right? How many respondents do you have in total? Did no one ever get B Hard correct? – gung - Reinstate Monica Aug 11 '15 at 20:43
  • Yes, nobody had B Hard correct, there were 29 respondents, so in B Hard 29 were incorrect. – Swifting Aug 11 '15 at 20:47
  • Are you interested in testing the interaction `Condition X Difficulty`? – gung - Reinstate Monica Aug 11 '15 at 21:02
  • Are you trying to determine if the Hard conditions are statistically more difficult than the Easy ones? Are you trying to compare A to B to C? – MikeP Aug 11 '15 at 21:02
  • Both, comparing Easy with Hard, and A with B. – Swifting Aug 11 '15 at 21:04
  • So you don't need the interaction, just the main effects? – gung - Reinstate Monica Aug 11 '15 at 21:16
  • The effect of ABC + Difficulty on the amount of correct, and with it I can compare them and say which one is most effective? – Swifting Aug 11 '15 at 21:26
  • Conditional logistic or log-linear regression is used for paired or stratified data situations. This tabulation has broken the association of responses within subjects. I suppose it might be necessary to use exact methods to make some analyst completely comfortable with a zero in one of the conditions, but generally the approximate methods give adequate results. – DWin Aug 12 '15 at 01:02

2 Answers2

0

Because your respondents are providing multiple scores, you need to account for the non-independence in your data. The primary two ways to do that are to fit a generalized linear mixed effects model (GLMM), or use the generalized estimating equations (GEE). To understand how these differ, and to decide which you might prefer, it may help you to read my answer here: Difference between generalized linear models & generalized linear mixed models in SPSS. My guess is that you would prefer the GLMM.

The catch is that GLMMs are finicky and sometimes just won't fit. The 0 corrects in B Hard is especially likely to cause problems. If you don't include an interaction term, you might be OK, though. If you did try to fit an interaction term you will have separation (also known as the Hauck-Donner effect), and the GLMM fit is unlikely to converge. Even if you don't include the interaction, and the fit does converge, you should probably fit nested models and use likelihood ratio tests instead of Wald tests.

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
-1

Have you considered Fisher's Exact Test? https://en.wikipedia.org/wiki/Fisher%27s_exact_test

Using it I get p-values for Hard vs Easy of 0.1545 for A, 0.0004 for B, and 0.0340 for C. Therefore, I think that you can probably reject the null hypothesis (no difference between easy and hard) for B and C, but not for A.

For A, I setup the table as follows:

enter image description here

choose(29,19)*choose(29,16)/choose(58,35) = 0.1545

Similar analysis (sum of easy and hard for each) shows all the types (A, B, and C) are different from one another (A and B p value is 1.4e-6, A and C p value is 0.0018, and B and C p value is 0.027).

MikeP
  • 2,117
  • 8
  • 9
  • Hmm but can you do a Fisher's Exaxt Test on paired data? – Swifting Aug 11 '15 at 23:15
  • I believe so, the examples I've seen generally are looking at whether there is a difference in one set of paired outcomes based on another set of paired characteristics. I also just noticed your comment from an hour ago about which test is most effective. I would say which ever is answered correctly closest to 50% of the time gives you the most information. – MikeP Aug 11 '15 at 23:25
  • -1, Fisher's exact test does not take the non-independence of the data into account. This suggestion is inappropriate for the OP's situation. – gung - Reinstate Monica Aug 12 '15 at 03:57