2

I would like to compare two error rates (proportion of incorrect answers in questions with dichotomous response-format) for two different items, and I am looking for some advice on how to do that.

Lets assume:
Control Group (CG) N = 100
Experimental Group (EG) N = 120

Item A: was answered incorrectly by 20 out of 100 participants in the CG (= 20%) and by 50 out of 120 participants in the EG (= 42%). --> I can calculate a chi-squared test whether the proportion of incorrect answers is higher in the EG than in the CG.

Item B: was answered incorrectly by 33 out of 100 participants in the CG (= 33%) and by 80 out of 120 participants in the EG (= 67%). --> Again, a chi-squared test can be calculated.

Question:
I would like to calculate whether the increase in the error rate is more pronounced for Item A (diff 33% to 67% = +34%) than for Item B (diff 20% to 42% = +22%).

EG and CG were manipulated between subjects; all items were answered by all participants, i.e., manipulated within subjects.

Reflections:
My idea is that I have to take into account the differences in the baseline errors (CG). So I thought about taking the baseline errors as 100% and compare the increase in the EG for Item A (+250%) to the one in Item B (+242%).

I'd be very grateful for any comments on whether this procedure is correct/makes sense in principle, and, if so, how the two values may be compared. I'm also unsure whether I still have to take into account that the data are kind of dependent as both items were answered by all participants?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
grey
  • 625
  • 1
  • 4
  • 16

2 Answers2

1

I would not use either chi-squared tests or McNemar's tests. The chi-squared does not take the dependency in the data into account, and neither incorporates information about both items into a proper, unified analysis. Instead, I would use a generalized linear mixed effects model (GLMM). You will have a logistic regression model with fixed effects for group and item, and a group X item interaction. In addition, you will have random effects for the subjects to take their dependence into account. At a minimum, you should have a random intercept; you may want to have a random slope as well, I'm not sure. (If the model will fit, it probably wouldn't hurt.) The test of the interaction is the critical test of your hypothesis: if it is significant, then the increases in the rates significantly differ. The directionality of that estimate (i.e., a positive vs. negative coefficient) will tell you which is a larger decrement, but it will depend on which levels are taken as the reference levels of your variables. At any rate, which is larger is clear enough from your summary statistics, all you need is a test of its significance. One last thing to bear in mind is that you phrase your question in terms of raw risk differences, whereas the model will test odds ratios, you will need to be clear on that.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • Thank you very much for your response. I also thought about using a GLMM, but was unsure whether it is the right method. Your answer gives me confidence to pursue this idea. It is clear to me why group and item are fixed effects. However, I am quite confused about the concept of mixed effects, random intercepts (this is not the same as the intercept inserted together with the fixed effects, or is it?), and random slopes. Would you mind explaining me (non-statistician) the basic idea of these concepts? I'd be especially interested in what function they have with respect to the dependent data. – grey Nov 25 '14 at 21:03
  • Btw: Do I understand you right that even if the proportions of incorrect answers were corrected for the baseline error, they should not be tested against each other (e.g. with a z-test as proposed here: http://www.socscistatistics.com/tests/ztest/)? – grey Nov 25 '14 at 21:07
  • For general info about fixed vs random effects, I have a couple recent answers [here](http://stats.stackexchange.com/a/125112/7290) & [here](http://stats.stackexchange.com/a/122587/7290) that may be helpful. [This](http://stats.stackexchange.com/a/68753/7290) is particularly good (I'm using the biostatistics version). Whether you should use random slopes is a substantive judgment: Do you think different people will be more or less affected by the items? I don't really follow your last question, but the test of the interaction in the GLMM is all you need; your suggested approach isn't valid. – gung - Reinstate Monica Nov 25 '14 at 23:26
  • Thank you @gung for [this](http://stats.stackexchange.com/questions/125093/how-to-use-a-linear-model-with-two-factors-and-repeated-measures/125112#125112) answer, which I think I understood best. I'd be also glad about further reading tips (books/book chapters), with some additional information on the topic (again understandable for a non-statistician). But else I'll just work through some more Q/A in this fantastic forum! – grey Nov 26 '14 at 07:52
  • Spend some time reading through the existing threads on the site. If there is still something you don't understand afterwards, your best bet is to ask a new question. – gung - Reinstate Monica Nov 26 '14 at 14:03
0

If your question is solely about comparing the error rates between items A and B regardless of the groups, a 2x2 McNemar test for dependent responses could work (2 nominal variables, i.e. item A and item B with 2 levels each, i.e. correct/wrong).

If however you are interested in examining error rate differences between groups you could create a new variable assessing the decrease/increase in the errors for the both items. You could for example assign a 0 for unchanged responses (correct/wrong for both items) and a 1 to those participants who got it right for item B only. A Chi-Square will compute the actual and % change in the error rates between groups (they 're not dependent frequencies anymore). If it's significant, you could conclude that a significantly higher error rate was observed for the CG in comparison to the EG.

StevenP
  • 504
  • 3
  • 13
  • Many thanks for your answer! I thought about how to do a McNemar's test here and came up with the following solution: The 2x2 table would be CG and EG - Item A and Item B. With this procedure, I'm not sure whether the baseline error can be taken into account, however, because only the errors would be inserted into the table... Or did you think of another way of filling in the 2x2 table? – grey Nov 25 '14 at 07:47
  • That's what I was thinking of. It won't be the errors that you will put have put in the Table, but the error rate (errors-baseline errors) – StevenP Nov 25 '14 at 14:22
  • I see. Would that be a special type of McNemar's test? Because for the chi-square test you won't get the same result with percent compared to the actual number of cases, as this changes the N (unless your N is 100)... Or am I completely wrong? – grey Nov 25 '14 at 16:10
  • I am not sure what you mean but after a bit more thinking I have edited my answer. I hope it makes things a bit clearer now – StevenP Nov 25 '14 at 17:04