Analysis of binomial variables with random effects

Question

I have a study where each of the 43 items is a question to which the response may be A or B and I want to infer whether A has been chosen significantly more than B. There are 20 participants and 43 items, I have thus 20*43= 860 rows in my datafile. I know I can run a binomial test/chi square, but my problem is that these tests don't take into account the unmeasured correlation between responses due to participants and questions.

Answer 1

You can use a mixed effects model, a logistic regression with random intercepts for question and for participant. If you have data in long format, something like

participant  question answer
1            1        A
1            2        B
 .
 .
 .
n            1        A
n            2        A

the logistic regression will have a linear predictor $$ \eta_{ij} = \mu + \alpha_i + \beta_j $$ where $\alpha_i$ is a random intercept for participant $i$ and $\beta_j$ a random intercept for question $j$. Then the model is completed by stipulating that $$ \alpha_i \stackrel{\text{iid}}{\sim} \mathcal{N}(0,\sigma^2_\alpha) \\ \beta_j \stackrel{\text{iid}}{\sim} \mathcal{N}(0,\sigma^2_\beta) $$

If you have some other covariates you can build from here.

In R something like

df <- <your data frame>
library(lme4)
mod0 <- glmer( answer ~ (1 | participant) + (1 | question), 
               data=df, family=binomial )

This is similar to the setting in item response theory, so look through item-response-theory. See also the tag elo, since such rating systems are based on similar models. An interesting applications is the chess puzzles on lichess where both solver and quizz will get its own random parameter (but they will be updated according to how the puzzle is solved).