1

Suppose that you have $N$ people passing an exam having 2 type of questions (type A and type B) which all have correct/uncorrect outcomes. After seeing the results, I want to modelize the probability of success for one person for every type of questions (A and B) after he has answer a few of this questions already.

I modelize the questions results by two bernoulli laws $X_1=B(p_1)$ and $X_2=B(p_2)$. Each person has his own $p_1, p_2$ but we think that they’re linked. I’m looking for a way to use this dependence …

So far, I have chosen an a priori law on $p_1, p_2$ : the beta$(1,1)$ function for both. We know that if a person answers with $S$ success and $E$ errors for the first type of questions, the a posteriori law will be beta$(1+S,1+E)$. But we don’t know anything about how it changes $p_2$. My guess is that $p_2$ could be something like beta$(1+kS,1+kE)$ with $k$ a coefficient representing how much the first type of question and the second type of question are correlated. This intuition comes from the facts that if the type of questions are independent then the a posteriori law of $p_2$ is still beta$(1,1)$ and if they are perfectly correlated, the a posteriori law of $p_2$ is beta$(1+S,1+E)$.

As someone told me in a previous post, the IRT theory could help but IRT seems to assume that each person has a fixed ability $\theta$ while for my problem, there is 2 abilities and I want to modelize their interdependence.

What about in between ? Any references/books/etc would be greatly appreciated.

Rodolphe LAMPE
  • 255
  • 1
  • 8
  • It is not clear for me, are you talking about dependence between persons, or "questions"? Notice that your $k$ does not change the expected value for beta. It also has nothing to do with dependence between $p_1$ and $p_2$... – Tim May 05 '17 at 10:19
  • The dependence between questions. If I see that type A questions success are moderately correlated to type B questions success, how do I modelize this dependence ? About the expected value, it could change if we consider both questions : let $S_A, E_A, S_B, E_B$ be the success/errors for type A/B questions. Then $p_2$ follow beta$(1+kS_A+S_B, 1+kE_A+E_B)$ – Rodolphe LAMPE May 05 '17 at 10:26
  • 2
    It is hard to answer or comment your question because you are mixing many things together. First you write about univariate Bernoulli distribution with fixed probabilities. Then about beta-binomial model where $p$ is a random variable. Then about dependence between trials in the binomial distribution, what makes it non-binomial, but [poisson-binomial](https://en.wikipedia.org/wiki/Poisson_binomial_distribution) so it is a different model. Then you want to somehow arbitrary make multivariate model from it with 2x2 dependence between variables etc. – Tim May 05 '17 at 10:31

1 Answers1

2

As I see your data, you have multiple questions of two kinds, and multiple people answer them. So there is some kind of overall effect for each person and for each kind of questions. You are considering to make changes in the basic beta-binomial model to account for the different kinds of questions, but this is not correct. Notice that binomial distribution describes outcome of $n$ trials with the same probabilities $p$. If there are different probabilities, then your data follows Poisson-binomial distribution or multivariate Bernoulli distribution (both pretty complicated). If there is two kind of questions and each question of it's kind has the same probability of success, then you could consider modeling them as a mixture of two binomial distributions. Moreover, if the kind of questions has effect on the probabilities of success, then you should use some kind of hierarchical model.

All this said, in my opinion you are overthinking this problem. It can, and should, be solved using Item Response Theory models, but you need to use something more complicated then the out-of-the-box models. As you do not provide detailed information about your data it is hard to tell what should be the exact form of the IRT model, but you probably should consider treating it as a mixed-effects model with different effects for people, questions and abilities, with higher-level effect for types of questions. These kind of IRT models with additional covariates are known as explanatory IRT models and you can google for them to find more references. If you want to learn more about this kind of models you can check the very accessible "IRT models and mixed models: Theory and lmer practice" slides by Paul De Boeck and Sun-Joo Cho and the following papers by the same authors:

De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuerlinckx, F., & Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39(12), 1-28.

Wilson, M., De Boeck, P., & Carstensen, C. H. (2008). Explanatory item response models: A brief introduction. Assessment of competencies in educational contexts, 91-120.

Wilson, M., & De Boeck, P. (2004). Descriptive and explanatory item response models. In Explanatory item response models (pp. 43-74). Springer New York.

and their book

De Boeck, P. and Wilson, M. (2004). Explanatory Item Response Models A Generalized Linear and Nonlinear Approach. Springer.

Tim
  • 108,699
  • 20
  • 212
  • 390