Voting system that uses accuracy of each voter and the associated uncertainty

Question

Let's say, we have simple "yes/no" question that we want to know answer to. And there are N people "voting" for correct answer. Every voter has a history - list of 1's and 0's, showing whether they were right or wrong about this kind of questions in the past. If we assume history as a binomial distribution, we can find voters' mean performance on such questions, their variation, CI and any other kind of confidence metrics.

Basically, my question is: how to incorporate confidence information into voting system?

For example, if we consider only mean performance of each voter, then we can construct simple weighted voting system:

$$result = sign(\sum_{v \in voters}\mu_v \times (-1)^{1-vote})$$

That is, we can just sum voters' weights multiplied either by $+1$ (for "yes") or by $-1$ (for "no"). It makes sense: if voter 1 has average of correct answers equal to $.9$, and voter 2 has only $.8$, than, probably, 1st person's vote should be considered as more important. On other hand, if 1st person have answered only 10 questions of this kind, and 2nd person have answered 1000 such questions, we are much more confident about 2nd person's skill level than about those of the 1st - it's just possible that 1st person was lucky, and after 10 relatively successful answers he will continue with much worse results.

So, more precise question may sound like this: is there statistical metric that incorporates both - strength and confidence about some parameter?

Answer 1

You should consider the expertise of a voter as a latent variable of your system. You may then be able to solve your problem with bayesian inference. A representation as graphical model could be like this :

Let's denote the variables $A$ for the true answer, $V_i$ for the vote of the voter $i$ and $H_i$ for its history. Say that you also have an "expertise" parameter $\mu_i$ such that $\Pr(A=V_i) = \mu_i$. If you put some prior on these $\mu_i$ -for example a Beta prior- you should be able to use the Bayes theorem to infer $\Pr(\mu_i \mid H_i)$, and then integrate over $\mu_i$ to compute $$\Pr(A \mid V_i, H_i) = \int_{\mu_i} \Pr(A, \mu_i \mid A_i, H_i)~ \mathrm{d}\mu_i$$

These systems are difficult to solve. You can use the EM algorithm as an approximation, or use complete likelihood maximisation scheme to perform exact Bayesian inference.

Take a look on this paper Variational Inference for Crowdsourcing, Liu, Peng and Ihler 2012 (presented yesterday at NIPS !) for detailed algorithms for solving this task.

Answer 2

I know this is really old now but I just stumbled across this question while searching and I think another way to think about how to solve it is using the framework of online learning with expert advice. In this setting, a learner receives predictions (votes) from a set of "experts" (voters), and must choose what to predict itself based on this advice. After the learner makes a prediction, the true outcome is revealed, and the learner adjusts its weightings for how much to attend to each expert's advice accordingly, to minimise long-term regret (the loss incurred by a wrong decision minus the loss incurred by the best expert).

Suitable references are "Tracking the Best Expert" (Warmuth & Herbster, 1998): https://users.soe.ucsc.edu/~manfred/pubs/J39.pdf and "Tracking a small set of experts by mixing past posteriors" (Bousquet & Warmuth, 2002): https://jmlr.csail.mit.edu/papers/volume3/bousquet02b/bousquet02bbw.pdf . These algorithms come with proven "regret bounds", but their actual performance will vary depending on how the population of experts changes over time.