0

I had an individual take a test with 500 binary questions before and after an intervention. Let's say they scored an 80% initially and a 90% on the second pass through.

I can measure the absolute change in accuracy (in the example 10%), but how do I construct a confidence interval for this change in accuracy?

Would the right step be bootstrap in pairs (i.e. bootstrap on the question number and include both the before and after) to get some large number of samples for the accuracy metric? If so, how many samples would be appropriate?

A Ti
  • 35
  • 4
  • You write as if bootstrapping were some magical way to make small amounts of data look bigger. That's not the case. If by "accuracy" you mean something like the proportion of "yes" answers, then your analysis will depend on your suppositions about the independence (or lack thereof) of an individual's answers. Bootstrapping won't help you with that. So: is your question about how to analyze your data or is it about bootstrapping? – whuber Jan 20 '22 at 05:12
  • I've made some edits that hopefully clarify the particular ask, which is how do I construct a confidence interval for the change in accuracy? My hunch is that bootstrapping will be involved, but I'm not sure. – A Ti Jan 22 '22 at 21:53
  • Although you could construct a confidence interval by making specific probabilistic assumptions, its utility would be doubtful, because you have essentially no basis to suppose this result generalizes to any other individual. There's no need to bootstrap such a simple parameter: the underlying theory is well established and analytical calculations are available. https://stats.stackexchange.com/a/360832/919 has a good discussion and authoritative references. – whuber Jan 22 '22 at 22:07
  • The intention of this confidence interval is not to generalize beyond this particular individual. The link appears to be for hypothesis testing for unpaired measurements, which I'm not sure how to relate back to constructing a confidence interval for this particular question. – A Ti Jan 22 '22 at 23:56
  • Are the questions the same before and after or not? – whuber Jan 23 '22 at 16:51
  • They are the same question before and after. – A Ti Jan 23 '22 at 22:57
  • You would get more information, and potentially learn more, by tracking all nine answer patterns (correct, incorrect, not answered on each test). – whuber Jan 24 '22 at 14:27
  • Not sure I know what you mean by "nine answer patterns" as far as I can tell there are only the three you mentioned. Apart form that, I'm not sure how that would help me construct the confidence intervals for an individual's accuracy – A Ti Jan 24 '22 at 22:34
  • An answer pattern is, as you have said, an (ordered) *pair* of answers: thus there are $3\times 3 = 9$ possible pairs. Changes from right to wrong and wrong to right tell you different things about a person's performance. – whuber Jan 24 '22 at 22:36

0 Answers0