I want to compare two confusion matrices, as I discuss here, and I have realized that examining the accuracy of each model is inadequate for such a comparison.
Gung gave a nice answer about how to use Poisson regression to compare two contingency tables. My reaction when I read that yesterday was that the method would apply to confusion matrices, since a confusion matrix is a contingency table.
However, I am now reading some material on comparing contingency tables that has me questioning that belief. Section 5.1.3 of this makes the following comment about a 2x2 contingency table (perhaps the confusion matrix of a binary classifier.
"Treat the four counts as realizations of independent Poisson random variables."
I do not believe that the entries of a confusion matrix are all independent. If I am classifying photos as dogs and cats, then the number of dogs I misclassify as cats depends on the number of dogs I classify as dogs. If I have 100 photos of dogs and classify 90 as dogs, then I must classify the remaining ten dog photos as cats.
I have been arguing with myself about whether or not this counts as dependence because I just as easily could look at the number of misclassifications and say that the number of correct classifications depends on the number of incorrect classifications, though this is looking like a weaker and weaker argument as I type it out.
What would be the way to approach the particular case of comparing two confusion matrices?
