McNemar's test is a special example of the binomial sign test, but the ('vanilla') sign test suffers from bias due to ignoring differences equal to zero.
McNemar's test statistic is given by:
$\chi^{2} = \frac{\left(|r-s|-1\right)^{2}}{r+s}$, where $r$ and $s$ are the counts of discordant pairs (0,1) versus (1,0), distributed $\chi^{2}$ with 1 degree of freedom under the null hypothesis.
I am having a hard time parsing Sribney on the sign test:
The test statistic for the sign test is the number $n_{+}$ of observations greater than zero. Assuming that the probability of an observation being equal to zero is exactly zero, then, under the null hypothesis, $n_{+} \sim \text{Binomial}(n, p=\frac{1}{2})$, where $n$ is the total number of observations. But what do we do if we have some observations that are zero?
Fisher’s Principle of Randomization
We have a ready answer to this question if we view the test from the perspective of Fisher’s Principle of Randomization (Fisher 1935). Fisher’s idea (stated in a modern way) was to look at a family of transformations of the observed data such that the a priori likelihood (under the null hypothesis) of the transformed data is the same as the likelihood of the observed data. The distribution of the test statistic is then produced by calculating its value for each of the transformed “randomization” data sets, considering each data set equally likely.
For the sign test, the “data” are simply the set of signs of the observations. Under the null hypothesis of the sign test, $P(X_{i}>0)= P(X_{i}<0)$, so we can transform the observed signs by flipping any number of them and the set of signs will have the same likelihood. The $2^{n}$ possible sign changes form the family of randomization data sets. If we have no zeros, this procedure again leads to $n_{+} \sim \text{Binomial}(n, p=\frac{1}{2})$.
If we do have zeros, changing their signs leaves them as zeros. So if we observe $n_{0}$ zeros, each of the $2^{n}$ sign-change data sets will also have $n_{0}$ zeros. Hence, the values of $n_{+}$ calculated over the sign-change data sets range from 0 to $n-n_{0}$, and the “randomization” distribution of $n_{+}$ is $n_{+} \sim \text{Binomial}(n-n_{0}, p=\frac{1}{2})$.
Because this seems to be saying go ahead and ignore zeros. But then later in the paper, Sribney provides an adjustment for the sign-rank test that accounts for zeros just along the lines I am wondering about:
The adjustment for zeros is the change in the variance when the ranks for the zeros are signed to make $r_{j}=0$; i.e., the variance is reduced by $\frac{1}{4}\sum_{i=1}^{n-{0}}{i^{2}}=n_{0}\frac{\left(n_{0}+1\right)\left(2n_{0}+1\right)}{24}$.
Should I instead be asking whether or not to apply the signed-rank test to individually-matched case control data?
A simple made up example will illustrate why ignoring zeros presents a problem. Imagine you've paired data with no differences equal to zero (this would correspond to data for a McNemar's test with only discordant pairs present). With a sample size of, say, 20, you find 15 positive signs of differences and 5 negative signs of differences, and conclude significant difference. Now imagine that you have 1000 observed differences equal to zero in addition to those 15 positive and 5 negative signs of differences: now you conclude difference is not significant. If McNemar's test is conducted on 1020 pairs, 1000 of which are zeros, and with discordant pairs of 15 and 5, we should not reject the null hypothesis (e.g. at $\alpha = 0.05$).
There is an adjustment to the sign test to correct for observed zero differences based upon Fisher’s "Principle of Randomization" (Sribney, 1995).
Is there a way of improving on McNemar's test that addresses the effect of observed zero differences (i.e. by accounting for number of concordant pairs relative to number of discordant pairs)? How? What about for the asymptotic z approximation for the sign test?
References
Sribney WM. (1995) Correcting for ties and zeros in sign and rank tests. Stata Technical Bulletin. 26:2–4.
