4

I have paired data from 9 individuals. The response variable is ordinal with levels 1-5. As the sample is very small and the data are categorical, I thought the appropriate test was McNemar's test. However I get McNemar's chi-squared = NaN and therefore a p-value = NA. What should I do to test whether the effect is significant?

My data:

before

4 3 4 5 4 4 4 5 3

after

5 4 3 5 5 4 5 5 4
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Mik meadow
  • 203
  • 2
  • 11

3 Answers3

3

I wonder if you really want McNemar's test (more specifically the McNemar-Bowker test). These are tests to see if the marginal proportions are the same (see here). Under the assumption that pairings can be recovered from the orders of the two vectors, here is a table of your data with the marginal proportions computed: 

bef = c(4, 3, 4, 5, 4, 4, 4, 5, 3)
aft = c(5, 4, 3, 5, 5, 4, 5, 5, 4)
table(bef, aft)
#    aft
# bef 3 4 5
#   3 0 2 0
#   4 1 1 3
#   5 0 0 2
table(bef)/sum(table(bef))
# bef
#         3         4         5 
# 0.2222222 0.5555556 0.2222222 
table(aft)/sum(table(aft))
# aft
#         3         4         5 
# 0.1111111 0.3333333 0.5555556

Since you state that your variables are ordered, it seems more appropriate to assess if one set has higher values than the other. That would use the Wilcoxon signed rank test: 

wilcox.test(bef, aft, paired=TRUE)
#   Wilcoxon signed rank test with continuity correction
# 
# data:  bef and aft
# V = 3.5, p-value = 0.1294
# alternative hypothesis: true location shift is not equal to 0

Because you have ties, you cannot use the exact version of the test; you use the asymptotic approximation. If that bothers you due to the small sample size, you could simulate the null and compute the p-value that way. There are different ways to do that, but I'm not sure how much difference they will make in practice. Bootstrapping is generally not recommended with small samples. You could try a permutation-based version, or you could do a Monte-Carlo simulation of the null based on the marginal distribution of the values (ignoring timepoint).

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • Thank you very much, I thought the Wilcoxon signed rank test would not be valid for qualitative data, because it doesn't take into account that the response variable can only take values from 1 to 5. – Mik meadow Jul 26 '16 at 20:11
  • 1
    The only assumption the Wilcoxon signed rank test makes about your response data is that you can say if 1 value is > another. You can do that here, so there isn't really a problem. You cannot use the exact version of the test since there are ties; you use the asymptotic approximation. If that bothers you due to the small sample size, you could simulate the null and compute the p-value that way. – gung - Reinstate Monica Jul 26 '16 at 20:19
2

It is almost certainly because you have a pair of cells which both have zeroes (3, 5) and (5, 3). If you look at the way the test is defined that will lead to problems.

What to do is a problem. You could always merge categories but that may not make scientific sense. I think you may have to program your own solution. You could define a log-linear model of quasi-symmetry without including the offending cells (treating them as structural zeroes) but that may not be a sensible assumption either. More data would help but I suppose that may not be an option.

mdewey
  • 16,541
  • 22
  • 30
  • 57
0

You can use the exact binomial test. Let a be the number of cases where before < after. Let b be the number of cases where before > after.

H0: Getting a smaller value after is as likely as getting a bigger value after.

So you can use

bef = c(4, 3, 4, 5, 4, 4, 4, 5, 3)
aft = c(5, 4, 3, 5, 5, 4, 5, 5, 4)
a <- length(bef < aft)
b <- length(bef > aft)
binom.test(a, a + b, p=0.5)
Atzs
  • 1
  • Just to be clear, this is ***not*** an implementation of McNemar's test in this case. It's something more akin to a sign test. It's important for the analyst to be clear if they wish to treat their categories as nominal in nature or ordinal in nature, as the analysis will be different. ... (Although a binominal case can be used as an analogous test for McNemar's test in the case of a 2 x 2 table.) – – Sal Mangiafico Oct 25 '21 at 13:04