2

I'm using a Wilcoxon signed-rank test to compare two related samples of non-normally-distributed data. The data contains some ties, so when I run the test in R, I get this warning:

wilcox.test(data$valueA, data$valueB, paired=T)

    Wilcoxon signed rank test with continuity correction

data:  isnapRequests$fullMean and isnapRequests$template
V = 181, p-value = 0.07691
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(wilcox.test(data$valueA, data$valueB, paired=T,  :
  cannot compute exact p-value with zeroes

I understand why the warning is given, since the test assumes a continuous distribution and my data is discrete (analogous to survey data). I also assume that the p-value reported by R is conservative, in the sense that the ties make it more difficult to reject the null hypothesis.

I read on this forum that one way around this problem is to jitter the data and run multiple trials to estimate what the p-value would be without these ties:

ps <- sapply(1:1000, function(i) {
    wilcox.test($valueA, jitter(data$valueB, amount=0.001), paired=T)$p.value
})
mean(ps < 0.05)
# 0.877, so 88% of trials were significant at 0.05
mean(ps)
# 0.03291361 is the average p-value over 1000 trials

This seems to be sufficient evidence that the null hypothesis should be rejected, but I'm not very familiar with this process, so I have three questions:

  1. Is this procedure acceptable and is my interpretation reasonable? I'm not trying to "p-hack", but I also want to report the difference if it's real.
  2. This process seems difficult to report in a scientific paper. Is there a commonly accepted way of describing this procedure that won't raise eyebrows?
  3. How would you report the test? As an averaged p-value? Would you also average test statistics? And then you have the SD of the p-value as well... This seems messy.

If there's a better way to approach this, I'm open to suggestions as well. Thanks!

thomas88wp
  • 175
  • 1
  • 7
  • If I may, I would suggest to consider two alternatives to the Wilcoxon signed-rank test: (1) Bayesian signed-rank test or (2) Bayesian hierarchical model. The two are pretty well introduced and motivated in "Time for a Change: A Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis" by Benavoli et al (2017). – Vladislavs Dovgalecs Mar 09 '18 at 20:15
  • If your goal is to see if the average difference within the pairs is not equal to zero, it might be easier to perform inference on the mean difference. In my mind, the simplest thing to do (and report) would be to take the vector of differences, and perform some randomization inference using it. `sampling.dist = replicate(1000,{mean(sample(data$valueA-data$valueB,size=nrow(data),replace=T))})` would give the distribution. A reasonable p-value would be twice (two-sided) the portion on the smaller side of 0. `portion = mean(sampling.dist < 0); pval = 2*min(portion,1-portion)`. – user5957401 Mar 09 '18 at 21:47
  • Sribney, W. M. (1995). [Correcting for ties and zeros in sign and rank tests](https://www.stata.com/products/stb/journals/stb26.pdf). *Stata Technical Bulletin*, 26:2–4. – Alexis Aug 21 '18 at 14:08

0 Answers0