How to provide statistical evidence that an experimental method is providing higher mean percentages?

Question

I performed an experiment using two different methods. The data are percentages and higher percentages indicate a better method. The results were as follows:

Method 1 :74.47,79.34,72.85,83.54,75.18,73.58,70.06,
          73.35,74.41,78.35,73.14,81.55,76.21,74.67

Method 2: 83.25,85.14,81.61,77.59,79.38,80.24,79.94,82.54,78.22,
          83.21,82.54,77.67,83.51,78.87

The mean of method 1 is 76.75 and that from method 2 is 80.97

I claimed that method 2 is better because of the value of the mean. I have been asked by reviewers to provide strongs statical evidence that method 2 performs better than method 1.

Specifically:

Comment 1: Results would become reliable within a statistical framework offering p-values or other statistical measures of difference.

Comment 2: A statistical analysis should be carried on in order to provide strong evidence with regard to the best performance of Method 2 over Method 1.

Answer 1

[A good place to find more on this stuff would be in many introductory college-level texts.]

My biggest concern would be that with percentages, the variance is unlikely to be constant. This would be an issue because the t-test (for example) would be based on assuming that the differences are observations from a population with a common mean and common variance, when in fact the differences for the larger observations are likely to become "squeezed up"; the closer they get to 100%, if they can't go over it. That is, the variance of the differences will depend on the sizes of the observations in the difference. The distribution of the test statistic will be impacted - to be specific, the nominal significance level will not reflect the actual rejection rate under the null hypothesis. The power, too, will be affected.

--

For the t-test:

Let x = score on method 1, y = score on method 2

To test the null hypothesis that the true mean difference is zero:

1. Calculate the difference ($d_i = y_i − x_i$) between the two observations on each pair, keeping the sign.

2. Calculate the mean of the differences, $\bar{d}$.

3. Calculate the standard deviation of the differences, $s_d$, and use this to calculate the standard error of the mean difference, $s_{\bar{d}} = \frac{s_d}{\sqrt{n}}$

4. Calculate the t-statistic, which is given by $T = \frac{\bar{d}}{s_{\bar{d}}}$ . Under the null hypothesis, this statistic follows a $t$-distribution with $n−1$ degrees of freedom, where $n$ is the number of pairs.

5. Use tables of the $t$-distribution to compare your value for $T$ to the $t_{n−1}$ distribution. This will give the $p$-value for the paired $t$-test.

Uh, is this homework? What was the experiment about?

Edit: example calculations:

In R:

Method1 <-c(74.47,79.34,72.85,83.54,75.18,73.58,70.06,
           73.35,74.41,78.35,73.14,81.55,76.21,74.67)

Method2 <- c(83.25,85.14,81.61,77.59,79.38,80.24,79.94,82.54,78.22,
           83.21,82.54,77.67,83.51,78.87)


> t.test(Method1,Method2,paired=TRUE)

    Paired t-test

data:  Method1 and Method2 
t = -4.071, df = 13, p-value = 0.001323
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -7.982491 -2.447509 
sample estimates:
mean of the differences 
                 -5.215 

OR:

> wilcox.test(Method1,Method2,paired=TRUE)

    Wilcoxon signed rank test

data:  Method1 and Method2 
V = 9, p-value = 0.004028
alternative hypothesis: true location shift is not equal to 0 

OR (sign test):

> binom.test(sum(Method1>Method2),length(Method1))

    Exact binomial test

data:  sum(Method1 > Method2) and length(Method1) 
number of successes = 2, number of trials = 14, p-value = 0.01294
alternative hypothesis: true probability of success is not equal to 0.5 
95 percent confidence interval:
 0.01779452 0.42812916 
sample estimates:
probability of success 
             0.1428571 

OR (permutation test):

diffs <- Method1 - Method2
f3 <- 2*outer(0:(2^14-1),2^(0:13),function(x,y) (x %/% y) %% 2) -1
res <- rowMeans(f3*matrix(rep(diffs,each=2^14),nc=14))
data.frame(pvalue = sum(abs(res)>=abs(mean(diffs)))/2^14,row.names=" ")
       pvalue
  0.003417969

etc . . .