Analyzing my company’s internal survey results, I was stuck with calculating statistical difference between percentages. I usually used a Z-test as described below, but I am not sure whether it is suitable for very small samples (5 to 10 respondents), and if it is not, then how to calculate the difference?
Now let me describe my common approach. Suppose there is a question with multiple answer options (let them be A, B, etc.), and each respondent can choose only one option. Then, in my case, we have n = 259 participants with A = 117 of them choosing A answer, and B = 88 choosing B answer. (The other 54 chose other options or were undecided.)
First, I calculate the percentages: $a = {A \over n} \approx 0.45$, $b = {B \over n} \approx 0.34$, and the difference $\Delta = {{\left| {A - B} \right|} \over n} \approx 0.11$.
Second, I calculate the standard error of difference, as suggested by Wikipedia (based on the book Survey Sampling by Leslie Kish, p. 500):
$$\bar \sigma = \sqrt {{{a + b - {{{\rm{(}}a - b{\rm{)}}}^2}} \over n}} = 0.05484\ldots$$
And finally, I calculate $Z = {\Delta \over {\bar \sigma }} \approx 2.04$ and compare it to the critical Z-score = 1.96 (for 95% level of confidence, two-tailed test). Since my Z is greater than the critical one, I reject the hypothesis that there is no difference between percentages.
But now a question arises: does this approach work for small samples (say, 7 or 9 respondents)? If it doesn’t, then how to calculate statistical difference in this case? If you will find it interesting to answer, then I would ask you please to give a calculations’ example, for I am not a skilled statistician, I perform all the math in MS Excel, and I will hardly understand the procedure without a live example.
