3

I guess this is a very basic question, but I find myself unable to find an answer.

I want to ask a sample of probands (e.g. N = 467) which one of three presidential candidates they would elect, A, B, or C. I will then have a vector of votes containing a certain number of "A"s, "B"s and "C"s each, that is, something like this: A, B, A, C, C, B, A, A, B, ....

When I count the votes, a possible result could be:

A: 345
B: 52
C: 70

How do I test whether these vote counts are significantly different from each other? And if possible, how do I calculate effect size?

I would be especially happy if you could provide an R function for your answer, if it exists.

2 Answers2

5

Hypothesis testing

We can rephrase your question from a hypothesis testing point of view: you want to test the null hypothesis that your data come from a uniform probability distribution across the three candidates, i.e. each candidate has a 1/3 probability to be chosen. For this, we can use a $\chi^2$-test of significance.

Calculation

For a $\chi^2$ test, we need both observed and expected values. In the example you gave, the expected counts $E$ for each cell under a uniform distribution are $\frac{467}{3} \approx 155.6667$. The observed counts $O$ are simply the counts that you indicated. The $\chi^2$ statistic for this particular case then equals $$\chi^2 = \sum_{i=1}^3{\frac{(O_i-E_i)^2}{E_i}}$$ where $i$ indicates the cell.

For the example, this amounts to $\chi^2 = 346.4625$. This value follows a $\chi^2$-distribution with $n-1 = 3-1 = 2$ degrees of freedom, which enables us to perform a significance test: we can test how far the value $346.4625$ is in the tail of the $\chi^2(2)$ distribution.

As it turns out, pretty far:

chisq graph

How to do this in R

R is pretty great. Here's how to do it in R:

chisq.test(c(345,52,70))

result:

    Chi-squared test for given probabilities

data:  c(345, 52, 70)
X-squared = 346.46, df = 2, p-value < 2.2e-16

Effect Size

There are many effect size measures of a $\chi^2$ test. See for instance this question. However, before deciding on an effect size measure, you should think about what you want to achieve. For your purpose, reporting the raw numbers could be enough to give your audience a good idea about the size and direction of the effect!

Community
  • 1
Erik-Jan
  • 101
  • 1
  • 3
  • As I understand it, this does not tell me if all of the three vote counts are significantly different from each other, e.g. if the difference between 52 und 70 is significant, too, or of only 346 is different. –  Jul 27 '16 at 09:06
  • for this, you could look at [this question](http://stats.stackexchange.com/questions/111986/post-hoc-chi2-test-with-r). It gives an example of a post-hoc test for $\chi^2$ tests. – Erik-Jan Jul 27 '16 at 09:12
  • As your answer stands, it does not answer my question. The relevant aspects are only given as links, one of them in a comment, and how one would go about applying the answer linked in the comment to my question remains unexplained. I have upvoted your answer for your detailed explanation of the chi-square test, but cannot accept it. –  Jul 28 '16 at 09:44
4

You are interested whether two or more pairs of candidates' votes differ in their distribution. The alternative (null hypothesis) to that is that no two pairs are different, that is, all are the same. As user etothej has nicely illustrated before, a simple omnibus test of differences in responses is the $\chi^2$-test. In R-code:

x <- c(345,52,70)
chisq.test(x)

which gives us a significant result

    Chi-squared test for given probabilities

data:  x
X-squared = 346.46, df = 2, p-value < 2.2e-16

Again, the null hypothesis states that the distribution of the votes are the same for each president. With a significant result, we are willing to reject this null hypothesis. Now, we still do not know which of the pairs do really differ. By inspection, it looks like the first (345 votes) president received a lot more votes than the others. But specifically testing this difference is not a proper confirmatory test anymore because we now have peeked at the data. To get an unbiased valid inferential result, we must really look at all pairwise comparisons and correct for multiple testing (unless you had suspected and hypothesized that president A is getting most votes before collecting the data). The so-called Marascuilo procedure enables us to simultaneously test the differences of all pairs of proportions while correcting for multiple comparisons. I have not found an R implementation, so I provide my own here:

p <- x/sum(x)
N <- length(x)
crit <- qchisq(0.95,length(x)-1)

# prepare storage of results
result <- data.frame(matrix(NA, nrow=(N^2-N)/2, ncol=5,
                 dimnames = list(NULL,c("i","j","statistic","crit.value","significant"))))

# iterate over all pairs with (i,j)
idx <- 1
for (i in 1:(length(p)-1)) {
  for (j in (i+1):length(p)) {
    stat <- abs(p[i]-p[j])
    rcrit <- sqrt(crit) * sqrt( p[i]*(1-p[i])/x[i] + p[j]*(1-p[j])/x[j]  )
    result[idx, ] <- c(i,j, round(stat,2),round(rcrit,2), ifelse(stat>rcrit,"yes","no"))
    idx<-idx+1
  }
}

To briefly explain this code: With the two for-loops, we iterate through all combinations of pairs and calculate a test statistic stat as the absolute difference of proportions and a critical value, rcrit for each of the pairs. Each statistic exceeding the critical value is denoted significant. The results are stored in a data.frame.

For your example, we get

print(result)

  i j statistic crit.value significant
  1 2      0.63       0.12         yes
  1 3      0.59       0.12         yes
  2 3      0.04       0.15          no

From this, we can obtain that the pairwise differences of (1,2) and (1,3) are significant but we found no significant difference between (2,3).

The Marascuilo procedure is explained in Marascuilo (1966) and – more comprehensively – Marascuilo & McSweeney (1977, p. 141-147).

  • Marascuilo, L. A. (1966). Large-sample multiple comparison. Psychological Bulletin, 65(5), 280-290. doi:10.1037/h0023189
  • Marascuilo, L. A., & McSweeney, M. (1977). Nonparametric and distribution-free methods for the social sciences. Monterey, CA: Brooks/Cole.
Brandmaier
  • 753
  • 4
  • 13