5

I have asked a "yes/no" question to a small group of people (n = 22). From the 22, 14 answered "yes" and 8 answered "no". I'd like to find out, with 90% confidence, if the majority of my population (> 50%) would answer "yes" to that question.

I would use a z-test for calculating the population proportion, although my sample size is too small to use this kind of test, according to what I have read. I've read t-tests should be used when the sample is small (i.e. n < 30), although I haven't been able to find examples of a t-test for a population proportion.

Should I perform a z-test even though my sample size is small? Should I do some other kind of calculation?

I apologise if the used terminology is not correct, or I'm asking something silly. My background is not in statistics and I haven't done any statistics in almost a decade.

Thanks!

Charmander
  • 153
  • 5

1 Answers1

7

To achieve that level of confidence, you would have to satisfy the most stringent critic. They would require you to establish

  1. Your sample truly is a simple random sample.

  2. The respondents answer honestly and correctly.

  3. That if the barest minority, just less than half, of the population actually would answer "yes", you would have less than a $100 - 90\% = 10\%$ chance of observing at least this many yeses in such a random sample.

You address $(1)$ by explaining and documenting your procedures to identify the population and obtain a sample from it.

You address $(2)$ by documenting how the questioning was carried out and including additional questions to assess reliability and internal consistency.

You address $(3)$ by computing the chance of observing $14$ or more yeses in a random sample when at most $50\%$ of the population would answer yes if asked. Let's do that.

Assuming the population is large (any larger than a few hundred would be fine), the distribution of the yes counts would be very close to Binomial with parameters $22$ and $p \lt 50\%$. In the worst case you have to deal with, take $p=50\%$.

Here is a partial chart of the relevant chances. The top row is a threshold count; below it is the chance that the count in a sample would equal or exceed it.

Threshold:   11   12   13   14   15   16   17   18   19   20   21   22
   Chance: 0.58 0.42 0.26 0.14 0.07 0.03 0.01 0.00 0.00 0.00 0.00 0.00

Since the chance of 14 or more is $0.14=14\%$ and that's greater than $10\%$, you cannot have $90\%$ confidence that a majority of the population would answer yes.

If the population is much smaller than several hundred, the confidence in these results noticeably increases. In the very best case, where the population is $28$ or smaller, your sample already contains half or more of the yes-responders and your confidence is $100\%$.

The R calculation of the table was 

x <- 11:22
y <- round(pbinom(x-1, 22, 1/2, lower.tail=FALSE), 2)
names(y) <- x
print(y)

Alternatively, you could use the Normal approximation to this Binomial distribution and compute instead

y <- round(pnorm(x-1+1/2, 11, sqrt(22*1/2*(1-1/2)), lower.tail=FALSE), 2)

To two decimal places the results are the same. With even less computation you could find the critical threshold approximately as ceiling(qnorm(0.90, 11, sqrt(22*1/2*(1-1/2))) + 1/2), which returns $15$.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
  • +1 But, please explain why this is nearly binomial "Assuming the population is large (any larger than a few hundred would be fine)." I must be missing something, it seems to be exactly binomial, like coin tossing, no matter what the populations size is. – Carl Aug 08 '17 at 20:45
  • Fantastic answer. Thank you so much for not just answering my question but going the extra mile. I've learned something new today. – Charmander Aug 08 '17 at 21:05
  • @Carl Imagine a population of exactly 22 people and a sample of 22. No matter how many times you repeat the sampling process (which, with a survey, is *without* replacement), you won't see a Binomial distribution emerge. If you think this is too trivial an example, consider a population of 23 people. Now there will be variation among samples of size 22, but (unless there's exactly one yes respondent in the entire population) that variation still will not be Binomial. – whuber Aug 08 '17 at 21:33
  • 1
    @whuber Is this correct? Are you saying that unlike the same coin being tossed to look for bias of that coin that the probability of any particular answer from each individual is different, such that is each person is, in effect, similar to a differently biased coin? – Carl Aug 08 '17 at 22:54