1

I ran a simple online survey to ask 249 people which of three donation requests they found most compelling. This survey was slightly improved by randomizing the order of the answers.

Message A received 116 votes, message B received 49, and message C received 85. My first guess would be to identify the 95% confidence intervals for the various population proportions (e.g., 46.2% +/- 0.5% is still higher than 34.1% +/- 0.3%, etc) and choose a winner if its lower bound is higher than any competitor's upper bound, but something doesn't feel quite right about this.

One other thing: the messages asked the users to donate a dollar to a charity. We advised in the survey that 10% of the respondents would be offered a $1 bonus. The respondents were then asked to tell us whether or not they'd be willing to donate that dollar to the charity if they were selected. Of the 249 respondents, 174 gave us permission to donate their funds on their behalf. Filtering the results for only users who gave us permission to donate their funds does not appear to materially change how people answered but I wonder if it would be an interesting cross-reference point?

Any thoughts or feedback?

Thanks!

Chris Gagné
  • 11
  • 2
  • 1
    By constructing confidence intervals, you implicitly assume the survey is a *simple random sample* of a population. What population? And what evidence do you have that it is random, or even representative? – whuber Aug 13 '12 at 22:08
  • whuber, thank you for the thoughts. Unfortunately, I can't discuss the exact specifics of the population but it is quite broad. I don't think my sample is perfectly random or representative, but it's the best I can do and I've found it to be reasonably predictive in the past. My alternatives are often a) not running a survey at all and going "blind" and b) running a survey with the sample available to me and trying to estimate whether it could be at all predictive. Think it's worth considering the survey? – Chris Gagné Aug 13 '12 at 22:43
  • Chris, it sounds like this is indeed a *sample* although you're not sure how *random/representative* the sample is. The point I think whuber is getting at is that if your whole population is e.g. a charity club of 249 members, there is no confidence interval, the winner is the winner and that's it. – Jonathan Aug 13 '12 at 22:48
  • Hi Jonathan, thank you! Yes, that makes a lot of sense to me and I'll keep that in mind when surveying a smaller population. For this particular use case, my population is much larger (hundreds of thousands). Much, much appreciated. – Chris Gagné Aug 13 '12 at 22:55
  • 1
    Presumably this survey is intended to guide future fundraising. You should be aware that what people report as the best approach is not necessarily in fact the best approach. For example there have been suggestions that a message of "everyone else is contributing" is more effective than people are prepared to admit when asked. Next time you could test actual approaches on a sample. – Henry Aug 14 '12 at 00:25
  • Chris, for a non-random survey, confidence intervals are at best nonsense and at worst deceptive, because you do not have sufficient information to quantify how close the survey results are likely to be to the population values. That doesn't invalidate the survey, but it indicates you should forget about computing confidence intervals and think about alternative ways to express the possible errors that might be made when using the survey to guide future actions. – whuber Aug 14 '12 at 12:00
  • Of interest: http://stats.stackexchange.com/questions/5462/ – whuber Aug 14 '12 at 12:44
  • whuber, thanks again for the follow up. You've given me great food for thought and I won't worry about confidence intervals in this case. – Chris Gagné Aug 14 '12 at 19:45

0 Answers0