4

Sites such as Survey Monkey allow a layperson to easily create and administer a questionnaire. However, the only thing I know about polling is that you can't just create a questionnaire, let anyone answer it, and get valid results from it.

Is there a layperson-accessible guide to the problems and (hopefully) solutions to creating Internet-based polls from which valid conclusions can be drawn? If not, what would be the best route for a small-business owner unable to afford hiring a national polling company?

(I'm not sure that this is on-topic for Cross Validated, but it seemed like the closest match...)

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Larry OBrien
  • 143
  • 4
  • There's a lot of literature on this. A simple [Google search](https://www.google.com/#hl=en&sclient=psy-ab&q=internet+survey+research&oq=internet+survey+research&gs_l=hp.1.0.0i30l4.1304.8840.0.11897.35.23.5.7.7.0.201.2417.13j8j2.23.0.les%3B..0.0...1c.GhhTVmbS5Rw&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.&fp=6c862fd85693d052&biw=1708&bih=696); an [Amazon search](http://www.amazon.com/s/ref=nb_sb_ss_i_0_15?url=search-alias%3Daps&field-keywords=internet+survey+methods&sprefix=internet+survey%2Caps%2C289) – Peter Flom Aug 28 '12 at 21:21
  • @PeterFlom I was hoping to get some professional/knowledgeable guidance that would help narrow things down. The Dillman book appears to be well-regarded but is it approachable by a non-statistician or someone without any background in polling? – Larry OBrien Aug 28 '12 at 21:34
  • I read Dillman a while back - it's not bad. But you need someone who DOES have a background in polling. There's lots of tricky issues, and it is not as easy as it looks. Not just the polling, but the question writing, the formatting, etc. Sorry, but there's no substitute for expertise. – Peter Flom Aug 28 '12 at 21:39
  • Don't. If you don't know statistics, if you don't know survey methodology (and you don't seem to know much about either), your poll will be crap, and you'll just waste your time. If you think that you will create a cheaper poll than the companies that have been doing this for 20-30-50 years, competing head to head as to who's got a more accurate methodology, that's just silly. I can't see how you can do a reasonable national poll for anything cheaper than $\$ $50K: you need to develop the instrument, buy the sample, factoring in the response rate of less than 10%, field it, analyze it... – StasK Aug 29 '12 at 13:45
  • It is like saying, "I don't have the money to buy a car. Where can I find a self-contained 20-page guide to make my own car from scratch?" – StasK Aug 29 '12 at 13:46
  • @StasK So your answer is that the minimum cost for a valid poll is $50K? That seems like a heck of a market opportunity for an entrepreneurial statistician. – Larry OBrien Aug 29 '12 at 19:40
  • 1
    I've never budgeted a survey, so my figures are ballparks, and I might be off by a factor of 2. A \$50K figure would give you a crappy sample size of 300-400 for a return mail survey. There are hundreds of companies who can deliver a survey like that, but you'd have to go a long way to convince anybody it is "a valid poll". Some people would be OK using that in their research projects, though. Serious face-to-face government surveys usually have budgets upwards of \$2M, and assume a cost of about $1000 per complete, with rigorous and robust designs, non-response follow-ups and such. – StasK Aug 29 '12 at 22:45
  • I'll add one data point: it helps to have some "ground truth" about your target population and to be able to compare it to your respondents' population. Demographics and propensity matters. [The Wikipedia Gender Gap Revisited: Characterizing Survey Response Bias with Propensity Score Estimation](https://doi.org/10.1371/journal.pone.0065782) – Nemo Aug 20 '17 at 08:41

1 Answers1

5

To draw inferences from your sample you would want the sample to be a random sample from a target population. The problem with running a survey on the internet is that it is passive and answered by those who find it and choose to answer. That will not be a random sample from your target population and it may include respondents from outside your target population. You have no control.

A possibility to do this properly would be to get a list of email addresses for all the people you would like to consider to be a part of your target population. Then you randomly sample names from the email list and send the survey to those people. it may help to give them incentives to respond. Then you will be left with only the problem of nonresponse bias. But that is a general problem that is inherent to most surveys. You just do everything you can to minimize the nonresponse. Sometimes nonresponse bias can be assessed by followup surveys for the nonrespondents to hopefully get an idea of how their answers tend to differ from the responders (if they do).

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • I would add a slight nitpick, and suggest that you actually want a *representative* sample. A random sample is just one way to achieve this goal. – probabilityislogic Aug 29 '12 at 01:35
  • Representative doesn't have a precise meaning. By chance a random sample may not look like a "representative sample". To draw inference you actually do need a form of randomness. – Michael R. Chernick Aug 29 '12 at 01:50
  • 1
    It doesn't have a precise meaning in general, but for specific problems there is a precise meaning - otherwise stratified sampling wouldn't be used so much. Polling is definitely one of these cases. For example you would want a good age distribution within your sample. If say there was nobody in a "random sample" aged over 50 then you would be suspicious of any polling result based on this sample. – probabilityislogic Aug 29 '12 at 06:19
  • 1
    It is stratified **RANDOM SAMPLING** that is used. The samplingis random within strata. The same is true for cluster sampling. The principles of inference fall under the probability model. Sampling without randomization, admits no form of inference. For example systematic sampling can in some instances lead to "representative samples", but it does not allow statistical inference. The nebulous definition of representative sample is "a sample that has a distribution of observations that is similar to the population distribution. – Michael R. Chernick Aug 29 '12 at 10:25