9

French polling institutes are currently facing a major crisis after they recently published what can only be called the most ridiculous poll so far on the 2012 presidential election horse race. The French Senate is now considering to legislate on the issue by forcing polling institutes to publish, among other things, the confidence intervals for their results.

However, some pollsters are opposing the measure, claiming that confidence intervals do not apply to quota sampling, which is the method used by polling institutes in France. Since quota sampling is formally non-probabilistic sampling, there is some truth to the claim. But since quota sampling is fundamentally stratified sampling, confidence intervals should apply, right?

May I ask for experiences about this issue outside of France, in countries where pollsters also use quota sampling?

Fr.
  • 1,343
  • 3
  • 11
  • 22
  • 5
    Quota sampling is distinct from stratified sampling in precisely this respect: unlike selection in a stratified sample, the lack of randomness in quota sampling means we cannot assure lack of bias. So: maybe you can't erect CI's for estimates from quota sampling, but you *can* establish intervals for the bias. Unfortunately, except when the sample is a substantial proportion of the population, the bias extends from almost -100% to almost +100%. This is why quota samples have long been discredited ("Dewey defeats Truman", etc.). *Caveat emptor.* – whuber Mar 16 '11 at 17:45
  • Pollsters are claiming that quota sampling, which is the industry standard in France, is also widespread in Britain and Southern European states. I wonder if Britain really uses quota sampling, and why use quota sampling in the first place if the bias is irremediable? – Fr. Mar 16 '11 at 22:30
  • 1
    @Fr Quota sampling is easier to carry out than random sampling: the pollster can specify what kinds of subjects are needed and how many to interview, send interviewers onto the street with these instructions, and easily check whether the job was correctly done: no need to do the difficult work of establishing a sampling frame, drawing a probability sample from it, hunting down its members, chasing after nonrespondents, *etc*. Quota sampling is a textbook poster child for outmoded, known-bad sampling methods: see http://www.math.uah.edu/stat/data/1948Election.pdf for instance. – whuber Mar 16 '11 at 22:40
  • @whuber: Thanks for the details. I knew about the "Dewey defeats Truman" story. I still doubt that British polling institutes are indeed using quota sampling, as claimed by the previously cited document, and would have to check. It would be interesting to know what are the different industry standards across the largest countries. I know for a fact that most of the data I download from [ICPSR](http://www.icpsr.umich.edu/icpsrweb/ICPSR/) is Random Digit Dialling. – Fr. Mar 17 '11 at 07:49
  • 1
    A lot of non-experts (i.e. people who don't understand the statistical theory behind sampling) throw around the term "quota sampling" without really knowing what it means. That may explain the claim that it's common practice in Britain and Southern Europe. – Abe Apr 18 '11 at 01:45
  • French opinion polling shortly before elections has more often than not been as accurate or more accurate than elsewhere in the world, though they do better when there are only two choices. [Other polls conducted in March 2011](http://fr.wikipedia.org/wiki/%C3%89lection_pr%C3%A9sidentielle_fran%C3%A7aise_de_2012_%28sondages%29#Sondages_r.C3.A9cents) seem to be fairly consistent with the one which provoked this question. – Henry Apr 18 '11 at 02:28

2 Answers2

6

As whuber says, the short answer is that quota samples are the "poster child for outmoded, known-bad sampling methods" and "have long been discredited." The longer answer is that there may be conditions under which "quota-like" samples can work reasonably well.

Exhibit A here is recent work on reconstructing representative results from opt-in Internet panels. This paper gives the statistical grounding for this approach. To make a long story short, typical sampling schemes 1) draw a random sample, 2) attempt to recruit subjects, and then 3) add post-stratification weights to compensate for differences in who responds. In the opt-in approach, you 1) recruit subjects non-randomly, 2) compare responses to a representative baseline, and 3) add weights to compensate for the differences.

In terms of practice, opt-in sampling is similar to quota sampling, but the statistical foundation is more developed. The upside is that you can make claims about representative sampling, confidence intervals, etc. The downside is that your claims are based on difficult-to-verify assumptions about how people self-select into your sample.

A lot of people are skeptical about these methods -- they sound too much like quota sampling. But some evidence suggests that opt-in sampling can work well at least some of the time. So despite the controversy, Polimetrix/YouGov (an early adopter of the opt-in sampling model) seems to be doing reasonably well. Among other things, they've done all the data collection for the Cooperative Congressional Election Study, a series of recent academic U.S. national election studies.

(I'm pretty sure ICPSR carries this data. If not, Harvard's social science dataverse certainly does. Lots of academics are using data from these samples.)

Anyway, you asked about quota sampling. As you can see already in the comment thread here, any well-trained pollster will tell you that quota sampling is bunk. The jury is still out on opt-in sampling. For the time being, if you want to draw confidence intervals around quota samples, I'd say these methods are your best bet.

Pierre
  • 412
  • 5
  • 16
Abe
  • 1,250
  • 3
  • 13
  • 22
  • Thanks for the detailed answer. I flagged the following points in it: (1) Quota sampling is used for Internet sampling in your example. Out of that context, there is no reason to use it over random sampling through RDD or equivalent techniques. (2) Even in the context you describe, it remains very controversial. So, to go back to my question, French pollsters justifying quota sampling for all their work are just being silly, or ignorant of sampling theory, or both. – Fr. Apr 18 '11 at 15:51
  • "French pollsters justifying quota sampling for all their work are just being silly, or ignorant of sampling theory, or both." Sounds about right to me. – Abe Apr 18 '11 at 19:40
  • Well, I am checking "Answered" on that one. Thanks for your time and detailed references. – Fr. Apr 19 '11 at 02:51
2

  1. In most non-compulsory survey contexts, there is a substantial problem with nonresponse. This from 2002: "the recently reported estimate of survey cooperation rates from CMOR, the Council for Market and Opinion Research [USA], averaged only 14.7 percent." and from Paul Gerhold, "I believe that it is still possible to draw random samples. I just don't believe that it is possible to execute them." In this context, the fact that the SAMPLE is random isn't very relevant, because the resulting data isn't.

  2. This makes bias adjustment the major issue in valid estimation, and field method design is an important component. The ways in which one might want to do this, and the resulting confidence estimates, are well beyond what can be discussed here.

zbicyclist
  • 3,363
  • 1
  • 29
  • 34