6

Questionaires are often used in social sciences. Many people try to complete them very quickly and very often they only "guess" answers. Is there any statistical technique or any research in this area, how to identify which questionnaires are completed poorly?

I think that this is similar to detection of outliers, but which technique is the most relevant? Some persons have very strange behaviour, so they would be identified as outliers although their questionnaires are not fraudulent.

So how to distinguish between fraudulent questionnaire and very strange person?

I have already tried to identify fraudulent questionnaires but they claimed that they were only unique persons :-).

sitems
  • 3,649
  • 1
  • 25
  • 52
  • 1
    An adequate answer to this question can be found by reading not a few paragraphs but by reading many books and/or taking several courses on survey design, survey bias, survey validity, and/or psychometrics. – rolando2 Oct 09 '12 at 16:51

1 Answers1

9

This is a fairly large topic in social psychology and questionnaire design. Here are some ideas:

  1. The person could be faking it, either good or bad. People do this in order to appear "good" to the person doing the study. There are scales to detect this sort of faking, such as the Crowne=Marlowe scale. These essentially ask questions to which virtually no one could answer "yes" (e.g. "I have never told a lie in my life").

  2. Often, people designing questionnaires will ask the same question in different ways. One well-known issue is that people will give different age answers if you ask "How old are you?" and "What is your birth date?" The latter have been found to be more accurate.

  3. Another type of pattern is to answer all the questions with one answer on multiple choice questionnaires. One way to detect this is to have some questions that are reverse coded. Then someone who answers (say) "nearly all the time" to both "I am happy" and "I am sad" may be suspect.

  4. You can also look at correlations among the questions and then identify people who have very different patterns.

Of course, none of these are fool-proof. But they are ways to investigate the issue.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • Thank You Peter, very nice tips. As I thought, more techniques in this area are not based on classical outlier detection approaches but rather on analyzing human behaviour. – sitems Oct 09 '12 at 14:09
  • 1
    +1 Peter. One situation that comes to mind is, for example, doing an online market research survey and being paid some nominal amount (there are websites you can do this on) and the more surveys you take, the more you are compensated (one could make **maybe** $100 a month doing this if they are pretty active). But, this dynamic would encourage just randomly filling in answers as quickly as possible, maybe not even reading the question. If you're using (2) from your list - do you just throw out the survey if there are contradictory answers? Are there other ways to detect "random answering"? – Macro Oct 09 '12 at 14:25
  • Thanks @macro. With any of these, judgement plays a role. Sometimes you can figure out what the "right" answer is. Sometimes you can figure that the person is just messing around. Sometimes you can't tell.... Sensitivity analyses may be called for. – Peter Flom Oct 09 '12 at 14:44
  • 1
    I would also have a look at the questionaire itself!!! ¿Is it to long? Does it appear badly designed in other aspects? I guess that to get good answers from a badly designed or toooo long questionaire would be very difficult ... – kjetil b halvorsen Oct 09 '12 at 16:14