12

I'm developing a questionnaire. To improve its reliability and validity I want to use statistical methods.

I want to eliminate questions whose answers are always the same. This means that nearly all participants gave the same answers on those questions.

Now my questions are:

  1. What's the technical term for such useless questions whose answers are always the same, independently from the context of use?
  2. What are methods to identify such questions?
StatiStudent
  • 172
  • 3
  • 10
Max
  • 121
  • 4
  • 4
    I don't know of a special name for questions where everyone gives the same answer (maybe someone else does). I would probably call them 'uninformative'. They probably don't cause much harm other than wasting respondents time. You should definitely get rid of them, but I might focus more generally on finding & removing questions that don't correlate w/ ('load on') the latent variables you want to assess. – gung - Reinstate Monica Feb 19 '14 at 16:26
  • 7
    One rule of thumb is that if 80% of respondents give the same answer, the question is no informative. BUT - sometimes you want to know that. "Are you a murderer" is not an informative question by that rule, but you'd really want to know it before you got a new roommate. So there's no hard and fast rule. – Jeremy Miles Feb 19 '14 at 16:59
  • 4
    Such questions (with very low variability) are bad as measures of the trait they belong to. But they are sometimes included and helpful to track down liar or shirker respondents. – ttnphns Feb 19 '14 at 17:11

2 Answers2

9

Both classical test theory (CTT) and item response theory (IRT) can provide guidance as far as which items are contributing to the latent trait you wish to measure, and which do not. With CTT, consider 1) item difficulty, 2) item correlation to total score, 3) item variance, and 4) impact on internal consistency estimates (e.g., Cronbach's alpha) if the item is removed.

Items that are too easy or too difficult tend not to help separate subject (discriminate between high scorers and low scorers). Unless you are interested in measuring differences between top performers, very difficult questions should be considered for removal. In a similar vein, very easy items are only suitable if you are interested in the performance of low performers.

All items should correlate positively with total score and you can set a lower bound for that correlation of around 0.20 as a guide. Low correlations or negative correlations may indicate that there are wording problems in your questionnaire and that the question should be reversed scored.

Items with low variance (variability of scores) should be considered for removal as they don't separate subjects and don't contribute to the information gathered from the survey. Items with very high variance may be measuring something else than the construct/trait you wish to measure.

If the estimate of internal consistency improves with the item removed, then the item should be considered for removal, or re-worded.

Items that everyone gets correct are sometimes maximum items and those everyone gets wrong are sometimes called minimum items. They don't contribute to the information you are trying to gather.

If you are developing a high stakes questionnaire or plan on marketing the questionnaire you should definitely consider IRT. However, it is a large subject area and unless you are truly interested it is not probably worth the space to get into it here.

Hope this helps.

StatiStudent
  • 172
  • 3
  • 10
doug.numbers
  • 833
  • 5
  • 16
4

I believe what you are looking for is Item Response Theory. The "useless" questions you refer to are items with poor discrimination. Using IRT analysis you can calculate the discrimination, difficulty, and the associated probability of guessing on items by survey participants. The R program has an easy package for using IRT and I imagine other statistical software packages do as well.

If you want a quick overview here's the wikipedia page, but I would advise researching it more. http://en.wikipedia.org/wiki/Item_response_theory

StatiStudent
  • 172
  • 3
  • 10