5

I have conducted a survey in which the questionnaires were sent out to 450 individuals, but only 30% of them answered the questionnaires.

  • Is it still valid to interpret the usual inference analysis (i.e., the inference analysis developed under the assumption of random sampling)?
  • Is it correct to do logistic regression analysis with these data?
  • If not, how to proceed?
Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
PaulS
  • 425
  • 4
  • 15
  • 3
    Do you have any information about the people who did not respond? You can always proceed with the analysis, assuming that the sample you have collected is still random. Ultimately though non-response diminishes the validity of the results. – Andy W Aug 02 '11 at 14:20
  • 5
    Please consider formally accepting answers to this question and/or some of your previous ones. (I realize this one has only been open a few hours, but you have a handful of others that are weeks or months old.) – cardinal Aug 02 '11 at 20:19

6 Answers6

14

If it is "correct" to do logistic regression with these data depends on the type of non-response you have. Usually, one distinguishes three types of mechanisms for non-response.

  1. Missing completely at random: The non-response does not depend neither on the variable of interest nor the covariates.

  2. Missing at random, given some covariates: The non-response depends on some covariates but not on the variable of interest. Some people call this "ignorable non-response"

  3. Not missing at random: The non-response depends on the variable of interest, and cannot be completely explained by the observed covariates.

If you are in the first situation, non-response does not bias the results. It will merely induce a loss of precision in your estimates

If you are in the second situation you should be able to model non-response successfully and to adjust the data accordingly.

If you are in the third case, you are unlucky. There is not much you can do.

If you want to read about the topic, have a look at a textbook such as e.g. Sharon Lohr's "Sampling: Design and Analysis"

HTH

  • 1
    +1 for a clear summary, nicely adapted to the level at which the question is posed. BTW, welcome to our site! – whuber Aug 02 '11 at 17:35
5

If at all possible, be very careful about scaling the results of your analysis. Non-response tends to be related to interest. For example, regardless of demographic match, people who fill out a survey about bus service tend to be people who are more interested in taking the bus than the average person. Therefore, estimates based on the survey respondents themselves tend to be much too high.

For example, the old rule of thumb about predicting consumer product sales from a standard 5-point buying intent question was: 75% of those who "definitely would buy", 25% of those who "probably would buy", and 5% of the rest. [Advertising Research Foundation Arrowhead project]

To directly answer the question, you can certainly do the logistic regression, but it is validly applied to the population you are analyzing (i.e., the population of survey respondents). Anything beyond that is an inference, similar to inferring opinions in Minnesota from surveys in Iowa and Wisconsin. That inference can be supported by some of the strategies suggested above, but is still an inference.

In many cases you MUST do the inference. The client paid for the survey, and wants to get at least some useful information out of it, so you have to give it your best shot, with whatever caveats you deem necessary.

k-dubs
  • 11
  • 4
zbicyclist
  • 3,363
  • 1
  • 29
  • 34
3
  1. Ignore it (at your risk). (30% is probably an excellent return rate by most standards.)
  2. Resample some of the non-responders (with a slightly different survey, and perhaps some "teasers" to gain interest), and see if they respond differently from the original sample (bearing in mind that you will get an even poorer response rate and more skew in this second sample).
  3. Analyze the demographics (and other interests) of responders vs the population (if you have any clues to demographics/interests in the survey). Scale responses based on that info.
Daniel R Hicks
  • 260
  • 1
  • 3
  • 14
2

Here is a simple element of answer

A simple way to decide whether or not having filled in the form is related to explanatory variables is to perform a logistic regression where the binary response variable is 1 if the person answered and 0 otherwise. If it turns out that having filled in the form is not related to explanatory variables then your sample of respondents may be seen as a random sub-sample of your sample, and hence as a random sample of the original population. Of course, the power of your tests will be affected...

You will find more if you look at the missing data problem.

Edit The previous technique requires a special framework --- see Whuber's comments.

ocram
  • 19,898
  • 5
  • 76
  • 77
  • Where, exactly, do you obtain *any* information about the people who didn't answer? What could the explanatory variables possibly be? – whuber Aug 02 '11 at 14:30
  • @Whuber: Maybe I misunderstood the situation... I assumed the information about the people who did not answer is available. As for the explanatory variables, I meant those that would be included in the subsequent analysis. – ocram Aug 02 '11 at 14:36
  • 2
    Usually, an analysis is based on the contents of the answers. When an answer is not forthcoming, there is nothing to analyze. In some circumstances, you might already know something about *all* the individuals (including the non-respondents), such as their gender, race, zip code, or whatever. In addition, *the respondents must be identifiable*: this rules out anonymous responses! In such cases, your suggestion is a good one. – whuber Aug 02 '11 at 14:48
  • @whuber: I have edited my answer according to your comments. Indeed, I made some assumptions and I did not mention them. Thank you for the clarification. – ocram Aug 02 '11 at 14:53
2

Hmm, I analyse data like this all the time. I know it's very naughty but I figure it's better than having no analysis.

The best I've ever come up with is to compare your sample with what you know about the population of interest. Are there more people from white backgrounds in the sample? Are there more women in the sample? Is the sample older than the population?

I've found this to be quite a reasonable approach with, say, a staff survey, where I can say

"well, there are a lot of doctors missing from my sample, nurses are over-represented. Based on other research I think doctors think x whereas nurses think y. So we should be careful where the missingness is consistent with this difference (e.g. everyone unhappy about pay) and reassured when it is inconsistent (e.g. everyone saying they feel supported to develop as a professional)".

Not very scientific, but as far as I know analysing data which is or could be MNAR never is!

I've always had some idea at the back of my head about thinking about WHY people don't respond (is it happy people, busy people, etc.) but I've never been able to formulate it properly (lack of time, data, etc.)

Chris Beeley
  • 5,465
  • 5
  • 36
  • 40
  • 2
    While these are some good answers, there is such a huge and fast-growing literature on this topic that it seems simplistic to try to reduce it to a question to be answered in one discussion thread. If you read further, you could look for terms such as survey validation, survey nonresponse, nonresponse bias, representativeness, and wave analysis. – rolando2 Apr 21 '12 at 11:25
1

Dealing with response rates gets less quantitative and more qualitative, as there are no statistical tests to compare the respondents and non-respondents. You can find some guidelines to computing and generally dealing with (low) response rates at http://www.aapor.org/Content/aapor/Resources/PollampSurveyFAQ1/DoResponseRatesMatter/ResponseRatesAnOverview/default.htm. I'd agree that 30% is a very good response rate on a mail-in survey; generally you'd expect response rates in single digit % for this mode these days.

If this is a list survey for an organization that has the information on its members, or for a medical provider who has the records, then you'd certainly want to do some matching of the sample with the existing records. You can either estimate response propensity models (assuming MAR, see an earlier comment), or you can calibrate the sampling weights to adjust the demographics of your survey to that of the original list.

Community
  • 1
StasK
  • 29,235
  • 2
  • 80
  • 165