5

One objective of a survey can be to understand the proportion male vs. female users. To the extent a specific gender correlates to particular use cases of a technology, product design, product/feature prioritization and marketing can be differently managed.

A survey of existing users will inherently have error bars based on sample size, frequency of sampling, etc. As I understand it, when asking the question 'are you male or female?' as part of a demographic survey where all fields are required, an additional selection bias is introduced into the survey.

Specifically, if a survey is sent to 100 users, of which 50 respond, the proportion of responders that are male may not represent the proportion of 100 customers that were initially surveyed.

Assuming there is not a existing sample set of known males and known females whom to survey in order to estimate the responder bias, what are some approaches to prevent or to correct for this error?

user2661
  • 151
  • 1
  • 3
  • @jonnyboats - I did not think my question was as quantitative as the questions I saw on stats. happy to repost if a few others think it qualifies. –  Feb 14 '12 at 12:19

4 Answers4

4

In some cases it is possible to make adjustments and "correct" survey non-response. The topic is i.a. discussed in this post. However, your situation seems to be a bit special. You have a list of users, but no information about their profiles.

The best thing to do about non-response, be it in your particular case or in a more general case, is to avoid it. When designing a survey a couple of measures can be taken. What follows is just some general advice. You have to see what can be applied in your particular situation.

  • When designing the questionnaire you have to stick firmly to the "KISS" principle: keep it short and simple! Only ask for the items an characteristics you really need. Only ask for the detail you really need. For instance, Rather than asking an exact birthday, ask for the age in years, rather than asking for a city, ask for a broader region, ...
  • Make sure that your questions are intelligible and rather straightforward to answer
  • The collection mode can also have an impact on the response rate. Some people might claim that face-to-face interviews yield the best rates. However, depending on the topic, respondents might feel more at ease with more anonymous online or telephone surveys.
  • Incentives may increase the response rate. The respondents can either be paid for taking part or they can, or a prize draw can be organized among the respondents.
  • If interviewers are used, they have to convince the persons to respond.
  • Similarly, follow-up and reminders are very important. If after a first contact, a person did not categorically refuse to take part in the survey, one or more further attempts should be done.
  • Confidence is crucial. If you want to obtain some more confidential information, or information that is perceived as such by the respondents, you have to be trustworthy. You need to guarantee that confidential data will be treated as such.
Community
  • 1
1

Make it optional to answer the gender question. That way, it's more likely to be accurate.

You will never know for sure the exact number of males and females because there will always be some sort of sampling error in your data -- you can't avoid that unless you interviewed every single customer.

Selecting how many samples you need for the correct confidence interval requires a calculator (like this one http://www.macorr.com/sample-size-calculator.htm).

  • 2
    Even if i made the gender question optional, if for some reason males are more likely to respond than females doesnt the error remain? –  Feb 14 '12 at 16:26
  • Can you give me any references for this interesting finding about optional items? – rolando2 Feb 15 '12 at 19:42
1

You called them customers. What is the nature of the relationship? Is it possible to gather the information for billing purposes? Of course if you are dealing with businesses the contact/billing person is the choice of the business, they may not be the decision maker.

mhoran_psprep
  • 111
  • 1
  • Its a subscription service where majority of customers are billed annually, and some are billed monthly. Are you suggesting some sort of coupon or rebate if a survey is filled out such that the responders as a % of total surveyed is high enough to mitigate the potential error? –  Feb 14 '12 at 16:28
  • linking the completion of a survey to a discount may increase the number of people responding, but it might not make the results more accurate. – mhoran_psprep Feb 15 '12 at 13:50
  • I thought about this more and corrected my question to focus on users not customers. For a freemium service the broader objective is to survey users most of which are unbilled. For paying customers, your answer is helpful. Thank you. – user2661 Feb 15 '12 at 16:58
1

This may be obvious, but I didn't notice it mentioned. I'd look for gender differences on every item on the survey--through comparisons of means, correlations, crosstabs with chi-square tests, even perhaps with logistic regression. If you find no substantial differences, great: gender doesn't matter to representativeness, at least when it comes to this survey's set of variables. If you do find big differences, you can weight your data so that aggregate results, at least, will be more in line with the proportion of (fe)males that you estimate (if you can) as existing in the larger population.

rolando2
  • 11,645
  • 1
  • 39
  • 60