Controlling for biased audiences in online surveys

Question

For a long time I have been meaning to set up a bunch of online surveys asking a whole range of social questions and publish the results.

I am well aware that there are various difficulties to overcome in this goal (e.g. privacy, excluding bots etc.). The hardest problem to overcome for me is the fact that any online survey will always have an inherently biased audience. People who visit my websites will certainly be biased (young, technical, probably left-wing), and the same is true for every single website on the internet.

However, my feeling is that some data is always better than no data, especially if I'm going into it with my eyes open to the potential for biases.

My question

Are there established methods for controlling for and understanding exactly how your audience is biased? Are there certain control questions I could include in my surveys which can then be compared against a well-known normal distribution to understand a particular area of bias? Is this a common practice?

Edit: Phrased a different way: What methods / principles / areas of study exist for helping to understanding the ways in which the set of respondents to a given survey may be skewed as compared to a known population? And how reliably can this understanding then be used to draw conclusions from said survey about the known population?

Off the top of my head I can think of asking age or race (although some respondents might baulk at being asked these questions) and using these to better understand my sample, but are there guidelines around how to ask these sort of questions so as not to put people off? And are there particular subtle questions I can ask to uncover and understand more subtle audience biases, e.g. in political leaning or social class?

Most importantly, is this a well-known area of study which has a specific term in academia? If I know what it's called then I can more easily explore further reading on the subject.

Answer 1

The sources of bias are many in survey research from the type of scale used, the wording of the questions, the order in which the questions are asked, the sampling frame (e.g., random, stratified, convenience), the method used (e.g., online, mall intercept, telephone, f2f), nonresponse, data management, and so on. In fact, each of the topics mentioned here constitute classic subdisciplines within the broader domain of survey research or else overlap with independent and massively researched topic areas such as psychometrics, cognitive psychology, sampling, methodology, and so on. Degreed programs are offered in it, e.g., the University of Michigan's ICPSR (Inter-university Consortium for Political and Social Research). Entities exist which profit from leveraging these techniques, supposedly in an unbiased manner, to produce and publish the results from national and international opinion polls such as Gallup, Harris, Pew, Research International, NORC, the US Census Bureau, to name just a few.

The field of behavioral economics, in large part, exists as a function of its decomposition of the various "heuristics and biases" which we fallible humans are subject to. Judgement and decision-making broadly overlaps with this field as well.

So, yes, this is a "well-known area of study" in academia.

That said, a commonly used heuristic (at least in marketing) is to acknowledge that online surveys are biased convenience samples and reweight them to nationally available demographic marginals. This doesn't remove the biases but it makes the information more palatable to a naive and gullible audience which never looks to closely under the rocks.