What if your random sample is clearly not representative?

Question

What if you take a random sample and you can see it is clearly not representative, as in a recent question. For example, what if the population distribution is supposed to be symmetric around 0 and the sample you draw randomly has unbalanced positive and negative observations, and the unbalance is statistically significant, where does that leave you? What reasonable statements can you make about the population based on a biased sample? What is a reasonable course of action in such a situation? Does it matter when in our research we notice this imbalance?

Answer 1

The answer given by MLS (use importance sampling) is only as good as the assumptions you can make about your distributions. The main strength of the finite population sampling paradigm is that it is non-parametric, as it does not make any assumptions about the distribution of the data to make (valid) inferences on the finite population parameters.

An approach to correct for sample imbalances is called post-stratification. You need to break down the sample into non-overlapping classes (post-strata), and then reweight these classes according to the known population figures. If your population is known to have a median of 0, then you can reweight the positive and negative observations so that their weighted proportions become 50-50: if you had an unlucky SRS with 10 negative observations and 20 positive observations, you would give the negative ones the weight of 15/10 = 1.5 and the positive ones, 15/20 = 0.75.

More subtle forms of the sample calibration do exist, in which you can calibrate your sample to satisfy more general constraints, such as having a mean of a continuous variable to be equal to the specific value. The symmetry constraint is pretty difficult to work with, although that might be doable, too. May be Jean Opsomer has something on this: he has been doing a lot of kernel estimation work for survey data.

Answer 2

I'm the Junior Member here, but I'd say that discarding and starting over is always the best answer, if you know that your sample is significantly unrepresentative, and if you have an idea of how the unrepresentative sampling arose in the first place and how to avoid it if possible the second time around.

What good will it do to sample a second time if you'll probably end up in the same boat?

If doing the data gathering again doesn't make sense or is prohibitively costly, you have to work with what you have, attempting to compensate for the unrepresentativeness via stratification, imputation, fancier modeling, or whatever. You need to clearly note that you compensated in this way, why you think it's necessary, and why you think it worked. Then work the uncertainty that arose from your compensation all the way through your analysis. (It will make your conclusions less certain, right?)

If you can't do that, you need to drop the project entirely.

Answer 3

This is a partial answer that assumes we know both the distribution $q$ from which was sampled, and the true (or desired) distribution $p$. Additionally, I assume that these distributions are different. If the samples were actually obtained through $p$, but they look wrong: the samples are still unbiased and any adaptation (such as removing outliers) will likely add bias.

I assume you want to find some statistic $s_p = E \{ f(X) | X \sim p \}$. For instance, $s(p)$ might be the mean of the distribution, in which case $f$ is the identity function. If you had samples $\{ x_1, \ldots, x_n \}$ obtained through $p$, you could simply use $$ s_p \approx \frac{1}{n} \sum_{i=1}^n f(x_i) \enspace. $$ However, suppose you only have samples that were obtained (from the same domain) with a sampling distribution $x_i \sim q$. Then, we can still get an unbiased estimate of $s_p$ by weighting each of the samples according to the relative probability of it occuring under each distribution: $$ s_p \approx \frac{1}{n} \sum_{i=1}^n \frac{p(x_i)}{q(x_i)} f(x_i) \enspace. $$ The reason this works is that $$ E \left\{ \frac{p(X)}{q(X)} f(X) \middle| X \sim q \right\} = \int p(X) f(X) dx \enspace, $$ as desired. This is called importance sampling.