4

For one of my academic project I ran a sample survey. But unfortunately the respondents were not very proactive & I get just 25 responses for my study. With this small response I guess I can't do any statistical studies (e.g. anova, regressions, etc).

So, I was wondering if based on the sample's mean & standard deviation I can simulate a larger dataset & then do my statistical analysis.

I'll also be grateful if somebody can tell me the method to do simulation with given mean & sd in R software.

Regards,
Ari

Beta
  • 5,784
  • 9
  • 33
  • 44
  • 6
    You can do statistical studies with any amount of data, even just one response. The important question is whether there is any valid way to draw conclusions beyond the sample itself. When non-response rates are high, one usually cannot say much, if anything, no matter how sophisticated or extensive the analyses of the data may be. The usual dodge for academic projects is to acknowledge this and then explicitly to pretend the sample is representative and conduct the planned analyses anyway, regardless of how small the dataset may be. – whuber Jul 25 '11 at 14:20
  • Also see http://stats.stackexchange.com/questions/726/famous-statistician-quotes/727#727 :-) – whuber Jul 25 '11 at 14:54
  • @whuber: Thanks for your comment & the link. The link was really funny :) I read it before, but reading again was refreshing. I just want to add one ques regarding your comment. Suppose I have 3 data points & 4 variables & I was to do a regression analysis. Can I still do the analysis within this setting? I'm not worried about interpretation of the result. But is this analysis feasible? Hope I made clear myself! – Beta Jul 26 '11 at 10:18
  • When there are more variables (including the constant) than data points you will usually get a perfect fit--assuming your software can handle collinearities, which is a mathematical, not a statistical, problem--so *something* can be done but it's not very informative and yields no information about error. Either collect more data or restrict the number of variables. – whuber Jul 26 '11 at 13:27
  • Thanks Whuber!Your comment made things bit more clearer. – Beta Jul 26 '11 at 14:36

3 Answers3

3

You should read about resampling.

Such techniques cannot increase the amount of information you have from your small original sample, so you should just take the mean and standard deviation of your sample, but resampling can give some helpful information about how uncertain the estimates you make from your sample are.

Henry
  • 30,848
  • 1
  • 63
  • 107
3

The first thing you need to do is determine (or at least think about) whether the poor return rate skewed your data. Ie, is there a statistical difference between the "average" person you attempted to survey (who was presumably representative of the population you wanted to study) and the "average" person who chose to respond?

You don't want to end up electing Alf Landon.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
Daniel R Hicks
  • 260
  • 1
  • 3
  • 14
  • Good advice, @Daniel! Often we need reminders to think about the statistical issues instead of just plunging into calculations. – whuber Jul 26 '11 at 13:31
  • (For those who may not remember, in spite of the Literary Digest predicting a landslide for Landon, Roosevelt was the one with the landslide, and Landon picked up only two states with 8 electoral votes. There has been some disagreement as to the particulars, but there's a general agreement that the Literary Digest poll was seriously skewed due to poor sampling -- they polled their own readers plus some addresses gleaned from telephone books and auto registrations. This resulted in a very lopsided sample, skewed towards more wealthy citizens.) – Daniel R Hicks Jul 26 '11 at 16:02
2

Simulating new data with the same mean/sd as your data and doing analysis on that is effectively the same as arbitrarily increasing your sample size while keeping your point estimates the same, thus increasing power, all while ignoring the fact that the mean/sd you've matched was based on only 25 observations - not a very diligent practice.

First off, if your data is normally distributed, then the $p$-values from ANOVA/regression are valid, since the sampling distribution of your estimates will still be normal, regardless of the sample size. More generally, you could consider non-parametric bootstrap resampling to get confidence intervals:

http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29

Essentially, you would re-sample from your data with and re-estimate a parameter (e.g. regression coefficients) repeatedly (say 1000 times) and treat these parameter estimates as draws from the sampling distribution of $\hat{\beta}$. You can then use the empirical quantiles (e.g. using the 2.5th and 97.5th percentiles) of this sample to construct confidence intervals.

As a side note, if the effects are large, and you can make a rational story from them you shouldn't over-emphasize the ability to formally "prove" your claims statistically just because your data is non-normal or your sample size is insufficient to invoke the central limit theorem.

Macro
  • 40,561
  • 8
  • 143
  • 148
  • Thanks Macro! I took time to reply to your answer as I'm still grappling with the fact that bootstrapping really solve my problem. Once I understand the bootstraping & how it's solving my problem I'll mark yours (& Henry's) answer as correct answer. – Beta Jul 26 '11 at 10:21