0

Social science PhD student here looking for a little help.

I set out to collect around 300 online surveys for a study, but ended up with over 4,000. At that sample size should I sub-sample my results because it's more likely that significant results will be found because of the sample size?

Thanks.

user48131
  • 1
  • 3
    Significance tells you if the effect is 'real' or not, not how much it matters. Look at the magnitude of the effects to see which ones are actually important in explaining the phenomenon you're researching. – conjectures Jun 10 '14 at 20:50
  • 3
    I think the bigger concern would be the likely effect of a self-selected sample on your inference. – Glen_b Jun 10 '14 at 21:02

1 Answers1

1

Minor but real effects are more likely to be identified if you have a larger sample size.

This is generally a good thing, so long as it is presented in a helpful way. For example, reporting confidence intervals in sensible units (the intervals will be narrower with a larger sample) rather than $p$ values can help the reader understand whether a reported result is worth worrying about.

False positives, i.e. apparently statistically significant results despite the null hypothesis being correct, are just as likely with small or large sample sizes. But with large sample sizes they will probably seem less important when quantified.

As for bias, such as Glen_b's self-selection point, it may be more, less or the same. You should consider why the response was higher than originally expected: did a single person or group run a campaign to generate particular responses; or did almost everybody you ask agree to respond when you had expected only a few to do so?

Henry
  • 30,848
  • 1
  • 63
  • 107