1

Imagine I have a population of a known size (e.g., 100,000 voters), and I know they can only vote for A, B, C, D or E.

If a random sample of 100 voters are, e.g., A = 40%, B = 30%, C = 10%, D = 10%, E = 10%, when I incrementally increase my sample size to 200 people, count the votes, then 300, 400, etc (samples without replacement). What type of analysis can I use to determine which sample size is sufficient to say:

"For this population size (100,000) with 5 options (A, B, C, D, E), this sample size of x has a 95% probability of correctly detecting the winner (e.g, A > B)"

Once I can't use a power analysis without a H0-H1. Are there any other tests or metrics that look into how incremental sample sizes impact variance accuracy? This might be used to inform me that after a certain point increasing the sample size is unlikely to change the reliability of my prediction.

Sergio Henriques
  • 31
  • 5
  • It helps if you think of "effect size" as meaning "smallest meaningful difference that I care about". – Alexis Aug 17 '18 at 19:50
  • 1
    Possible duplicate of [power analysis of a proposed study](https://stats.stackexchange.com/questions/260308/power-analysis-of-a-proposed-study) – Karolis Koncevičius Aug 17 '18 at 20:22
  • It differ from this question, as I am specifically looking for a distinct metric/analysis from Power Analysis, that focus on precision of estimate, rather than the effects that I can expect to detect with a sample size of n (e.g. with power = 80%). – Sergio Henriques Aug 19 '18 at 08:59

2 Answers2

5

Traditional power analysis is situated within the discovery (or lackthereof) of population parameters based on statistical significance, p = .05 (or some other alpha).

You are correct in saying that you don't really have an alternative hypothesis against which you are testing, so it isn't as straightforward to do a power analysis as in other situations when you are asking, "If there is an effect here, how often am I going to capture it?"

You are onto something in the latter part of your question when you say:

Are there any other tests or metrics that look into how incremental sample sizes impact variance accuracy? This might be used to inform me that after a certain point increasing the sample size is unlikely to change the reliability of my prediction.

Instead of power analysis, which does sample size planning to get a "significant" p-value, you are interested in precision of a parameter estimate.

There's a good paper, Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609-612. doi:10.1016/j.jrp.2013.05.009, that examines sample size planning in the way you are looking at it (.pdf here: https://osf.io/5u6hv/).

They are looking at correlations, whereas you are interested in proportions, but the idea is the same. You could take a similar Monte Carlo approach by simulating different proportions (e.g., A = 40%, A = 75%, etc.) and doing different sample sizes and seeing when the standard errors start to plateau. That could give you the sample size at which you start to get diminishing returns in precision. So I think one way to get at your question is to think of it as looking for precision, not power.

In traditional power analyses (which could test a hypothesis like A < B), we don't really know the effect size, so we often (a) guess or (b) calculate it for many different effect sizes. This gives us an understanding of the range of effect sizes and what our power would be for each. You could also just do power for the smallest effect size you care about or, if you want to be safe, do it at the lower bound of previously-done research (http://journals.sagepub.com/doi/full/10.1177/1745691614528519).

Mark White
  • 8,712
  • 4
  • 23
  • 61
  • Thank you @MarkWhite. I had looked at a "plateau approach", but I found it difficult to determine a corridor of stability for parameter estimate (something I'll have to look deeper into). I also considered that if instead of votes, participants are placed in categories (i.e. A to F) representing increasing degrees of a feature (e.g. Health Risk-HR) when I assign them weights (e.g Extreme HR=0, Critical HR=0.2, Great HR=0.4, HR=0.6, Low HR=0.8, No HR=1) and just average these scores. Would it be possible to find an approach that insures that with a sample size of x, I can detected this value? – Sergio Henriques Aug 20 '18 at 08:30
3

You can, in a fairly straightforward fashion, run a number of power analysis calculations based on varying assumptions. For example, below is a figure where the effect size and a property of the ratio of people who had or didn't have a variable of interest were both varied.

enter image description here

Fomite
  • 21,264
  • 10
  • 78
  • 137