4

I am trying to estimate the probability of an event using a low number of observations. The naive estimator

$\hat{p} =\frac{\text{number of positive observations}}{\text{total number of observations}}$

works well when the total number of observations is big enough, but if you have only a few observations, there is a decent chance that you will erroneously conclude to a 0 or 100% probability.

I suppose you could set a prior distribution on the estimated probability (say, uniform), and look for better estimators. I suppose this problem has already been tackled many times, so where should I look?

static_rtti
  • 745
  • 1
  • 11
  • 24
  • 3
    [This document](http://www.upassoc.org/upa_publications/jus/2006_may/lewis_small_sample_estimates.pdf) might be of interest. There is an [online calculator](http://www.measuringusability.com/wald.htm) that calculates point estimates and confidence intervals based on the different methods discussed in the paper. – COOLSerdash Jul 12 '13 at 09:32
  • 1
    With small N, another, but related, problem to consider is that the number of possible estimates is fairly limited. Say N=10, then there are only 11 possible estimates of $\hat{p}$: 0%, 10%, 20% ... 90%, 100%. This is probably trivial, but it is also suprising easy to overlook. – Maarten Buis Jul 12 '13 at 09:37

1 Answers1

2

Yes, if you do actually have some prior information, the Bayesian approach might be helpful. In which case you might want to consider a Beta distribution for the prior, because this is a conjugate prior when your random variable is binomially distributed, i.e. makes the maths simple by also giving you a Beta distribution for the posterior. So with $X \sim B(n,\theta)$, and a Beta prior, $\theta \sim \mathrm{Beta}(a,b)$, then with $x$ successes in $n$ trials, your posterior is given by $\theta|x \sim \mathrm{Beta}(a + x,b + n -x)$.

The uniform distribution is a special case of the Beta distribution (for the parameters $a=b=1$). However, a uniform prior is uninformative (it's flat, i.e. asserts each possible value of $\theta$ is equally likely), and you don't do any better than the classical approach (in fact I believe you essentially get the same point estimate / credible interval). So if you don't have any prior information, it looks like @COOLSerdash's link in the comment on your question is the way to go to get appropriate point estimates / confidence intervals.

Note that in Bayesian statistics, credible intervals have the same role as the confidence intervals of classical statistics. If you don't have any prior beliefs (your prior is uninformative, or non-informative), your credible intervals will, under certain conditions, correspond to the confidence intervals you would have got from a classical approach.

TooTone
  • 3,631
  • 24
  • 33
  • 1
    +1 and a short note for @static_rtti: see [here](http://stats.stackexchange.com/a/58792/21054) for a complete worked-out example using the $\mathrm{Beta}$ for proportions. – COOLSerdash Jul 12 '13 at 16:58
  • One thing I don't understand about your answer: you say the uniform distribution is uninformative. However if I look at the expected value of the posterior beta distribution, it already looks like a better estimator than my naive one. Am I correct? – static_rtti Jul 13 '13 at 08:51
  • @static_rtti This is a question in itself! Briefly, Bayesian statistics says how beliefs should change when you get new data. So if, e.g., you have a strong belief about θ and get only a few data points, your belief won't be changed much by the data. But if you have a weak belief about θ and you get lots of data, your belief will change according to the data. If you have no beliefs, i.e. think all possible values of θ are equally likely (uniformly distributed), then you have an uninformative prior and your posterior is completely determined by the data, just as a classical confidence interval. – TooTone Jul 13 '13 at 13:02
  • @static_rtti You might want to take a look at this question and its answers: http://stats.stackexchange.com/questions/64259/what-is-meant-by-a-uniform-prior. – TooTone Jul 14 '13 at 11:43