8

I was surprised not to find anything about this with Google.

Consider a geometric distribution with $\text{Pr}[X=k]=(1-p)^{k-1}p$, so the mean is $\sum_{k=1}^\infty k\,\text{Pr}[X=k]=\frac{1}{p}$.

Now assume we observe a single outcome (number of trials until success, including the success) $n$. What is our estimate of $p$? The "natural" estimate (whatever that means) seems to be $\frac{1}{n}$. However, this is a biased estimate of $p$. Indeed, we have $\sum_{k=1}^\infty \frac{1}{k}\text{Pr}[X=k]=\frac{p}{1-p}\log\frac{1}{p}$.

Can we find an unbiased estimate of $p$ from $n$? What is the MSE estimate of $p$?

Many thanks.

Tarvoc
  • 115
  • 1
  • 5
  • 2
    You seem to have caused yourself a problem by using the same symbol for both the observed outcome ($k$) and also the dummy variable in your sum ($k$ again). You should avoid that. – Glen_b Aug 03 '15 at 14:53
  • Ok, I changed some $k$'s to $n$. – Tarvoc Aug 03 '15 at 14:57
  • I calculate $\sum_{k=1}^\infty \frac{1}{k}\text{Pr}[X=k]$ to see if $\frac{1}{n}$ is an unbiased estimate of $p$. If it were, the sum should evaluate to $p$. – Tarvoc Aug 03 '15 at 15:23
  • @Glen_b I didn't see anything the matter with the original notation, nor does it appear that using "n" has clarified anything. There is no notational problem with using the same symbol for a bound variable in the sum and a specific value outside the sum--it is well-defined and unambiguous. Tarvoc: what do you mean by an "MSE estimate"? – whuber Aug 03 '15 at 15:24
  • @whuber: What I mean is: Given an outcome $n$, what is the function $f$ which minimizes $E[(f(n)-p)^2]$? – Tarvoc Aug 03 '15 at 15:30
  • @whuber You're right. I don't know what was going on in my head; this cold seems to have addled my brain. Tarvoc, my apologies, I have needlessly caused you work. Feel free to roll it back. – Glen_b Aug 03 '15 at 15:33
  • 1
    Perhaps you should look at Rao-Blackwell estimators (and further, perhaps for the combination $p(1-p)$) as in this question: http://math.stackexchange.com/questions/384929/rao-blackwell-unbiased-estimator-geometric-distribution – Eric Towers Aug 03 '15 at 23:10

2 Answers2

5

By definition, an estimator is a function $t$ mapping the possible outcomes $\mathbb{N}^{+} = \{1,2,3,\ldots\}$ to the reals. If this is to be unbiased, then--writing $q=1-p$--the expectation must equal $1-q$ for all $q$ in the interval $[0,1]$. Applying the definition of expectation to the formula for the probabilities of a geometric distribution gives

$$1-q = \mathbb{E}(t(X)) = \sum_{k=1}^\infty t(k) \Pr(X) = (1-q)\sum_{k=1}^\infty t(k) q^{k-1}.$$

For $q\ne 1$ we may divide both sides by $1-q$, revealing that $t(k)$ are the coefficients of a convergent power series representation of the function $1$ in the interval $[0,1)$. Two such power series can be equal in that interval if and only if they agree term by term, whence

$$t(k) = \begin{cases} 1 & k=1 \\ 0 & k \gt 1 \end{cases}$$

is the unique unbiased estimator of $p$.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
-2

UPDATE. Re-writing my previous sloppy answer.

There is no unbiased estimator for $p$, here is the proof.

The estimator $p=\frac{1}{k}$ is biased, but it's best you can get in the sense of MLE or method of moments. Here's the derivation in Math SE.

Community
  • 1
Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • 2
    In what sense is this method of moments estimate "best"? – whuber Aug 03 '15 at 15:16
  • @whuber, it's unbiased, and I'm pretty sure efficient, but I removed "best". I'm too lazy to prove it. – Aksakal Aug 03 '15 at 15:27
  • 1
    Thanks for the link. In what way do I not compute the bias properly? The bias is defined as $\sum_{k=1}^\infty \frac{1}{k}\text{Pr}[X=k]$ minus $p$. Since this is non-zero, the estimator is biased. – Tarvoc Aug 03 '15 at 15:27
  • @Tarvoc, look at Glen's comments. He's hinting you to issues in your formula – Aksakal Aug 03 '15 at 15:28
  • 2
    @Aksakal: Glen_b retracted his/her comments. In the link to Math SE that you posted, the conclusion is actually that $p=1/k$ _is_ biased (see the last two comments to the accepted answer). – Tarvoc Aug 03 '15 at 15:40
  • @Tarvoc, me too – Aksakal Aug 03 '15 at 15:48
  • 1
    You appear to have misread your new reference, which shows there is a *unique* unbiased estimator, not that there is no unbiased estimator! – whuber Aug 03 '15 at 15:51
  • @whuber, it's for p=0 and 1, degenerate cases. – Aksakal Aug 03 '15 at 17:39
  • 2
    I'm sorry, I don't see anything in the question or your answer that indicates you are discussing only those two cases. Remember, an unbiased estimator has to be unbiased for the entire parameter space, not just for some of the possible values. If you still think that an analysis of these cases responds to the question, then please clarify that in your answer. – whuber Aug 03 '15 at 17:43