0

I have a set of $n$ realizations $y_j \in \{a,b,c,...\}$ in an alphabet with $k$ symbols.

Each letter from this alphabet has an unknown prior probability $x_i$. The goal is to estimate $\mathbb{x}$ from the observations $\mathbb{y}$ so as to minimize the mean square error.

I assume the probabilities $x_i$ follow a uniform distribution, with the constraint that $\sum_{i}{x_i}=1$.

I already know that the case where the alphabet has only two letters is called the bayes estimator with beta prior, which has well-known solutions. However, I'm at loss for the more general case with $k$ letters.

Any insight?

Community
  • 1
static_rtti
  • 745
  • 1
  • 11
  • 24
  • 1
    If your data come from a multinomial distribution then the Dirichlet distribution is the conjugate prior, which generalizes the Beta distribution to $K$ dimensions. http://en.wikipedia.org/wiki/Dirichlet_distribution#Conjugate_to_categorical.2Fmultinomial – jerad Feb 19 '14 at 22:18
  • @jerad - might want to expand that a little and make it an answer, I'd upvote it. – jbowman Feb 19 '14 at 23:02
  • @jerad: I unfortunately don't think the Dirichlet prior applies, because of the constraint that the prior probabilities must sum up to one. Of course I'd love to be proven wrong :) – static_rtti Feb 20 '14 at 08:27

1 Answers1

0

Turns out @jerad was right: the problem works out very nicely with a Dirichlet prior distribution. This pdf answers the question in-depth.

In the end, with a uniform prior distribution, the minimum mean square estimator is:

$p_i = \frac{n_j+1}{n+k}$

Where $n_j$ is the number of times the j-eth symbol has been observed.

static_rtti
  • 745
  • 1
  • 11
  • 24