2

Suppose $X_1, X_2, \dots, X_N$ are $N$ random samples of a discrete probability distribution such that $X_i \in \{1, 2, \dots, K\}$.

The probability distribution $p$ used for sampling is parameterized by $\pi_1, \pi_2, \dots, \pi_K$ where $\pi_i$ denotes the probability of occurrence of the ith category.

I have a few questions about the problem of parameter estimation (estimation of $\pi_1, \pi_2, \dots, \pi_K$ from $x_1, x_2, \dots, x_N$) of $p$:

  1. Is the problem of parameter estimation of $p$ ill-posed?
  2. If not, how can the parameters of $p$ be estimated?
Bharat Khatri
  • 295
  • 2
  • 10

2 Answers2

2

Is the problem of parameter estimation of p ill-posed?

No, it's fine. You're trying to estimate a set of multinomial population probabilities. The parameter is vector valued, with a linear restriction that $\sum_i \pi_i=1$.

(There's no unique definition of how to estimate a parameter, of course. But I don't think that's what you're asking, because that's the case with parameters in general)

If not, how can the parameters of p be estimated?

As with other parameters, in all manner of ways - define what criterion you want to optimize, and away you go. Maximum likelihood is the most common and corresponds to the "obvious" estimator.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • It's not equivalent to a multinomial distribution as $X_i$ doesn't represent the number of occurrences of the i*th* category, but the outcome of the i*th* trial, and so $P(X_1 = x_1, X_2 = x_2, \dots, X_N = x_n)$ is different from the probability mass function of a multinomial distribution. – Bharat Khatri Feb 13 '15 at 10:28
  • Bharat, your comment frankly contradicts the information in your question. I will therefore vote to close this thread until you can clarify what is actually going on. – whuber Feb 13 '15 at 14:47
0

It is obviously ill posed as there is no unique mathematical solution to the problem.

The parameters can be estimated in infinitely many ways. Obviously, the empirical frequencies are natural estimators. If something can be assume on the $\pi$s, then other options become available. Say, if $p$ belongs to some low dimensional parametric family ($<k$), you can gain accuracy by estimating these parameters using maximum likelihood.

If you are not willing to assume a finite dimensional parametric family, you can still add regularization by assuming the marginal distribution of $\pi$s. This corresponds to a Bayesian approach. Say, you assume $\pi$s are beta distributed, you could use the mean (or the mode, or the median) of the posterior distribution of $\pi$ given the observed frequency of category $i$ as an estimate: $$ \pi \sim Q \;; \hat{\pi}_i:=\mathbb{E}[Q|x_i,N] $$

JohnRos
  • 5,336
  • 26
  • 56