How to design an algorithm for k means-like problem?

Question

I have some observation about a random variable x and I want to estimate the parameters of its probability distribution function. For example: data = {1, 2, 3, 4, 5} and choose normal distribution as prior, then I can compute mu and sigma. However, if the data become {1, 1.1, 0.9, 100, 101, 99}, I will want the probability distribution function has two peaks(or centers) and mu1=1, mu2=100.

1. What is the terminology of this problem?
2. If I has a prior that the number of centers is 1 or 2, how to design an algorithm to estimate mu and sigma by assuming each center is a normal distribution?

Answer 1

The model you seem to be talking about is mixture distribution. Finite mixture model defines the distribution of the data as

$$ f(x) = \sum_{k=1}^m \pi_k \, f_k(x,\, \vartheta_k) $$

where $m$ is the number of components, $\pi_k \ge 0$ and $\sum_{k=1}^m \pi_k = 1$ are mixing weights, and $f_k$ are the components, the distributions with their own parameters $\vartheta_k$, in your example they are all Gaussian.

There are finite and infinite mixtures, in the second case the number of components is unlimited rather than fixed. You can use Bayesian approach to fitting them, with using priors for the parameters.

See also other threads tagged as mixture, or gaussian-mixture.