4

Given a generic mixture model $X$ of $k$ components, with distribution

$$ f(x)=∑_i\pi_if_i(x), $$

It is easy to show that the $k-th$ moment is just the weighted mean of the $k-th$ moments of the mixture components

$$ E_f[x^k]=∑_i\pi_i E_{f_i}[x^k] $$

However I can not find any result to estimate the weighted mean of the variances of the mixture components

$$ ?=∑_i\pi_i Var_{f_i}[x] $$

Is there any known estimator for such a value? Consider that the $\pi_i$ are unknown and the the $f_i$ come from the same distribution.

Should I do any hypothesis about the distribution of the $f_i$? I would like to work in the highest possible generality.. Appearently it looks very simple but I can't come out with a result.

Ulderique Demoitre
  • 271
  • 3
  • 8
  • 1
    The answer by whuber [here](http://stats.stackexchange.com/questions/16608/what-is-the-variance-of-the-weighted-mixture-of-two-gaussians?rq=1) might be what you are looking for. – Greenparker Apr 13 '16 at 20:01

1 Answers1

3

This answer by whuber is close to what you are seeking. The variance of a mixture is not the weighted average of the variance since there is also additional variability due to the component uncertainty. If you start from $$\mathbb{E}[X^2]=\sum_{i=1}^k \pi_i \mathbb{E}_{f_i}[X^2]$$ you can see that $$\mathbb{E}[X^2]=\sum_{i=1}^k \pi_i \{\text{var}_{f_i}(X)+\mathbb{E}_{f_i}[X]^2\}=\sum_{i=1}^k \pi_i\text{var}_{f_i}(X) + \sum_{i=1}^k \pi_i\mathbb{E}_{f_i}[X]^2$$ while $$\mathbb{E}[X]^2=\left(\sum_{i=1}^k \pi_i \mathbb{E}_{f_i}[X]\right)^2=\sum_{i=1}^k \pi_i^2 \mathbb{E}_{f_i}[X]^2+\sum_{i\ne j}\pi_i\pi_j\mathbb{E}_{f_i}[X]\mathbb{E}_{f_j}[X]$$ There is thus no direct moment expression linking $\sum_{i=1}^k \pi_i \{\text{var}_{f_i}(X)$ with an expectation under $f$. The quantity is a conditional variance in that $$\text{var}_f(X)=\mathbb{E}_\pi[\text{var}(X|Z)]+\text{var}_\pi(\mathbb{E}[X|Z])$$ where $Z$ denotes the auxiliary variable "component of $X$", i.e. $$X|Z=z\sim f_z(x)$$ In this representation, $$\text{var}(X|Z=z)=\text{var}_{f_z}(X)$$ and $$\mathbb{E}_\pi[\text{var}(X|Z)]=\sum_{i=1}^k\pi_i \text{var}_{f_i}(X)$$ while $$\mathbb{E}[X|Z=z]=\mathbb{E}_{f_z}[X]$$ and $$\text{var}_\pi(\mathbb{E}[X|Z])=\sum_{i=1}^k \pi_i \mathbb{E}_{f_i}[X]^2 - \left\{ \sum_{i=1}^k \pi_i \mathbb{E}_{f_i}[X]\right\}^2\,.$$ This essentially rephrases the answer by whuber!

Community
  • 1
Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • Thanks for answering. In my case I can estimate easily from sample $E[X^2]$ and $E[X]^2$, but I have no clue on how to estimate $\sum_{i\neq j}\pi_i\pi_j E_{f_i}[X]E_{f_j}[X]$ nor $\sum_{i}\pi_i E_{f_i}[X]^2$ – Ulderique Demoitre Apr 13 '16 at 20:07