5

I have 2 normally distributed random variable $H_0$ and $H_1$, which are combined to give the weighted distribution $H$ as follows:

$H_0 \sim N(\mu_0, \sigma_0)$

$H_1 \sim N(\mu_1, \sigma_1)$

$$f_H = p * f_1(x) + (1-p) * f_0(x),$$

where $H$ has pdf $f_H$ and $H_1$ and $H_0$ have pdfs $f_1$ and $f_0$ respectively. The mean value $\mu$ of the combined distribution $H$ is:

$\mu = p * \mu_1 + (1-p) * \mu_0$

Now, what is the standard deviation $\sigma$ of $H$?

Is there a simple formula for $\sigma$, in terms of $p, \mu_0, \sigma_0, \mu_1, \sigma_1$?

StasK
  • 29,235
  • 2
  • 80
  • 165
Kelvin
  • 1,051
  • 9
  • 18
  • See the relevant part of the [mixture distribution page](https://en.wikipedia.org/wiki/Mixture_distribution#Moments) at wikipedia; also can be found here on CV, e.g. [here](http://stats.stackexchange.com/questions/128912/possible-to-morph-a-bimodal-distribution-into-a-normal-distribution-slowly/129007#129007) – Glen_b Aug 11 '16 at 02:43

2 Answers2

7

(I am assuming $H_1$ and $H_0$ are independent)

Let $f_H = pf_1(x) + (1-p)f_0(x)$, where $f_1$ and $f_0$ are pdfs of $H_1$ and $H_0$. Then the random variable $H$ is the mixture of two normal distributions. For the mean of $H$

$$E(H) = \int x\left(pf_1(x) + (1-p)f_0(x) \right) dx = p\mu_1 + (1-p)\mu_0. $$

Similarly for the second moment of H \begin{align*} E(H^2)& = \int x^2 \left(pf_1(x) + (1-p)f_0(x) \right) dx\\ & = pE(H_1^2) + (1-p)E(H_0)^2\\ & = p(\sigma_1^2 + \mu_1^2) + (1-p)(\sigma_0^2 + \mu_0^2) \end{align*}

Finally, \begin{align*} Var(H) & = E(H^2) - [E(H)]^2\\ & = p(\sigma_1^2 + \mu_1^2) + (1-p)(\sigma_0^2 + \mu_0^2) - \left[p\mu_1 + (1-p)\mu_0 \right]^2\\ & = \left[p\sigma_1^2 + (1-p)\sigma_0^2\right] + [p\mu_1^2 + (1-p)\mu_0^2]- \left[p\mu_1 + (1-p)\mu_0 \right]^2\\ & = p\sigma^2_1+(1−p)\sigma^2_0+p(1−p)(\mu_1−\mu_0)^2 \end{align*}

Taking the square root of $Var(H)$, you get the standard deviation.

Greenparker
  • 14,131
  • 3
  • 36
  • 80
  • But why does $\sigma$ not depend on $\mu_0$ and $\mu_1$, and the difference between them? Surely $\sigma$ will be greater if the two means are further apart, no? – Kelvin Apr 02 '16 at 17:10
  • @Kelvin you are adding two normally distributed (independent) random variables, which is different from a mixture of Normally distributed random variables. – Greenparker Apr 02 '16 at 17:18
  • For example, if $\mu_0$ and $\mu_1$ were very different, but $\sigma_0$ and $\sigma_1$ were both zero, the combined standard deviation $\sigma$ would not be zero, would it? Something is not right here... – Kelvin Apr 02 '16 at 17:20
  • @Kelvin I may have misread your question, then. It seems like you are talking about a mixture of two normal distributions. – Greenparker Apr 02 '16 at 17:21
  • OK, thanks @Greenparker, then I guess I am looking for a weighted mixture rather than a weighted sum. Sorry if that was not clear, I wasn't aware there was a difference. But my formula for the weighted mean is still correct, right? – Kelvin Apr 02 '16 at 17:22
  • Perhaps I should substitute $\phi$ for $N$ in the question to clarify? – Kelvin Apr 02 '16 at 17:28
  • 1
    I have fixed the solution now. – Greenparker Apr 02 '16 at 17:35
  • @Greenparker - thank you. Am I right that your formula for variance simplifies to: $[p\sigma_1^2+(1-p)\sigma_0^2+p(1-p)(\mu_1-\mu_0)^2]$? – Kelvin Apr 02 '16 at 17:55
  • 1
    Yes, I believe you are correct. – Greenparker Apr 02 '16 at 18:01
  • OK, thanks. Perhaps you could add this for reference to the end of your answer, and I'll accept as correct. – Kelvin Apr 02 '16 at 18:06
  • I left it unsolved, since I wasn't sure if this was a homework problem or not. I think my answer is complete as it stands. – Greenparker Apr 02 '16 at 18:09
  • 1
    It's not homework, but maybe the simplified formula would help anyone else that reads this. – Kelvin Apr 02 '16 at 18:14
3

Let $Z$ denote a Bernoulli random variable with parameter $p$. Then, the random variable $H$ can be thought of as having conditional density $N(\mu_i, \sigma_i^2)$ according as $Z$ equals $i$, $i=0,1$, and thus unconditional density $$f_H(x) = pf_{H_1}(x) + (1-p)f_{H_0}(x).$$ The unconditional mean is thus the weighted sum of conditional means, viz. $$\mu_H = p\mu_1 + (1-p)\mu_0$$ while the unconditional (or total) variance is given by the total variance formula:

mean of the conditional variance plus the variance of the conditional mean

The first quantity is clearly $p\sigma_1^2 + (1-p)\sigma_0^2$. On the other hand, the conditional mean takes on value $\mu_1$ and $\mu_0$with probabilities $p$ and $1-p$ respectively and thus has variance \begin{align}p\mu_1^2 + (1-p)\mu_0^2 - \left(p\mu_1 + (1-p)\mu_0\right)^2 &= (p-p^2)\mu_1^2 + (1-p-(1-p)^2)\mu_0^2 -2p(1-p)\mu_0\mu_1\\ &= p(1-p)(\mu_1-\mu_0)^2 \end{align} Adding these two quantities together, we have that $$\sigma_H^2 = p\sigma_1^2 + (1-p)\sigma_0^2 + p(1-p)(\mu_1-\mu_0)^2$$ as in Greenparker's answer.

Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200