1

I have an arbitrary number of independnet bivariate normal distributions with $\mu_i = [x_i,z_i]$ & $ \Sigma_i= \left(\begin{array}{cc} \sigma^2_{x_i} & \sigma^2_{x_i,z_i}\\\ \sigma^2_{x_i, z_i} & \sigma^2_{z_i} \end{array}\right) $

Where i is arbitrarily large

I want to take a linear combination of these bivariate normal distributions with weights $c = [c_1,...,c_i]$ where $\sum c_i = 1$ & $c_i >0$

Obviously, the linear combination of $\mu_{mixture} = [\sum c_ix_i,\sum c_iz_i]$

However, I am not sure about the linear combination of the variance-covariance matrix.

Does anyone know how I can calculate this pooled & weighted variance-covariance? Looking for the variance-covariance matrix for the mixture distribution.

Thanks so much!

CJR
  • 13
  • 3
  • Your question is ambiguous. Are you interested in properties of (a) the distribution of a linear combination of random variables having these distributions or (b) a *mixture* of these distributions? – whuber Jul 08 '21 at 19:33
  • 1
    @whuber Interested in the weighted mixture distribution of the i bivariate normal distributions – CJR Jul 08 '21 at 19:56
  • Please edit your post to state that clearly, because the answer that has been posted starts off with the other interpretation. – whuber Jul 08 '21 at 20:33
  • @whuber - thanks for catching that. I have edited the post – CJR Jul 08 '21 at 20:52
  • Your question has been asked and answered at https://stats.stackexchange.com/questions/16608 – whuber Jul 08 '21 at 22:02
  • @whuber thanks for adding this link. It was very helpful. I am still missing that last piece of how to apply that result to calculate the new mixture 2x2 variance-covariance matrix. The result you posted makes sense for $\sigma^2_{x}$ & $\sigma^2_{z}$, but how does this translate to the covariance $\sigma^2_{xz}$? – CJR Jul 08 '21 at 22:41
  • That result applies directly to *all* moments, without exception. The covariance can be computed from three such moments: the expectation of $XZ,$ the expectation of $X,$ and the expectation of $Z.$ Each of those three expectations is a linear combination of expectations of the mixture components. – whuber Jul 09 '21 at 13:43

2 Answers2

2

Let $X_i\stackrel{\text{indep}}\sim\mathcal N(\mu_i,\Omega_i)$ and let $S = \sum_{i=1}^n c_iX_i$. A linear combination of independent Gaussians is Gaussian so we just need the mean and covariance. By linearity we have $$ \text E[S] = \sum_i c_i\mu_i $$ and by independence we have $$ \text{Var}[S] = \sum_i \text{Var}[c_iX_i] = \sum_i c_i^2 \Omega_i $$ so $$ S\sim\mathcal N\left(\sum_i c_i\mu_i, \sum_i c_i^2\Omega_i\right). $$ This applies no matter what the $c_i$ are and for any dimension of $X_i$.

The above part assumed $n < \infty$. If we have a countably infinite number of $X_i$ then whether or not the series $\sum_{i=1}^\infty c_i X_i$ converges depends on how the $c_i$, $\mu_i$, and $\Omega_i$ evolve and we can use Kolmogrov's three series theorem to understand when this happens.

I interpreted this to mean you wanted the distribution of a linear combination of Gaussians. If you meant a finite mixture of Gaussians then we can work it out in the following way. Let $f_i$ be the density of $X_i$ and let $S \sim \sum_{i=1}^n c_i f_i$ be the mixture distribution. You didn't state that $c_i \geq 0$ but I'll assume that so that this is a valid density. Then we have $$ \text E[S] = \int s \sum_i c_i f_i(s)\,\text ds = \sum_i c_i \text E[X_i] = \sum_i c_i \mu_i $$ as before, except now this represents a convex combination of the $\mu_i$ where that was not guaranteed before. I'll use $\mu_\text{mix} = \sum_i c_i\mu_i$ as the mixture mean.

For the variances we need $$ \text E[SS^T] = \int ss^T \sum_i c_i f_i(s)\,\text ds = \sum_i c_i \text E[X_iX_i^T] $$ so all together $$ \text{Var}[S] = \text E[SS^T] - (\text E S)(\text ES)^T \\ =\sum_i c_i \text E[X_iX_i^T] - \mu_\text{mix}\mu_\text{mix}^T $$ which is more complicated than $\sum_i c_i^2\Omega_i$

jld
  • 18,405
  • 2
  • 52
  • 65
  • Thanks! This was my intuition & I appreciate the quick response. – CJR Jul 08 '21 at 19:24
  • @CJR for sure! glad this helped – jld Jul 08 '21 at 19:30
  • 1
    @CJR i just updated for the mixture case since it seems like maybe that's what you meant? – jld Jul 08 '21 at 20:16
  • Thanks for updating the post. I am still but unclear how the last line produces the variance-covariance matrix for the resulting mixture bivariate normal? If you could clarify that a bit more I'd really appreciate it. @jld – CJR Jul 08 '21 at 21:01
  • @CJR yeah sure. I'm using the result that for a random vector $S$ with mean vector $\mu$ the variance is $\text{Var}[S] = \text E[SS^T] - \mu\mu^T$. We know $\mu$ so the only remaining thing is the first term of $\text E[SS^T]$ and that's just the expected value of the 2x2 matrices $ss^T$ weighted by the mixture density – jld Jul 08 '21 at 21:45
  • Thanks for expanding. I am still not quite sure how to calculate this expected value given the information I have (the weights $c_i$, the mean vector $\mu_{mix}$, the mean vector for each bi-var normal $\mu_i$, and the var-cov matrix for each bi-var normal $\Sigma_i$) . With that information, how do I calculate this expected value? Thank you! @jld – CJR Jul 08 '21 at 21:59
  • @CJR looks like you’ve got it but I’m happy to add any extra details if there’s any extra points that you’d like clarification on! – jld Jul 09 '21 at 00:56
0

Thanks to @jld & @whuber for their answeres. Both were super helpful as I tried to solve this problem. With continued research, I found the post which I'll link below. It has the same info that jld & whuber shared, but it helped me solve it so I wanted to link it here

https://math.stackexchange.com/questions/195911/calculation-of-the-covariance-of-gaussian-mixtures

CJR
  • 13
  • 3