17

Suppose we have two vectors of random variables, both are normal, i.e., $X \sim N(\mu_X, \Sigma_X)$ and $Y \sim N(\mu_Y, \Sigma_Y)$. We are interested in the distribution of their linear combination $Z = A X + B Y + C$, where $A$ and $B$ are matrices, $C$ is a vector. If $X$ and $Y$ are independent, $Z \sim N(A \mu_X + B \mu_Y + C, A \Sigma_X A^T + B \Sigma_Y B^T)$. The question is in the dependent case, assuming that we know the correlation of any pair $(X_i, Y_i)$. Thank you.

Best wishes, Ivan

Ivan
  • 458
  • 1
  • 4
  • 17

2 Answers2

9

In that case, you have to write (with hopefully clear notations) $$ \left(\begin{matrix}X\\Y \end{matrix}\right) \sim \mathcal{N}\left[ \left(\begin{matrix}\mu_X\\\mu_Y\end{matrix}\right), \Sigma_{X,Y} \right] $$ (edited: assuming joint normality of $(X,Y)$) Then $$ AX+BY=\left(\begin{matrix}A& B \end{matrix}\right) \left(\begin{matrix}X\\Y \end{matrix}\right) $$ and $$ AX+BY+C \sim \mathcal{N}\left[ \left(\begin{matrix}A& B \end{matrix}\right) \left(\begin{matrix}\mu_X\\\mu_Y\end{matrix}\right) + C, \left(\begin{matrix}A & B \end{matrix}\right)\Sigma_{X,Y} \left(\begin{matrix}A^T \\ B^T \end{matrix}\right)\right] $$ i.e. $$ AX+BY+C \sim \mathcal{N}\left[A\mu_X + B\mu_Y +C, A\Sigma_{XX}A^T+B\Sigma_{XY}^TA^T+A\Sigma_{XY}B^T+B\Sigma_{YY}B^T \right] $$

Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • 4
    In case it's overlooked, please note that the comment thread to another reply indicates (a) these covariance calculations are fine (understanding that they involve a natural but unstated block matrix notation) but (b) we cannot validly conclude that the linear combinations are *normally* distributed until we make an additional assumption; namely, that $X$ and $Y$ have a *joint* multivariate normal distribution. – whuber Feb 15 '12 at 16:41
  • 3
    Could you explain how you got from $B\Sigma_{XY}^TA^T+A\Sigma_{XY}B^T$ to $2A\Sigma_{XY}B^T$ in the last line? I would have thought that $$B\Sigma_{XY}^TA^T+A\Sigma_{XY}B^T = (A\Sigma_{XY}B^T)^T + A\Sigma_{XY}B^T$$ and the result does not simplify further. Here $\Sigma_{XY}$ is _not_ a symmetric matrix since its $(i,j)$-th element is $\text{cov}(X_i,Y_j)$ while its $(j,i)$-th element is $\text{cov}(X_j,Y_i)$, and there is no reason why these covariances must be equal. – Dilip Sarwate Feb 15 '12 at 20:10
  • 2
    @DilipSarwate: (+1) you are right, in the general case, there is no reason for these two terms to be equal. – Xi'an Feb 15 '12 at 20:19
3

Your question does not have a unique answer as currently posed unless you assume that $X$and$Y$ are jointly normally distributed with covariance top right block $\Sigma_{XY}$. i think you do mean this because you say you have each covariance between X and Y. In this case we can write $W=(X^T,Y^T)^T$ which is also multivariate normal. then $Z$ is given in terms of $W$ as:

$$Z=(A,B)W+C$$

Then you use your usual formula for linear combination. Note that the mean is unchanged but the covariance matrix has two extra terms added $A\Sigma_{XY}B^T+B\Sigma_{XY}^TA^T$

probabilityislogic
  • 22,555
  • 4
  • 76
  • 97
  • Thanks for pointing out this issue, in fact, I didn't even think about it, but it seems that the variables are indeed can be viewed, in my case, as jointly normally distributed, even if their components are correlated. – Ivan Feb 15 '12 at 12:45
  • 1
    I agree that the question cannot be solved as posed. It _can_ be solved _in a straightforward manner_ if one assumes, as @Xi'an's answer does, that $X$ and $Y$ are jointly normally distributed. It could be solvable, presumably with more difficulty, if the joint distribution was specified as something other than joint normal. But just knowing $\text{cov}(X_i,Y_j)$ for all $i, j$, **does not mean that $W = (X^T,Y^T)^T$ is multivariate normal**. **Any** two random variables with finite variances have a covariance. Covariance is not defined **only** for normal or jointly normal random variables. – Dilip Sarwate Feb 15 '12 at 13:19
  • In my case, X and Y are jointly normal, I will try to explain why, please correct me if I am wrong. Suppose there is a set of independent univariate normal r.v.'s. Each element of X and Y is an arbitrary linear combination of these univariate variables from the set. Therefore, since the initial variables are independent and only linear transformations are involved, the resulting vectors X, Y, and Z are all multivariate normal r.v.'s. It follows the definition of a multivariate normal r.v., where $a^T X$ should be a univariate normal r.v. for any vector $a$. Does it make sense? – Ivan Feb 15 '12 at 14:05
  • 1
    @Ivan Your explanation makes sense but the complaint is about the statement "Suppose we have two vectors of random variables, both are normal, i.e., $X\sim N(\mu_X,\Sigma_X)$ and $Y\sim N(\mu_Y,\Sigma_Y)$" which **does not mean** that $X$ and $Y$ are **jointly normal**. Nor does saying that "we know the correlation of any pair $(X_i,Y_i)$" mean that $X_i$ and $Y_i$ are **jointly normal** even though, as you correctly state, $X\sim N(\mu_X,\Sigma_X)$ implies that $X_i$ is normal (and similarly for $Y_i$.) Univariate normality **does not** imply joint normality. See reference below. – Dilip Sarwate Feb 15 '12 at 14:25
  • @Ivan See the discussion following [this question](http://stats.stackexchange.com/q/19948/6633) – Dilip Sarwate Feb 15 '12 at 14:27
  • @Ivan [wikipedia](http://en.wikipedia.org/wiki/Normally_distributed_and_uncorrelated_does_not_imply_independent) has some examples where two variables are normally distributed individually, but not jointly normally distributed. – probabilityislogic Feb 15 '12 at 15:26
  • Yes, I understand that my question was not completely correct. And yes, I am aware of the fact that two normal random variables are not necessary jointly normal, I always try to keep it in my mind. Fortunately, in this case, it works, they are multivariate normal. Thank you for the answers, guys! I really appreciate such a quick help. – Ivan Feb 15 '12 at 16:14