8

Suppose I have two independent normal variables $X$ and $Y$ with known mean and variance. Defining $Z = X+Y$, what is the most straightforward way to compute $\mathbb{E}\left[X|Z\right]$?

I am writing \begin{align} \mathbb{E}\left[X|Z\right] & = \int_x x f_{X}(x|Z=z) \mathrm{d}x \\ & = \int_x x \frac{f_{XZ}(x,z)}{f_Z(z)} \mathrm{d}x \\ & = \int_x x \frac{f_{Y}(z-x)f_X(x)}{f_Z(z)} \mathrm{d}x, \\ \end{align}

but I am not sure if the best (and only?) way is to compute this expression using our knowledge of the pdf's. Thanks.

David Melkuev
  • 188
  • 1
  • 1
  • 7
  • they wouldn't be independent & *identically* distributed in this case. – qoheleth Aug 25 '14 at 05:39
  • 1
    See section 7.4 here: http://arxiv.org/pdf/1304.7480v2.pdf Hope it was helpful. – sefi Aug 25 '14 at 07:33
  • 1
    My answer at http://stats.stackexchange.com/a/71303 works out the conditional mean and the conditional variance, giving explicit formulas for both (skip to the end at the "Conclusions" section). The conditional mean is the "regression curve." It is given by the standard least-squares regression formulas. Apply this result to the binormal distribution of $(Z,X)$, whose means and covariance matrix are easily computed. (Although it is a lengthy post, the derivation in it is notable for not employing any integrals nor even any explicit representation of the PDF.) – whuber Aug 25 '14 at 13:27

2 Answers2

3

As @StéphaneLaurent points out, $(X,Z)$ have a bivariate normal distribution and $E[X\mid Z] = aZ+b$. But even more can be said in this case because it is known that $$a = \frac{\operatorname{cov}(X,Z)}{\sigma_Z^2}, \quad b = \mu_X - a\mu_Z = \mu_X - \frac{\operatorname{cov}(X,Z)}{\sigma_Z^2}\mu_Z,$$ and we can use the independence of $X$ and $Y$ (which implies $\operatorname{cov}(X,Y) = 0$) to deduce that $$\begin{align} \operatorname{cov}(X,Z) &= \operatorname{cov}(X,X+Y)\\ &= \operatorname{cov}(X,X) + \operatorname{cov}(X,Y)\\ &= \sigma_X^2\\ \sigma_Z^2 &= \operatorname{var}(X+Y)\\ &= \operatorname{var}(X)+\operatorname{var}(Y) + 2\operatorname{cov}(X, Y)\\ &= \sigma_X^2+\sigma_Y^2\\ \mu_Z &= \mu_X+\mu_Y. \end{align}$$ Note that the method used above can also be applied in the more general case when $X$ and $Y$ are correlated jointly normal random variables instead of independent normal random variables.

Continuing with the calculations, we see that $$E[X\mid Z] = \frac{\sigma_X^2}{\sigma_X^2+\sigma_Y^2}(Z-\mu_Z) + \mu_x \tag{1}$$ which I find comforting because we can interchange the roles of $X$ and $Y$ to immediately write down $$E[Y\mid Z] = \frac{\sigma_Y^2}{\sigma_X^2+\sigma_Y^2}(Z-\mu_Z) + \mu_Y\tag{2}$$ and the sum of $(1)$ and $(2)$ gives $E[X\mid Z] + E[Y\mid Z] = Z$ as noted in Stéphane Laurent's answer.

Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200
  • There's something strange when comparing our two approaches: mine yields a result that doesn't involve any variance. – Stéphane Laurent Aug 25 '14 at 12:25
  • @StéphaneLaurent Your approach results in the equations $$\begin{align}\mu_X&=a(\mu_X+\mu_Y)+b\\\mu_Y&=\alpha(\mu_X+\mu_Y)+\beta\\a+\alpha &=1\\b+\beta &=0\end{align}$$ but do these four linear equations give $a$ and $b$ in terms of the means alone? or do they involve $\alpha$ and/or $\beta$ as well? – Dilip Sarwate Aug 25 '14 at 12:38
  • Thank you both for your answers. @DilipSarwate I recognize your expression for $a$ in terms of covariance/variance as the coefficient in an OLS linear regression. Is that what it is based on or is there a more general idea that both are linked to? – David Melkuev Aug 25 '14 at 23:11
  • @DilipSarwate After replacing $\alpha$ with $1-a$ and $\beta$ with $-b$, there are two equations and the solutions $a$ and $b$ depends on $\mu_X$ and $\mu_Y$ only. I think my approach is wrong: the system is not invertible. – Stéphane Laurent Aug 26 '14 at 10:31
  • @StéphaneLaurent Carrying out the replacements you suggest gives $$\begin{align}\mu_X&=a(\mu_X+\mu_Y)+b\\\mu_Y&=(1-a)(\mu_X+\mu_Y)-b\\\end{align}$$ whose sum is $$\mu_X+\mu_Y = \mu_X+\mu_Y$$ and so, yes, the system is not invertible. – Dilip Sarwate Aug 26 '14 at 11:12
  • 1
    @DavidMelkuev Yes, the expressions are based on the essential idea behind OLS: we seek coefficients $a$ and $b$ that minimize the mean-square error $E[(X - (aZ+b))^2]$. This is standard stuff usually covered in undergraduate probability textbooks in the chapters on expectation which usually also point out that for the special case when $X$ and $Z$ have a bivariate normal distribution, the **general** _minimum-mean-square-error_ (MMSE) estimator $E[X\mid Z]$ coincides with the **linear** _minimum-mean-square error_ (LMMSE) estimator $aZ+b$. – Dilip Sarwate Aug 26 '14 at 11:20
1

Each of the pairs $(X,Z)$ and $(Y,Z)$ has a bivariate normal distribution. Then we know that $$E(X\mid Z) = a Z+b \quad\textrm{ and } \quad E(Y \mid Z)=\alpha Z + \beta.$$ Taking the expectation yields $E(X)=aE(Z)+b$ and $E(Y)=\alpha E(Z) + \beta$. But we also have $E(X\mid Z) + E(Y \mid Z) = Z$, therefore $a+\alpha=1$ and $b+\beta=0$. Finally we have to solve a linear system of two equations and two unknown variables.

Stéphane Laurent
  • 17,425
  • 5
  • 59
  • 101