How to obtain the Gaussian conditional pdf, as discussed in section 2.3.1 in Pattern Recognition and Machine Learning, Bishop?

Question

From Statistical Inference of Casella, a conditional pdf is obtained through the joint pdf by the following equation,

(1) $$ p_a(x_a | x_b) = \frac {p(x_a, x_b)}{p_b(x_b)} $$

In section 2.1.3 of PRML, the author did not mention explicitly the equation, but I guess he using the same approach. The author obtain the conditional mean and covariant matrix by directly manipulating the quadratic form of the exponent of the joint pdf. I understand how he obtain the conditional mean and covariant matrix, but since he does not explain how to obtain the corresponding Gaussian pdf, I have no idea if that pdf is equal to (1).

This are the derived conditional mean and covariant matrix derived from the joint pdf, and other relevant vectors and matrices

The result conditional mean of $x_a$, given $x_b$ $$ \mu_{a|b} = \mu_a - \Lambda_{aa}^{-1} \Lambda_{ab} (x_b - \mu_b) \\ $$

The resulted conditional covariant matrix $$ \Sigma_{a|b} = \Lambda_{aa}^{-1} \\ $$

The random vector $$ x = \binom{x_a}{x_b} \\ $$

The joint mean $$ \mu = \binom{\mu_a}{\mu_b} \\ $$

The joint covariant matrix and its inverse $$ \Sigma = \begin{bmatrix} \Sigma_{aa} & \Sigma_{ab}\\ \Sigma_{ba} & \Sigma_{bb} \\ \end{bmatrix} \\ \Lambda = \Sigma^{-1} = \begin{bmatrix} \Lambda _{aa} & \Lambda _{ab}\\ \Lambda _{ba} & \Lambda _{bb} \\ \end{bmatrix} \\ $$

The quadratic form of the exponent of the joint pdf $$ (x - \mu)^T \Sigma (x - \mu) $$

(2) The general multivariate Gaussian pdf on a random vector $X \in R^D$ $$ N(x | \mu, \Sigma) = \frac{1}{(2\pi)^{0.5D}} \frac{1}{|\Sigma|^{0.5}} \exp\big(-0.5(x-\mu)^T \Sigma^{-1} (x-\mu) \big) $$

Answer 1

The author (C. Bishop) intentionally doesn't use the classic relation (1) because it'd then be required to calculate the denominator by marginalising the joint PDF. This integral is quite cumbersome in may cases in bayesian literature and there are approximate ways of getting it. However, in the Gaussian case, we have a way of calculating it in an indirect way (i.e. not by marginalising the joint). The author simply exploits the form of the joint PDF to come up directly with the conditional PDF. The idea can be outlined as follows:

The conditional PDF is a function of $x_a$, and since $x_b$ is given, any expression containing $x_b$ can be regarded as a constant. Therefore, the denominator has no effect on the form of $x_a$'s conditional PDF.

Ex: If let's say $p(x_a,x_b)=g(x_b)e^{-x_a h(x_b)}$ (assume $x_a$ is not a vector for this particular case), then $p(x_a|x_b)$ is in the form of an exponential random variable's PDF because all other terms including $p(x_b)$ in the denominator don't change the form of the conditional ($Ce^{-\lambda x})$.

That's why the author manipulates the joint PDF and says that it's actually in the following form: $$\begin{align}p(x_a,x_b)&=C\exp(-0.5(x-\mu)^T\Sigma^{-1}(x-\mu))\\&=C\exp(-0.5(x_a-b)^TS^{-1}(x_a-b)+g(x_b))\end{align}$$

where $g(.)$ is a function. And, this form is Gaussian for $x_a$. So, the conditional PDF is Gaussian. We don't worry exactly what would the denominator be or what exactly $g(x_b)$ brings us since it'd all have to satisfy the normalization property of PDFs anyway.

Noting that the form is Gaussian, it all boils down to finding what $S$ and $b$ is which are conditional covariance and the mean. Of course, there are a lot of steps between the first and the second line here, and if you have questions about specific steps, you can ask them as follow-ups.