Explanation of Equation 5.3 from Gaussian Processes for Machine Learning

Question

I am currently reading through C. E. Rasmussen & C. K. I. Williams' Gaussian Processes for Machine Learning and was going through chapter 5. I could not exactly understand the derivation of equation 5.3. It would be helpful if any of you can explain how the equation was derived.

My attempt was as follows: \begin{equation*} P(w |X, \theta,y, H_i) = \frac{P(\theta | X, y, w, H_i).P(y | X, w, H_i).P(X|w, H_i).P(w|H_i).P(H_i)}{P(X, \theta, y, H_i)} \end{equation*} In the numerator, the first and the third terms are 1. I do not know how the authors derived the numerator.. Any help would be greatly appreciated. Here $w$, $\theta$, $y_i$, $X$ and $H_i$ are respectively the model parameters, the model hyperparameters, the labels, the data and the hypothesis set.

Answer 1

First, use a conditional Bayes' rule where we keep conditioning on $X$, $\theta$, and $\mathcal H_i$, and only swap $\mathbf w$ and $\mathbf y$: $$ p(\mathbf w \mid \mathbf y, X, \boldsymbol\theta, \mathcal H_i) = \frac{p(\mathbf y \mid \mathbf w, X, \boldsymbol\theta, \mathcal H_i) \; p(\mathbf w \mid X, \boldsymbol\theta, \mathcal H_i)}{p(\mathbf y \mid X, \boldsymbol\theta, \mathcal H_i)} .$$

Now, (5.3) will follow from this if we establish both of \begin{gather*} p(\mathbf y \mid \mathbf w, X, \boldsymbol\theta, \mathcal H_i) = p(\mathbf y \mid \mathbf w, X, \mathcal H_i) \tag{1} \\ p(\mathbf w \mid X, \boldsymbol\theta, \mathcal H_i) = p(\mathbf w \mid \boldsymbol\theta, \mathcal H_i) \tag{2} .\end{gather*}

But remember what these things are:

• $X$ is the matrix of training inputs
• $\mathbf y$ is the vector of training outputs
• $\mathbf w$ is the vector of parameters, e.g. the weights of a neural network
• $\boldsymbol\theta$ is the vector of hyperparameters, e.g. the regularization weight
• $\mathcal H_i$ is which of several discrete model classes we're considering using

So $\theta$ only determines how we choose $\mathbf w$ given the other stuff. If we're conditioning on $\mathbf w$ itself, then there's no added information by also conditioning on $\boldsymbol\theta$: $\mathbf y$ is independent of $\boldsymbol\theta$ given $\mathbf w$. Thus (1) holds.

(2) assumes that $\mathbf w$ doesn't depend on $X$ when we don't also have $\mathbf y$: that is, we know which points we're training on, but not the labels. This is actually kind of an assumption, one which I think was more reasonable to make in 2006 when this book was written than it would be today. For instance, if we're running (stochastic) gradient descent on a linear model, then we actually know that the difference between our final weights $\hat{\mathbf w}$ and the initial weights $\mathbf w_0$ will be in the span of the data $X$ – if $\mathbf w_0$ is small and the dimension of the data is much higher than the number of samples, this is actually very important. But, machine learning people have only really realized how important this kind of "implicit bias" is in the past few years, and it's quite tough to model, so assuming $\mathbf w$ is approximately independent of just $X$ given $\boldsymbol\theta$ and $\mathcal H_i$ is probably close enough.

---

Also: it is not true as you say in your question that $p(\boldsymbol\theta \mid X, \mathbf y, \mathbf w, \mathcal H_i) = 1$ or $p(X \mid \mathbf w, \mathcal H_i) = 1$.