2

Suppose $y$ is a Gaussian process given by $y \sim f + \epsilon$, where $\epsilon$ is a Gaussian noise model with zero mean, and $f$ is a deterministic yet unknown mean function (or a Gaussian process independent of $\epsilon$). Therefore, one would find that $\mathbb{E}[y] = \mathbb{E}[f]$ since $\mathbb{E}[\epsilon] = 0$. But my question is: does $\mathbb{E}[{{ \bf y}_b \vert { \bf y}_a}] = \mathbb{E}[{{ \bf f}_b \vert { \bf y}_a}]$? Namely, are the conditional means of $\bf f_b$ and $\bf y_b$ equivalent?

The reason I ask is because we know $\text{Var}[y] \neq \text{Var}[f]$ and $\text{Var}[{{ \bf y}_b \vert { \bf y}_a}] \neq \text{Var}[{{ \bf f}_b \vert { \bf y}_a}]$. Additionally, the covariance matrix of $y$ is given by: $$\Sigma_y(x_1,x_2) = k(x_1,x_2) + \sigma^2 (x_1) \delta(x_1 - x_2),$$ while the covariance matrix of $f$ is given by (c.f. the lines below equations 5.8 or below 2.30): $$\Sigma_f(x_1,x_2) = k(x_1,x_2),$$ i.e. $y$ has an additional (possibly) heteroscedastic noise model, $\sigma$, added along the diagonal of covariance matrix to represent the variance of the noise, $\epsilon$. But after observing a set of measurements, $\boldsymbol y_a$ at inputs $\boldsymbol x_a$, the conditional mean of $\boldsymbol y_b$ is given by:

$$\mathbb{E}[{ \bf y}_b \vert { \bf y}_a] = \boldsymbol\mu_b+\Sigma_y(\boldsymbol x_b,\boldsymbol x_a){\Sigma_y(\boldsymbol x_a,\boldsymbol x_a)}^{-1}({\boldsymbol y_a}-\boldsymbol\mu_a) $$

but the conditional mean of $\boldsymbol f_b$ is given by:

$$\mathbb{E}[{ \bf f}_b \vert { \bf y}_a] = \boldsymbol\mu_b+\Sigma_f(\boldsymbol x_b,\boldsymbol x_a){\Sigma_f(\boldsymbol x_a,\boldsymbol x_a)}^{-1}({\boldsymbol y_a}-\boldsymbol\mu_a) $$

Therefore since $\Sigma_y(x_1,x_2)$ does not necessarily equal $\Sigma_f(x_1,x_2)$, is it accurate to state that $\mathbb{E}[{{ \bf y}_b \vert { \bf y}_a}]$ does not necessarily equal $\mathbb{E}[{{ \bf f}_b \vert { \bf y}_a}]$?

Mathews24
  • 417
  • 4
  • 20
  • Unless $f$ is also Gaussian (or a deterministic quantity (not necessarily a constant)), $y$ is not a Gaussian process at all. – Dilip Sarwate Jul 31 '19 at 19:17
  • @DilipSarwate I have updated the question accordingly. To gain better insight, why is it not appropriate to call $y({\bf x}) \sim f({\bf x}) + \epsilon({\bf x})$ a Gaussian process? Where $f({\bf x})$ could be some deterministic function and $\epsilon({\bf x})$ a Gaussian variable with zero mean and variance that is possibly dependent upon $\bf x$? – Mathews24 Jul 31 '19 at 20:37
  • A Gaussian process is one in which _every_ finite subset of the random variables comprising the process has a _jointly_ Gaussian distribution (which implies that all the random variables are (marginally) Gaussian too). If $f$ and $\epsilon$ are Gaussian processes, $y=f+\epsilon$ is _not_ a Gaussian process unless $f$ and $\epsilon$ are _jointly_ Gaussian. If $f$ is a deterministic function and $\epsilon$ a zero-mean Gaussian noise process, then $y=f+\epsilon$ is a nonstationary Gaussian process etc. – Dilip Sarwate Aug 01 '19 at 02:54
  • Correct, but [joint Gaussianity is guaranteed if the random processes, $f$ and $\epsilon$, are each from a Gaussian process](https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables), correct? The primary case of interest here is the latter one you identified, i.e. a nonstationary Gaussian process where $f$ is a deterministic function and $\epsilon$ a zero-mean Gaussian process, but I would also be interested in knowing how the answer differs if $f$ is a Gaussian process itself. – Mathews24 Aug 01 '19 at 04:19
  • No, if all you are told is that $f$ and $\epsilon$ are Gaussian, this _does not_ automatically mean that $f$ and $\epsilon$ are _jointly_ Gaussian. There are numerous counterexamples to your notion [here](https://stats.stackexchange.com/a/30205/6633). The Wikipedia page you refer to talks of the case when $f$ and $\epsilon$ are given to be jointly Gaussian (and hence marginally Gaussian too) including as an important special case when they are _independent_, a word that is nowhere to be found in your question. – Dilip Sarwate Aug 01 '19 at 14:22
  • Thank you for the clarification. The above is now updated and I would be interested in knowing how the answer varies for $f$ being a 1) deterministic function or 2) Gaussian process independent of $\epsilon$. – Mathews24 Aug 01 '19 at 14:47

0 Answers0