A main benefit of Gaussian Process Regression is, that we not only get a prediction, but also a variance that we might use as indication of the prediction confidence.
While bayesian linear regression also provides a variance, it is well published that it can be over confident especially out side the space that was covered by training data.
See e.g. Murphy Machine Learning: A Probabilistic Perspective p.533
[In bayesian linear regression the covariance] has at most D non-zero eigenvalues. Intuitively this reflects the fact that the model can only represent a limited number of functions. [...] It can result in overconfidence, since the model’s prior is so impoverished that its posterior will become too concentrated. So not only is the model wrong, it think it’s right!
My question is:
If we do a low rank approximation of the kernel matrix in gaussian process regression (such as with Nyström or Fourier) because of big data sets, does the model suffer from the same problem of over confidence?
More generally: How similar is a low rank approximated gaussian process to a bayesian linear regression model with comparable dimensionality?
