10

The Dirichlet Pocess and Gaussian Process are often referred to as "distributions over functions" or "distributions over distributions". In that case, can I meaningfully talk about the density of a function under a GP? That is, do the Gaussian Process or Dirichlet Process have some notion of a probability density?

If it does not, how can we use Bayes' rule to go from prior to posterior, if the notion of the prior probability of a function is not well defined? Do things such as MAP or EAP estimates exist in the Bayesian Nonparametric world? Thanks a lot.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
snickerdoodles777
  • 655
  • 3
  • 13
  • 1
    Given that the (e.g.) Gaussian process realisation is only observed on a finite collection of points, the corresponding product of the Lebesgue measures is the dominating measure. Which means that for the observation of the random function $f$ at a finite collection of points, there exists a density. – Xi'an Feb 17 '19 at 19:55
  • The answer about densities is yes, and the appropriate mathematical formulation is called the Radon-Nikodym derivative. – whuber Jul 01 '19 at 21:36

1 Answers1

4

A "density" or "likelihood" relates to the Radon-Nikodym theorem in measure theory. As noted by @Xi'an, when you consider a finite set of so-called partial observations of a stochastic process, the likelihood corresponds to the usual notion of derivative w.r.t. the Lebesgue measure. For instance, the likelihood of a Gaussian process observed at a known finite set of indices is that of a Gaussian random vector with its mean an covariance deduced from that of the process, which can both take parameterized forms.

In the idealized case where an infinite number of observations is available from a stochastic process, the probability measure is on an infinite-dimensional space, for instance a space of continuous functions if the stochastic process has continuous paths. But nothing exists like a Lebesgue measure on an infinite-dimensional space, hence there is no straightforward definition of the likelihood.

For Gaussian processes there are some cases where we can define a likelihood by using the notion of equivalence of Gaussian measures. An important example is provided by Girsanov's theorem, which is widely used in financial math. This defines the likelihood of an Itô diffusion $Y_t$ as the derivative w.r.t the probability distribution of a standard Wiener process $B_t$ defined for $t \geq 0$. A neat math exposition is found in the book by Bernt Øksendal. The (upcoming) book by Särkkä and Solin provides a more intuitive presentation which will help practitioners. A brilliant math exposition on Analysis and Probability on Infinite-Dimensional Spaces by Nate Elderedge is available.

Note that the likelihood of a stochastic process that would be completely observed is sometimes called infill likelihood by statisticians.

Yves
  • 4,313
  • 1
  • 13
  • 34
  • Very helpful explanation! I think part of my confusion regarding topics like these in Bayesian Nonparametrics is due to my lack of familiarity with measure theory and functional analysis, so I'll be sure to check your references out. – snickerdoodles777 Feb 21 '19 at 15:10