5

Suppose I have a set of draws from a distribution with density $f$ and a set of draws from a distribution with density $g$, where $f$ and $g$ are unknown. Is there some way to use these use these draws to get a (perhaps smaller) set of draws whose pdf is proportional to the product of the pdfs of these two distributions?

I know it will be possible to get draws with approximately this distribution by kernel smoothing to estimate the two pdfs, but i was wondering if there was a way of getting draws with exactly the correct distribution.

Xi'an
  • 90,397
  • 9
  • 157
  • 575
SecretlyAnEconomist
  • 631
  • 3
  • 7
  • Not sure I understand your question. $f$ and $g$ are unknown but you want to sample from $fg$ *exactly*? – dsaxton Sep 26 '16 at 19:01
  • 2
    Because the product of PDFs almost always shows up in questions due to a mistaken idea that this would be the PDF of a sum or product of the corresponding random variables, could you explain what this product is intended to represent in your case? – whuber Sep 26 '16 at 19:27
  • The motivation is this: Suppose that we do not know the prior density but we have draws from it and we do not know the likelihood but we have draws from the distribution proportional to it, and we want to get draws from the posterior. – SecretlyAnEconomist Sep 27 '16 at 22:11
  • 3
    The likelihood is _not_, I repeat N.O.T., a density in the parameter. – Xi'an Sep 29 '16 at 15:37
  • 1
    Indeed, that's why I said 'distribution proportional to it'. – SecretlyAnEconomist Sep 29 '16 at 20:11
  • It is not simply a proportionality issue: a likelihood is not always integrable in the parameter and worse when it exists the integral does depend on the chosen parameterisation. In other words, the likelihood function does not obey the Jacobian transform when moving from one parameterisation to another. – Xi'an Sep 30 '16 at 10:32

1 Answers1

7

From a pure simulation point of view (Warning: link to my book), aiming at simulating from a product $f\times g$ of two (positive) functions such that $fg$ is integrable over the proper region suggests some specific methods:

  1. Sampling-importance-resampling: to apply when samples from both $f$ and $g$ can be produced and when the corresponding numerical values of the normalised versions of both $f$ and $g$ at those points can be numerically computed (resampling is not necessary for computing integrals related with $fg$).
  2. Accept-Reject: to apply when either $f$ or $g$ is proportional to a standard probability density, and the other term in the product is bounded.
  3. Slice sampling: to apply when either $f$ or $g$ is proportional to a standard probability density, easy to simulate, and the other function is such that the set $\{x; f(x)\ge a\}$ is easy to construct. Or when both $f$ and $g$ are such that $\{x; f(x)\ge a\}$ is easy to construct.
  4. Metropolis-Hastings algorithm: to apply when either $f$ or $g$ is proportional to a standard probability density, easy to simulate, and the other function can be (numerically) computed.
  5. Bridge sampling: to apply when both $f$ and $g$ are proportional to a standard probability density, when samples from both $f$ and $g$ can be produced in a reasonable time, and when the corresponding numerical values of both $f$ and $g$ at those points can be (numerically) computed.
  6. Nested sampling: to apply when either $f$ or $g$ is proportional to a standard probability density, easy to simulate, and the other function can be (numerically) computed.

Those different solutions do not necessarily take into account the fact that $f$ is a prior density and $g$ is a likelihood. Actually, it is not necessarily the best possible decomposition of the product $fg$, because $f$ can be too flat or even improper, while $g$ is not necessarily integrable in the parameter.

Note: A similar question was asked on Cross Validation last year but did not attract any traffic. I added links to scalable Monte Carlo solutions on that page.

Community
  • 1
Xi'an
  • 90,397
  • 9
  • 157
  • 575