Naive Monte Carlo, MCMC and their use in Bayesian Theory

Question

So let's suppose I have a random variable X which follows a PDF fX(x) which is known. I can use the Naive Monte Carlo method (with unfiltered random sampling) to obtain n samples of fX(x) and get empirical PDF and estimates of its parameters.

Now suppose we have three random variables (X, Y and Z).

First case: Suppose we know the relationship (function) M between X, Y and Z so that Z=M(X,Y). Suppose we know the marginals of X and Y and the covariance matrix between X and Y. In this case we can use Naive Monte Carlo method to get empirical PDF of Z and estimates of its parameters.

Second case: 2.A Suppose we only know the marginals of X and Y and how the PDF of Z is related to the PDFs of X and Y. In this case we can get the empirical PDF of Z and estimates of its parameters directly (although not knowing M). Although I'm not sure how the dependency between X and Y is taken into account...

2.B Suppose we only know the marginals of X and Y and how the PDF of Z is related to the PDFs of X and Y (up to a normalizing constant). This in the context of Bayesian Theory is equivalent of knowing the prior and the likelihood and the Bayes rule. In this case we can no longer use the Naive Monte Carlo method to get empirical PDF of Z and estimates of its parameters since we don't know M(X,Y). Here we need to resort to MCMC.

Is the above reasoning right?

Third case: Suppose we only know the marginals of X and Y, and some observations of Z. What methods can one apply to estimate not only the PDF of Z but also the function M assuming the marginals of X and Y are general and representative.

Answer 1

As a preliminary, let me point out that the issue of reconstituting the joint from the marginals is a constant theme on this forum, the answer being invariably that it is not possible without further assumptions.

"Suppose we know the marginals of $X$ and $Y$ and the covariance matrix between $X$ and $Y$."

This information is not enough for simulating $(X,Y)$, except in the bivariate Normal setting, and other parameterised cases [like exponential families] when the covariance matrix suffices to define the joint distribution. In general, the distribution of $Z$ is given by $$\mathbb P_Z(Z\in \mathcal A)=\mathbb P_{X,Y}(M(X,Y)\in A)=\mathbb P_{X,Y}((X,Y)\in M^{-1}(\mathcal A))=\int_{M^{-1}(\mathcal A)} p_{X,Y}(x,y)\text{d}(x,y)$$ and hence depends on the joint distribution of $(X,Y)$.

"...we only know the marginals of $X$ and $Y$ and how the PDF of $Z$ is related to the PDFs of $X$ and $Y$. In this case we can get the empirical PDF of $Z$"

This question is quite unclear or too vague, but in general wrong if $X$, $Y$, and $Z$ are dependent (for the same reason as above). If for instance it is known than $p_z=\Phi(p_X,p_Y)$ in the sense that $p_Z(z)$ can computed for all $z$'s then it is possible to build a Monte Carlo strategy based on this information. Further, the empirical pdf of $Z$ is unrelated to the true pdfs of $X$ and $Y$, but requires a sample of $Z$'s.

"This in the context of Bayesian Theory is equivalent of knowing the prior and the likelihood and the Bayes rule."

In Bayesian theory there are two random variables, the parameter $\theta$ and the experiment random variable $X$ (called the observation once realised as $x$). The likelihood function is a conditional density of the experiment random variable given the parameter random variable, not a marginal. And Bayes rule gives the conditional density of the (same) parameter random variable $\theta$ given the experiment random variable, not a marginal.

"...we can no longer use the Naive Monte Carlo method to get empirical PDF of $Z$"

This intuition is far from 100% correct as the knowledge of the posterior density up to a constant may be sufficient to run (a) analytical calculations (e.g., with conjugate priors) and (b) regular Monte Carlo simulations. MCMC is not a sure solution for all cases (as in the doubly intractable likelihood problem).

"Suppose we only know the marginals of $X$ and $Y$, and some observations of $Z$. What methods can one apply to estimate not only the PDF of $Z$ but also the function $M$ assuming the marginals of $X$ and $Y$ are general and representative."

This question is once again too vague. Observing $Z$ allows for the estimation of its PDF by non-parametric tools, but if $X$ and $Y$ are not observed, it is difficult to imagine estimating $M$ solely from the $Z$'s and the marginal densities.