General questions on rejection sampling

Question

I am new to Bayesian methods. I was going through a chapter on sampling. I have a few questions related to it. Please help me get these clarified.

1. As far as I understand, rejection sampling will not overestimate the density of the target distribution. This is inherent in it's acceptance probability. Do we always get exact density if we sample enough and if we have appropriate proposal distribution?

2. How many samples are usually rejected in rejection sampling? Is there a general formula to determine this?

3. How to determine the constant c for the proposal distribution such that c*Q(x) > P(x)? Is there any trick if we cannot visualize the distribution??(Especially in high dimensional cases?

4. This is related to previous question? Do people use rejection sampling alone while sampling from high dimensional distributions? Or is it used say in combination with Gibbs sampling? I would prefer the latter. But is the former better than the latter in any case?

Answer 1

Do we always get exact density if we sample enough and if we have appropriate proposal distribution?

Even if you only sample until you get a single acceptance you're still sampling from the correct distribution, as long as it's all done correctly.

1 How many samples are usually rejected in rejection sampling?

The average proportion rejected is $\frac{c-1}{c}$ (which is obvious from a description of the algorithm), but the conditional probability depends on the relative height of the majorizing function and the desired density.

How to determine the constant c for the proposal distribution such that c*Q(x) > P(x)?

$c$ can be any value as long as $c> \min P(x)/Q(x)$. You want to make $c$ small.

Do people use rejection sampling alone while sampling from high dimensional distributions? Or is it used say in combination with Gibbs sampling?

It certainly can be used with Gibbs - for example it can be used to when generating the full conditionals in some of the Gibbs steps.