handling metropolis hastings rejection during a Gibbs sweep

Question

Suppose I have a MCMC involving a 2 step Gibbs sampler. The first part uses metropolis hastings to find the next parameter value. If during one sweep, the result for the first part is a rejections, should the process immediately try the first step again until acceptance or simply move down to the second step in the sweep?

Answer 1

As you state, the

"first part uses Metropolis Hastings to find the next parameter value"

this means the exact simulation from the posterior is replaced with one (or several) Metropolis Hastings steps. Since one step ends up with one acceptance or one rejection, it is correct to take the outcome as is and move to the other conditional. Waiting for acceptance induces a bias in the simulation.

In fact, that this is embedded in a Gibbs sampler does not matter: if you target a distribution $f$ and call Metropolis-Hastings until you accept, you produce a biased example. Here is one example targeting the Normal $\text{N}(0,1)$ distribution and using a Normal $\text{N}(x_{t-1},2)$ proposal:

T=1e5
x=rep(rnorm(1),T)
for (t in 2:T){
  y=x[t-1]+rnorm(1,sd=2)
  while ( log(runif(1))>dnorm(y,log=TRUE)-dnorm(x[t-1],log=TRUE) )
    y=x[t-1]+rnorm(1,sd=2)
  x[t]=y}

Once you plot the outcome of this simulation, you can spot the bias: