Need to understand a statement for Random Walk Metropolis algorithm's proposal distribution?

Question

I was told that the proposal distribution of Random Walk Metropolis needs to be symmetric. But today I was reading a book about Bayesian Analysis which contains the following statement:

"The proposal distribution does not have to be a rotationally symmetric bivariate normal. For example, it could be a bivariate normal with nonzero covariances, so that proposals are more likely to be made in some diagonal directions more than others. The proposal distribution could even be nonnormal. It is only for the present illustrative purposes that we assume a simple symmetric proposal distribution."

The above statements say that "the proposal distribution does not need to be rotationally symmetric bavariate normal". What does it mean?

Answer 1

Consider a "rotationally symmetric bivariate normal distribution". For your context, what this means is that the proposal distribution is bivariate normal with no correlation (see here maybe. Essentially, rotating the contours of this bivariate normal has not effect). That is, if $\Sigma$ is the matrix such that $$\Sigma = \left( \begin{array}{cc}1 & 0 \\ 0 & 1 \end{array} \right)\,, $$ then the proposal distribution is $$ q(\mathbf{x}, \mathbf{y}) \propto \exp \left\{-\dfrac{ \left(\mathbf{x} - \mathbf{y} \right)^T \Sigma^{-1} \left(\mathbf{x} - \mathbf{y} \right)}{2} \right\} = \exp \left\{-\dfrac{ \left(\mathbf{x} - \mathbf{y} \right)^T \left(\mathbf{x} - \mathbf{y} \right)}{2} \right\} = \exp \left\{-\dfrac{ \|\mathbf{x} - \mathbf{y} \|^2 }{2} \right\}\,. $$

Then, this proposal fits into the definition of a "rotationally symmetric bivariate normal distribution" as @Xi'an describes it. For some function $h$, $$ q(\mathbf{x}, \mathbf{y}) = h \left(\|\mathbf{x} - \mathbf{y} \|^2 \right)\,. $$

For a random walk MH algorithm, the proposal does not need to be rotationally symmetric as long as $q(\mathbf{x}, \mathbf{y}) = q(\mathbf{y}, \mathbf{x})$. If we choose a generation $\Sigma$ such as $$\Sigma = \left( \begin{array}{cc} \sigma^2_{11} & \sigma^2_{12} \\ \sigma^2_{12} & \sigma^2_{22} \end{array} \right)\,, $$ then, the proposal distribution $$ q(\mathbf{x}, \mathbf{y}) \propto \exp \left\{-\dfrac{ \left(\mathbf{x} - \mathbf{y} \right)^T \Sigma^{-1} \left(\mathbf{x} - \mathbf{y} \right)}{2} \right\} = \exp \left\{-\dfrac{ \left(\mathbf{y} - \mathbf{x} \right)^T \Sigma^{-1} \left(\mathbf{y} - \mathbf{x} \right)}{2} \right\} $$ is is still symmetric, but not rotationally symmetric.

Answer 2

The proposal for a general Metropolis-Hastings algorithm, \begin{align*}q\,:\, \mathcal X^2&\longmapsto \mathbb R_+\\ (x,y)&\longmapsto q(x,y)\end{align*} need not be symmetric provided it is included in the acceptance probability \begin{align*}\alpha\,:\, \mathcal X^2&\longmapsto \mathbb R_+\\ (x,y)&\longmapsto \min\left\{1,\dfrac{f(y)q(y,x)}{f(x)q(x,y)}\right\}\end{align*} When called a random walk Metropolis-Hastings algorithm, this algorithm is usually associated with a symmetric proposal $q(\cdot,\cdot)$ which simply means that $$q(x,y)=q(y,x)\stackrel{\text{def}}{=}g(x-y)$$with $g(x)=g(-x)$. Note that the rotationally symmetric constraint $$q(x,y)=q(y,x)\stackrel{\text{def}}{=}h(||x-y||^2)$$ is much stronger.