0

After asking a question about Gibbs sampling earlier, I have another one for you.

I have not been able to find laymen's background on this, the only referenced use I've found for this is in statistical papers (Roberts et al. 2003, Gelfand et al. 1995), to name a few.

In an apparent try to speed up the convergence of Gibbs sampling, we might try a re-paramatrization of the parameters of interest. This is in order to make them less dependent on one another, as I understand it. Because, the less dependent, the faster the convergence.

The two different parametrization have been termed 'ancillary' and 'sufficient', or alternatively 'non-centered' and 'centered'. These were introduced to me in a fairly specific example, and I would therefore very much appreciate it if someone can introduce these concepts more generally (i.e., how is an ancillary parametrization different from a sufficient one). In my case, the sufficient parametrization is faster - is the sufficient parametrization always ideal?

Community
  • 1
user1991
  • 229
  • 3
  • 7
  • The major entry on this topic is found in the papers of Xiao-Li Meng and his co-authors. – Xi'an Sep 07 '15 at 15:07
  • I will check that out, thanks! As for this topic - would it be possible to give an example of Gibbs sampling, for instance for a bivariate normal like in the other topic, and give examples of the ancillary and sufficient parameterizations? – user1991 Sep 07 '15 at 16:30
  • I suggest you read XL's papers, they are very well-written and full of examples. – Xi'an Sep 07 '15 at 18:53

1 Answers1

0

My answer is based on Yu and Meng (2011).

Suppose there is a parameter $\theta$ and we are interested in its posterior distribution $p(\theta|y)$ based on some data $y$. We are interested in augmentation's $\phi$ with joint posterior $p(\theta,\phi|y)$ such that $$ \int p(\theta,\phi|y) d\phi = p(\theta|y). $$ We say $\phi$ is sufficient for $\theta$ if $$ p(y|\theta,\phi) = p(y|\phi), $$ i.e. the data $y$ are conditionally independent of $\theta$ given $\phi$. We say $\phi$ is ancillary for $\theta$ if
$$ p(\phi|\theta) = p(\phi), $$ i.e. the prior for $\phi$ does not depend on $\theta$.

In Yu and Meng (2011), there is a particularly simple example with $$ Y \sim N(\theta,1+V) $$ where a sufficient augmentation works well if $V$ is large while an ancillary augmentation works well if $V$ is small. So, no the sufficient parameterization is not always ideal.

jaradniemi
  • 4,451
  • 13
  • 25