Identifying a conditional posterior for $\mu$ and $\sigma^2$

Question

My Question

Consider the model $X_1, \dots, X_n \overset{\mathrm{iid}}{\sim} \cal N(\mu, \sigma^2), (\mu, \sigma^2) \in (-\infty, \infty) \times (0, \infty)$ with a product prior pdf $\xi(\mu, \sigma^2) = \xi_1(\mu)\xi_2(\sigma^2)$ where $\xi_1(\mu) = \cal N(a, b^2)$ and $\xi_2(\sigma^2) = \chi^{-2}(r,s)$ for some constants $a \in , b > 0, r > 0$ and $s > 0$. Let $\xi(\mu, \sigma^2| x)$ denote the posterior pdf given observations $x = (x_1, \dots, x_n)$.

Identify the conditional posterior pdf $\xi_1(\mu, | x, \sigma^2)$ of $\mu$ for a given $\sigma^2$.

Identify the conditional posterior pdf $\xi_2(\sigma^2 | x, \mu)$ of $\sigma^2$ for a given $\mu$.

My Thoughts

First, a helpful fact from my previous searching: $$\xi_1(\mu | x, \sigma^2) \propto \xi(\mu, \sigma^2 | x)$$ Given this, I fixed $\mu, \sigma^2$ in turn and saw what kind of pdf I got. For $\mu$, fixing $\sigma^2$, I got another Normal pdf, so my belief is that $\xi_1(\mu, | x, \sigma^2)$ should take the form of a Normal pdf.

Fixing $\mu$, I saw that $\sigma^2$ was in the denominator of the exponential term and in the denominator of the ${1\over \sigma \sqrt{2\pi}}$ term outside of the exponential term, and so I thought of the inverse-chi-square family or the inverse gamma family.

My Question

Given that I have reasonable beliefs about what form my answers should take, how would I go about finding explicit formulae for the updated parameters? I want to be able to explicitly write down a formula for $\xi_1(\mu, | x, \sigma^2)$ and right now, given that I identified the form they should take through (liberal) use of "$\propto$", I'm not sure what to do.

Answer 1

You have random variables $X_1,\dots,X_n$ that are, given that $M=\mu$ and $\Sigma^2=\sigma^2$, conditionally independent and identically distributed as $\mathrm{N}(\mu,\sigma^2)$.

You believe that a priori $M$ and $\Sigma^2$ are independent with $M\sim\mathrm{N}(a,b^2)$ and $\Sigma\sim\mathrm{IG}(r,t)$.

(please, check the definition of the Inverse-Gamma distribution).

Define $X=(X_1,\dots,X_n)$ and $x=(x_1,\dots,x_n)$. Also, define $\bar{x}=\sum_{i=1}^n x_i/n$ and $s^2=\sum_{i=1}^n (x_i-\bar{x})^2/n$.

The likelihood is $$ f_{X\mid M,\Sigma^2}(x\mid\mu,\sigma^2) = \prod_{i=1}^n f_{X_i\mid M,\Sigma^2}(x_i\mid\mu,\sigma^2) $$ $$ = (2\pi)^{-n/2} \sigma^{-n} \exp\left(-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i-\mu)^2\right) \, . $$ But $$ \sum_{i=1}^n (x_i-\mu)^2 = \sum_{i=1}^n (x_i-\bar{x}+\bar{x}-\mu)^2 $$ $$ = \sum_{i=1}^n (x_i-\bar{x})^2 +2 (\bar{x}-\mu)\sum_{i=1}^n (x_i-\bar{x}) + \sum_{i=1}^n (\bar{x}-\mu)^2 \, . \quad (*) $$ The middle term in $(*)$ is zero (check it out), giving us the useful decomposition $$ \sum_{i=1}^n (x_i-\mu)^2 = n(\mu-\bar{x})^2 + ns^2 \, , $$ which allows us to rewrite the likelihood in terms of the sufficient statistics $\bar{x}$ and $s^2$ as $$ f_{X\mid M,\Sigma^2}(x\mid\mu,\sigma^2) = (2\pi)^{-n/2} \sigma^{-n} \exp\left(-\frac{n}{2\sigma^2}\left((\mu-\bar{x})^2 + s^2\right)\right) \, . $$ Hence, using the most beautiful theorem ever, the posterior density is $$ f_{M,\Sigma^2\mid X}(\mu,\sigma^2\mid x) \propto f_{X\mid M,\Sigma^2}(x\mid\mu,\sigma^2) f_M(\mu) f_{\Sigma^2}(\sigma^2) $$ $$ \propto \sigma^{-n} \exp\left(-\frac{n}{2\sigma^2}\left((\mu-\bar{x})^2 + s^2\right)\right) $$ $$ \times \exp\left(-\frac{1}{2b^2}(\mu-a)^2\right) (\sigma^2)^{-r-1} \exp\left(-\frac{t}{\sigma^2}\right) \, , $$ in which the $\propto$ symbol means proportionality up to factors that do not depend on $\mu$ and $\sigma^2$.

To compute the full conditionals, note that, by the product rule for densities, we have $$ f_{M, \Sigma^2\mid X}(\mu,\sigma^2\mid x) = f_{M\mid\Sigma^2,X}(\mu\mid\sigma^2,x) f_{\Sigma^2\mid X}(\sigma^2\mid x) \, , $$ and hence $$ f_{M\mid\Sigma^2,X}(\mu\mid\sigma^2,x) \propto f_{M,\Sigma^2\mid X}(\mu,\sigma^2\mid x) \, , $$ in which the $\propto$ symbol means proportionality up to factors that do not depend on $\mu$.

Therefore, $$ f_{M\mid\Sigma^2,X}(\mu\mid\sigma^2,x) \propto \exp\left(-\frac{1}{2}\left(\frac{n}{\sigma^2}(\mu-\bar{x})^2 +\frac{1}{b^2}(\mu-a)^2\right)\right) \, . $$

Completing the square, we have $$ \frac{n}{\sigma^2}(\mu-\bar{x})^2 +\frac{1}{b^2}(\mu-a)^2 \propto \left(\frac{nb^2+\sigma^2}{\sigma^2 b^2}\;\;\right) \mu^2 - 2\mu\left(\frac{nb^2\bar{x}+\sigma^2a}{\sigma^2b^2}\;\;\right) $$ $$ \propto \left(\frac{nb^2+\sigma^2}{\sigma^2 b^2}\;\;\right)\left(\mu - \frac{nb^2\bar{x}+\sigma^2a}{nb^2+\sigma^2}\;\;\right)^2 \, , $$ which gives you, by inspection, that $$ M\mid\Sigma^2=\sigma^2,X=x \sim \mathrm{N}\left(\frac{nb^2\bar{x}+\sigma^2a}{nb^2+\sigma^2}\quad, \frac{\sigma^2 b^2}{nb^2+\sigma^2}\;\;\right) \, . $$

Now we take a deep breath and do it mutatis mutandis for the other conditional. $$ f_{\Sigma^2\mid M,X}(\sigma^2\mid \mu,x) \propto (\sigma^2)^{-n/2-r-1} \exp\left( -\frac{1}{\sigma^2}\left(\frac{n}{2}\left((\mu-\bar{x})^2 + s^2\right) + t \right)\right) \, , $$ which says that $$ \Sigma^2\mid M=\mu,X=x \sim \mathrm{IG}\left( \frac{n}{2}+r, \frac{n}{2}\left((\mu-\bar{x})^2 + s^2\right) + t\right) \, . $$

Answer 2

If your hyper parameters are fixed (without any random distribution), you can use the MCMC method to simulate and update your parameters. MCMC does not need exact distribution function of the posteriors (with normalization constant). It only needs some candidate distributions for your parameters that you can define based on your opinion about each posteriors.