6

How to get the marginal pdf of $p(y)$? Do you just integrate out $p({\sigma}^{2})$?

Say, the following joint distribution for $y \in {{R}^{d}}$ and ${{\sigma }^{2}}\in {{R}^{d}}$ IG: means inverse Gamma $${{\sigma }^{2}}\sim IG(\alpha ,\beta )\propto {{({{\sigma }^{2}})}^{-(\alpha +1)}}{{e}^{-\beta /{{\sigma }^{2}}}}$$

$$y|{{\sigma }^{2}}\sim N(\mu ,{{\sigma }^{2}}\Sigma )$$

where $a\in R$, $b\in R$,$\mu \in {{R}^{d}}$,$\Sigma \in {{R}^{d\times d}}$ are known parameters.

I know that $$p(y|\mu,\Sigma)\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{1}{2}\sum\limits_{i=1}^{n}{{{\left( {{y}_{i}}-\mu \right)}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}({{y}_{i}}-\mu )} \right]$$ $$\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{1}{2}\sum\limits_{i=1}^{n}{{{\left( {{y}_{i}}-\bar{y} \right)}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}({{y}_{i}}-\bar{y})-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu )} \right]$$ $$\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{n-1}{2}tr\left( {{({{\sigma }^{2}}\Sigma )}^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu ) \right]$$

what I have done is using

$$p{({\sigma}^{2})}{\times}p(y|{{\sigma }^{2}})$$

which gives me, $$\propto {{\left( {{\sigma }^{2}} \right)}^{-(\alpha +1)}}{{e}^{-\beta /{{\sigma }^{2}}}}\times \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{n-1}{2}tr\left( {{({{\sigma }^{2}}\Sigma )}^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu ) \right]$$

This doesn't look like anything to me, am I even on the right track?! Anyways, if any experts knows, please point me out, thanks sooo much!

mpiktas
  • 33,140
  • 5
  • 82
  • 138
user1061210
  • 1,005
  • 3
  • 13
  • 19
  • The answer is no! – Xi'an Feb 20 '12 at 19:11
  • More seriously, your last line is completely correct. So you are on the right track: remember that only $\sigma^2$ varies in this expression and that you can factorise most of the term in the exponential in $\sigma^2$. Then you should see a standard distribution in $\sigma^2$. – Xi'an Feb 20 '12 at 21:04
  • So, I think I will integrate the joint density(my last line) w.r.t. $\sigma^{2}$, then I get p(y). Is that also what you are saying? But I am not too sure about integration matrix. I don't get what you mean by "factorise them". Something like complete the square? – user1061210 Feb 20 '12 at 21:27
  • @Xi'an, I am reading Bayesian Core, in chapter 3.2.1 page 54, you had a similar example, how did you arrive to the results. I am having trouble integrating out the $\sigma^{2}$. It is attached to the matrix $\Sigma$ in the exponent. – user1061210 Feb 21 '12 at 18:29

2 Answers2

4

What you get as your bottom line is of the form $$ (\sigma^2) ^{-\alpha-1-nd/2}\exp\{-A\sigma^{-2}\} $$ so the posterior distribution in $\sigma^{-2}$ is an inverse gamma distribution. (Note that $$ \text{tr}((\sigma^2\Sigma)^{-1}S)=\sigma^{-2}\text{tr}(\Sigma^{-1}S)\,.) $$ From this property, you can derive the normalising constant.

Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • Does this look right? $p(y)=\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\frac{\Gamma (a+n/2)}{{{\left( b+\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)+\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right)}^{a+n/2}}}$ – user1061210 Feb 22 '12 at 03:31
  • this is not any distribution I could recognize, more like a normalising constant with y bar in it. – user1061210 Feb 22 '12 at 03:32
  • 2
    Note that you can write the denominator as $\beta+(y-\mu)^T\Sigma^{-1}(y-\mu)$. note that the degrees of freedom is equal to double the power of this term minus the dimension of $y$ which gives us $v=2\alpha+n-d$. thus we can write posterior as $$p(y)\propto\left[1+\frac{1}{v}(y-\mu)\left(\frac{\beta}{v}\Sigma\right)^{-1}(y-\mu)\right]^{-\frac{v+d}{2}}$$ this is the kernel of a student t distribution. – probabilityislogic Feb 22 '12 at 05:09
  • Yes, this is a Student's $t$ density. – Xi'an Feb 22 '12 at 05:19
  • Apologies the above is a bit wrong. The exponent should be $\frac{v+nd}{2}$ and $v=2\alpha+n-nd$. we should also have a block diagonal matrix with $n$ blocks equal to $\Sigma$ instead of just $\Sigma$. also the term $(y-\mu)$ should be replaced by the $1\times nd$ row vector $\left[(y_1-\mu)^T,(y_2-\mu)^T,\dots,(y_n-\mu)^T\right]$. – probabilityislogic Feb 22 '12 at 05:44
  • That's an interesting mistake in that, indeed the vector $(y_1,\ldots,y_n)$ is Student but not the marginals. – Xi'an Feb 22 '12 at 06:18
  • @xian you made the same mistake i did. The exponent for $\sigma^2$ should be$-(\alpha+1+\frac{nd}{2})$ as $|\sigma^2\Sigma|=\sigma^{2d}|\Sigma|$. this means the degrees of freedom is simply $2\alpha$. – probabilityislogic Feb 22 '12 at 11:02
  • @probabilityislogic: right, I did! – Xi'an Feb 22 '12 at 13:07
  • @probabilityislogic and Xi'An, thanks for the help, the problem wasn't that bad, I think the trick was separating ${\sigma}^{2}$ from the matrix, integrating it, and then recognizing Student's t. Anyways, thanks, it was very helpful! – user1061210 Feb 22 '12 at 22:48
2

Note that the normalising constant for a IG variable is

$$\frac{b^a}{\Gamma(a)}$$

This is equal to the reciprical of the integral over $\sigma^{2}$ of the kernel of the pdf. hence we have

$$\int_0^{\infty}(\sigma^{2})^{-(a+1)}\exp\left(-\frac{b}{\sigma^2}\right)d\sigma^2=\frac{\Gamma(a)}{b^a}$$

Your integral is of this form for certain choice of $a$ and $b$.

probabilityislogic
  • 22,555
  • 4
  • 76
  • 97
  • @user1061210 - the inverse gamma distribution is undefined for $\sigma^2\leq 0$. The integral is then infinite for negative range. You cannot have negative range for this parameter, as it is a normal distribution variance. Similarly $\Sigma$ must be positive semi-definite (i.e. $x^T\Sigmax\geq 0$ for any vector $x$) for your distribution for $p(y|\sigma^2)$ to be valid – probabilityislogic Feb 22 '12 at 02:45
  • Does this look good? continued from my last line $$={{\left( {{\sigma }^{2}} \right)}^{-(\alpha +1)-n/2}}\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\exp \left\{ -\frac{1}{{{\sigma }^{2}}}\left[ b-\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right] \right\}$$ – user1061210 Feb 22 '12 at 03:28
  • integrate out ${sigma}^{2}$ $$=\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\int_{0}^{\infty }{{{\left( {{\sigma }^{2}} \right)}^{-(a+1+n/2)}}\exp \left\{ -\frac{1}{{{\sigma }^{2}}}\left[ b+\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)+\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right] \right\}}d{{\sigma }^{2}}$$ – user1061210 Feb 22 '12 at 03:30
  • $p(y)=\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\frac{\Gamma (a+n/2)}{{{\left( b+\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)+\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right)}^{a+n/2}}}$ – user1061210 Feb 22 '12 at 03:31
  • Yep you got it. Note that this is called a multivariate student distribution. It is not properly normalised though. So you should replace $=$ with $\propto$ – probabilityislogic Feb 22 '12 at 03:51
  • got it, isn't this suppose to follow some distribution? We have IG and normal as prior and likelihood, respectively. From experience, the posterior should fall into either one or a hybrid of both. – user1061210 Feb 22 '12 at 04:25