7

Optimizing the full likelihood function is sometimes time consuming and contains a lot of numerical issues and instabilities especially when matrix inversion is needed. If we have 3 input vectors $\boldsymbol{y}_{1},\boldsymbol{y}_{2},\boldsymbol{y}_{3}$ let $\mathbf{y}=[\boldsymbol{y}_{1},\boldsymbol{y}_{2},\boldsymbol{y}_{3}]$. Based on this, I want to model $\mathbf{y}$ $$ \mathbf{y}\sim N(0,K)$$ where K is a joint covariance matrix between all three input vectors. The Inputs are connected as shown in the figure below: $\boldsymbol{y}_{1}$ and $\boldsymbol{y}_{2}$ are independent while both $\boldsymbol{y}_{1}$ and $\boldsymbol{y}_{2}$ depend on $\boldsymbol{y}_{3}$. In other words the covariance matrix $K$ is expressed as follows $$K=\begin{pmatrix} k_{11} & 0 & k_{13}\\ 0& k_{22} & k_{23} \\ k_{13} & k_{23} & k_{33} \end{pmatrix}$$ Where $k$ is any postive semidefinite covariance function. Is there any specific way for me to exploit this independence and factorize the full likelihood function for example can I write $$f(\boldsymbol{y}_{1},\boldsymbol{y}_{2},\boldsymbol{y}_{3})=f(\boldsymbol{y}_{3}|\boldsymbol{y}_{1},\boldsymbol{y}_{2})*f(\boldsymbol{y}_{1})*f(\boldsymbol{y}_{2})$$ and optimize each part seprately? and what is the expression of $f(\boldsymbol{y}_{3}|\boldsymbol{y}_{1},\boldsymbol{y}_{2})$ given the covariance matrix above. Also is there a specific way I can use a composite likelihood approach such as a pairwise likelihood method.

Any good references on such normal likelihood factorizations is greatly appreciated

enter image description here

Wis
  • 2,044
  • 14
  • 31
  • It doesn't really make sense to optimize each part separately since all three of the factored likelihood functions share parameters. – Alex R. Aug 22 '16 at 19:38
  • @AlexR. Yes this is true , for this reason I am searching for a possible different method to factorize my likelihood and thus simplify the optimization problem. – Wis Aug 22 '16 at 19:45

1 Answers1

2

I don't know whether this winds up being a good thing to do, but you can express the distribution of $y_3$ conditional on $y_1,y_2$, which is a 1D Normal, using the standard Schur complement approach shown in https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Conditional_distributions .

Let $\mu_1,\mu_2,\mu_3$ be the means of $y_1,y_2,y_3$.

Denote $K_{12}$ as the diagonal matrix with entries $k_{11}$ and $k_{22}$, i.e., the covariance matrix of $[y_1,y_2]^T$. Its inverse is obtained by inverting the diagonal elements.

$y_3$ conditional on $[y_1,y_2]^{T} = [x_1,x_2]^{T}$ is Normal and has $$mean = \mu_3 + [k_{13},k_{23}] K_{12}^{-1} [x_1 - \mu1,x_2 - \mu_2]^{T}$$ and $$variance = k_{33} - [k_{13},k_{23}] K_{12}^{-1} [k_{13},k_{23}]^T$$

Mark L. Stone
  • 12,546
  • 1
  • 31
  • 51
  • You can use this to form a factorized likelihood for your scenario. – Mark L. Stone Aug 22 '16 at 21:54
  • Based on your derivation, I cannot optimize each part of my likelihood separately since the three factors share parameters. Is there a well known approximation for such cases ? – Wis Aug 23 '16 at 02:12
  • Not that I know of. But I don't know everything. If you want to get a feel for what's going on with multivariate normal likelihood and constraints, look at exercise 7.4 on pp. 303-304 of http://stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf . – Mark L. Stone Aug 23 '16 at 02:26