Use Expectation-Maximization algorithm for obtaining maximal likelihood estimator

Question

For $X = {(Z_{i}, Y_{i}) : i = 1, ... ,n}$, consider the model:

$Y_{i} = \beta_{1} + \beta_{2}Z_{i} + \epsilon_{i}$ where $\epsilon_{1}, ... ,\epsilon_{n}$ are i.i.d $N(0,\sigma^2)$, $Z_{i},...Z_{i}$ are i.i.d $N(\mu_{1},\sigma_{1}^2)$ and independent of $\epsilon_{1}, ... ,\epsilon_{n}$.

For $1\le i\le m$, we observe both $Z_i$ and $Y_i$, and for $m + 1 \le i\le n$, we observe only $Y_i$.

I try to complete the E- and M-steps of the EM algorithm for estimating $\theta=(\mu_{1},\beta_{1},\sigma_{1}^2,\sigma^2,\beta_{2})$

E-step: $Q(\theta; \theta_{old}) = E [\ell(θ; X , Y) | X , \theta_{old}]$

M-step: $\theta_{new} := \max\limits_{\theta} Q(\theta; \theta_{old})$.

still stuggle how to apply these step to solve the problem

Answer 1

You could express the complete log-likelihood as $$\ell(\theta) = -n\log\sigma_1-\frac{1}{2}\sum_{i=1}^n \sigma_1^{-2}(z_i-\mu_1)^2 -n\log\sigma^2-\frac{1}{2}\sum_{i=1}^n \sigma^{-2}(y_i-\beta_1-\beta_2 z_i)^2$$ to derive $$\mathbb E_{\theta^\text{old}}[\ell(\theta)|X^\text{obs}]$$as \begin{align}&-n\log\sigma_1-\frac{1}{2}\sum_{i=1}^m \sigma_1^{-2}(z_i-\mu_1)^2\\&-\frac{\sigma_1^{-2}}{2}\sum_{i=m+1}^n \mathbb E_{\theta^\text{old}}[(Z_i-\mu_1)^2|y_i]\\ &-n\log\sigma^2-\frac{1}{2}\sum_{i=1}^m \sigma^{-2}(y_i-\beta_1-\beta_2 z_i)^2\\ &-\frac{\sigma^{-2}}{2}\sum_{i=m+1}^n \mathbb E_{\theta^\text{old}}[(y_i-\beta_1-\beta_2 Z_i)^2|y_i] \end{align} and then use the conditional distribution of $Z_i$ given $Y_i=y_i$ and $\theta^\text{old}$: $$f(z_i|y_i,\theta)\propto\exp\{-\sigma^{-2}(y_i-\beta_1-\beta_2z_i)^2/2\}\times\exp\{-\sigma_1^{-2}(z_i-\mu_1)^2/2\}$$ which means $$Z_i|Y_i,\theta\sim\mathcal N\left(\{\sigma^-2\beta_2^2+\sigma_1^{-1}\}^{-1}\{\sigma^-2\beta_2[y_i-\beta_1]\},\{\sigma^-2\beta_2^2+\sigma_1^{-1}\}^{-1}\right)$$ from which $$\mathbb E_{\theta^\text{old}}[(Z_i-\mu_1)^2|y_i]\qquad\text{and}\qquad\mathbb E_{\theta^\text{old}}[(y_i-\beta_1-\beta_2 Z_i)^2|y_i]$$ are easily derived.