4

For a simple linear model such as $y_i = X_i\beta + \epsilon $ for i = 1,2...

and we assume that $E[\epsilon_i | X] = 0$ and that $E[\epsilon_i^2|X] = \sigma_i^2 I_T$ for i=1,2...,n and that $E[\epsilon_i,\epsilon_j'] = \sigma_{ij} I_T$ for $i \neq j$

Can I use MLE to estimate this model? Will the correlation between errors cause problems with the standard errors of the MLE estimates of $\beta$?

MHall
  • 275
  • 1
  • 8
  • I think your question a little bit confusing, for a simple linear regression $E[\epsilon_i,\epsilon_j'] = 0$ for $i \neq j$, then MLE and OLS are the same. If $E[\epsilon_i,\epsilon_j'] \neq 0$ you may consider REML. – Deep North Nov 24 '17 at 04:21
  • The OP is setting up the problem with a known correlation structure for the error terms. But if the variances and covariances are to be estimated that is a lot of terms to estimate and won't even be estimable if the sample size not sufficiently large. Certainly estimating so many error terms will lead to a larger standard error for the estimate of $\beta$. – Michael R. Chernick Nov 24 '17 at 15:23
  • Do you actually assume a panel-data like structure where observations vary along two dimensions, so that you need $n$ and $T$, or is $n=T$? In the first case, it does not seem to be appropriate to call it the "simple linear model". In the second, we have a standard GLS problem, it seems. – Christoph Hanck Nov 24 '17 at 15:40

2 Answers2

6

The error (correlation) structure you assume simply needs to be incorporated in the likelihood function that you are trying to maximize. Remember that the logarithmic Gaussian likelihood function is generally written $$ \ell(\mathbf{\beta},\mathbf{\Sigma}) = -\frac{n}{2} \log ( 2 \pi) - \frac{1}{2} \log \left| \mathbf{\Sigma} \right| -\frac{1}{2}\left( \mathbf{y} - \mathbf{X} \mathbf{\beta} \right)^{\mathsf{T}} \mathbf{\Sigma}^{-1} \left( \mathbf{y} - \mathbf{X} \mathbf{\beta} \right) $$ where in the homoskedastic model we assume spherical errors, i.e. $\mathbf{\Sigma} = \sigma^{2} \mathbf{I}_{n}$, such that the variance-covariance matrix of the error has a constant $\sigma^{2}$ on the diagonal and zeros off-diagonally. But nothing prevents us from assuming (and estimating) a more complex variance-covariance matrix, including heteroskedasticity and/or serial correlation.

Durden
  • 860
  • 10
  • 12
  • 1
    I wonder if there is a typo in your logarithmic Gaussian likelihood function. Should it be this instead? $$ \ell(\mathbf{\beta},\mathbf{\Sigma}) = -\frac{n}{2} \log ( 2 \pi) - \frac{\textbf 1}{2} \ln \left| \mathbf{\Sigma} \right| -\frac{1}{2}\left( \mathbf{y} - \mathbf{X} \mathbf{\beta} \right)^{\mathsf{T}} \mathbf{\Sigma}^{-1} \left( \mathbf{y} - \mathbf{X} \mathbf{\beta} \right) $$ – Sean Yun-Shiuan Chuang Oct 10 '20 at 10:07
  • 1
    Yes, absolutely. Thank you for spotting this mistake. – Durden Oct 12 '20 at 02:03
0

Yes, the solution is to use weighted least squares. The correlation structure is exchangeable, if I understand your notation. Generalized least squares simultaneously estimate the linear model and correlation structure with the EM algorithm.

AdamO
  • 52,330
  • 5
  • 104
  • 209