0

In simple (univariate) OLS $y = \beta_0 + \beta_1 x$, the estimate of the slope coefficient is asymptotically equal to the ratio of covariances, i.e. $$ \hat{\beta_1} \rightarrow \frac{Cov[X,Y]}{Cov[X,X]} $$ so if we know what the covariance matrix for the joint distribution of $[X,Y]$ is we can derive OLS coefficients without needing any data.

Can we do the same for the multivariate case? For example, if we know the covariance matrix of the data-generating process for $[Y,X_1,X_2]$, what would be the asymptotic expression for OLS coefficients in $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2$?

Add 1 If the data was centered (or the means where zero) then the data matrix $\textbf{X}^T\textbf{X}$ (without column of 1s) would asymptotically be equal to the co-variance matrix of $[X_1,X_2]$. But the estimate includes the column of 1s and work with non-centered data, so it's not clear to me what this becomes in this case.

Add 2 If do apply the OLS formula for zero-mean RVs without adding the column of 1s to the data matrix, then for 2 explanatory variables we get $$ \textbf{X}^T \textbf{X} = \begin{bmatrix} x_{1,1} & x_{1,2} \\ x_{2,1} & x_{2,2} \\ \vdots & \vdots \\ x_{n,1} & x_{n,2}\\ \end{bmatrix}^T \begin{bmatrix} x_{1,1} & x_{1,2} \\ x_{2,1} & x_{2,2} \\ \vdots & \vdots \\ x_{n,1} & x_{n,2}\\ \end{bmatrix} = \begin{bmatrix} \sum_{i=1}^{n} x_{i,1}^2 & \sum_{i=1}^{n} x_{i,1}x_{i,2} \\ \sum_{i=1}^{n} x_{i,1}x_{i,2} & \sum_{i=1}^{n} x_{i,2}^2 \\ \end{bmatrix} $$ and its inverse is $$ (\textbf{X}^T \textbf{X})^{-1} = \frac{1}{\sum_{i=1}^{n} x_{i,1}^2 \sum_{i=1}^{n} x_{i,2}^2 - (\sum_{i=1}^{n} x_{i,1}x_{i,2})^2} \begin{bmatrix} \sum_{i=1}^{n} x_{i,2}^2 & -\sum_{i=1}^{n} x_{i,1}x_{i,2} \\ -\sum_{i=1}^{n} x_{i,1}x_{i,2} & \sum_{i=1}^{n} x_{i,1}^2 \\ \end{bmatrix} $$ The other term is given by $$ \textbf{X}^T \textbf{y} = \begin{bmatrix} x_{1,1} & x_{1,2} \\ x_{2,1} & x_{2,2} \\ \vdots & \vdots \\ x_{n,1} & x_{n,2}\\ \end{bmatrix}^T \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \\ \end{bmatrix} = \begin{bmatrix} \sum_{i=1}^{n} x_{i,1}y_{i} \\ \sum_{i=1}^{n} x_{i,2}y_{i} \\ \end{bmatrix} $$ Putting it together gives $$ \begin{bmatrix} \hat{\beta}_{1} \\ \hat{\beta}_{2} \\ \end{bmatrix} = (\textbf{X}^T \textbf{X})^{-1} \textbf{X}^T \textbf{y} = \frac{1}{\sum_{i=1}^{n} x_{i,1}^2 \sum_{i=1}^{n} x_{i,2}^2 - (\sum_{i=1}^{n} x_{i,1}x_{i,2})^2} \begin{bmatrix} \sum_{i=1}^{n} x_{i,2}^2 \sum_{i=1}^{n} x_{i,1}y_{i} -\sum_{i=1}^{n} x_{i,1}x_{i,2} \sum_{i=1}^{n} x_{i,2}y_{i} \\ \sum_{i=1}^{n} x_{i,1}^2 \sum_{i=1}^{n} x_{i,2}y_{i} -\sum_{i=1}^{n} x_{i,1}x_{i,2} \sum_{i=1}^{n} x_{i,1}y_{i} \\ \end{bmatrix} $$ which seems to agree with the result in the OP of the question pointed to by whuber. Asymptotically, this looks like $$ \begin{bmatrix} \hat{\beta}_{1} \\ \hat{\beta}_{2} \\ \end{bmatrix} \rightarrow (Cov(\textbf{X},\textbf{X}))^{-1} Cov(\textbf{X},Y) $$ where $\textbf{X}$ now indicates a multivariate RV, not the data matirx of observations. Since covariances do not depend on means, I believe that this result also holds for non-zero mean RVs.

Confounded
  • 513
  • 2
  • 12
  • Re the edit: it's unclear where you're going with this, because the estimates are the same whether or not the columns of $X$ are centered provided only that the span of the columns includes the constant vector. The asymptotic distribution doesn't know or care how you go about actually computing the estimates! – whuber Feb 27 '20 at 14:42
  • @whuber Thank you for you comment and for pointing me to the other post. On where I was going with the edit: I thought that maybe the generalization can be expressed as "the ratio" of covariance matricies, i.e. $(Cov[\textbf{X},\textbf{X}])^{-1}(Cov[\textbf{X},Y])$ that is the inverse of covariance matrix of Xs times the covariance matrix of Xs with Y. – Confounded Feb 27 '20 at 14:51

0 Answers0