2

Assuming $X$ is a full-rank $n \times p$ design matrix, and $y$ is an $n$ vector, it is possible to solve the regression coefficient vector $\hat{\beta}$ using either $\hat{\beta}=(X^T X)^{-1} X^T y$ ie the OLS method, or by calculating the covariance matricies $S_{XX}$ and $S_{Xy}$, and then $\hat{\beta} = S_{XX}^{-1} S_{Xy}$.

I would like to prove that both expressions are equivalent to each other. The step I can do is expand the sample covariance terms to:

$$\begin{aligned} \hat{\beta} &= S_{XX}^{-1} S_{Xy}\\ &= \left( \frac{1}{n-1} X^TX - \bar{X}^T\bar{X} \right)^{-1} \left( \frac{1}{n-1} X^Ty - \bar{X}^T\bar{y} \right)\\ &=\left( X^TX - \bar{X}^T\bar{X} \right)^{-1} \left( X^Ty - \bar{X}^T\bar{y} \right) \end{aligned}$$

Where I'm stuck is proving that these mean terms $\bar{X}^T\bar{X}$ and $\bar{X}^T\bar{y}$ cancel out to result in the above expression for the OLS coefficients. How can this be done?

Migwell
  • 273
  • 2
  • 11
  • What does it mean to calculate the covariance between $X$ and $y?$ – Dave Nov 14 '21 at 04:09
  • https://en.wikipedia.org/wiki/Cross-covariance_matrix – Migwell Nov 14 '21 at 04:10
  • Although we are dealing with empirical covariances in this case – Migwell Nov 14 '21 at 04:12
  • What are $\bar X \bar X$ and $\bar X \bar y?$ Are these supposed to be vectors? Scalars? Matrices? – Dave Nov 14 '21 at 04:31
  • These are sample means. Y bar is a scalar and X bar is a p vector – Migwell Nov 14 '21 at 04:34
  • So then what is $\bar X \bar X?$ In other words, let $\bar X = (1, 2)^T$. What is $\bar X \bar X?$ – Dave Nov 14 '21 at 04:36
  • 1
    Oh whoops I forgot the transpose there – Migwell Nov 14 '21 at 04:37
  • The duplicate--which we have discussed in comments there--gives a perfectly rigorous mathematical demonstration. Another approach is to write, in matrix form, the Gaussian elimination steps described there. This leads to an easily proven identity. – whuber Nov 14 '21 at 16:08
  • I disagree, it uses a computational argument when I am after a symbolic, logical one. This Gaussian elimination argument is comparable to just showing me the results from both methods in R and telling me they are the same. – Migwell Nov 14 '21 at 23:51
  • Nothing about that argument requires any reference to software. As I pointed out in the preceding comment, all you are asking is to express the *same* argument in matrix notation. The answer is no different. – whuber Nov 15 '21 at 15:06

0 Answers0