2

So, I have been looking at this post, and others that are similar and know that the least square estimation of $\beta_1,\beta_2$ will be $(X^TX)^{-1}X^TY$, where the model is $Y_i = \beta_1x_{1i}+ \beta_2x_{2i} + \epsilon_i$. But what is the generally derived formula in this case for $\hat{\beta_1}$ and $\hat{\beta_2}$. How can represent them in a scalar multiplied out value/weight format rather than a matrix/vector format?

User_13
  • 75
  • 8
  • 1
    Have you tried writing out the steps when you carry out the matrix arithmetic? What have you found? – Sycorax Jun 06 '21 at 02:51
  • 1
    Yes, I seem to get a symmetric matrix $X^T X$, but not sure how I would write its inverse and the entries in that? – User_13 Jun 06 '21 at 02:54
  • LU factorization is one way to solve the linear system, and avoid the explicit inverse. What problem are you trying to solve, and why are you trying to avoid linear algebra in solving it? – Sycorax Jun 06 '21 at 02:58
  • I want to use linear algebra but not sure how to derive the least squares estimators for the two regression weights here. I've set up the data generating model and objective function, solved the optimization problem by taking the partial derivatives with respect to each of the estimators and so forth, but I'm not clear on how I can show/represent the actual values for the estimated $\beta_1$ and $\beta_2$ from the derivation. – User_13 Jun 06 '21 at 03:08
  • Why is $(X^T X)^{-1}X^T Y$ not showing the values of the estimated $\beta$? Your description says that you've already demonstrated that it's the optimal value because it minimizes the objective function. Are you asking how to compute $\beta$ given $X$ and $Y$ when $X$ is $n \times 2$? – Sycorax Jun 06 '21 at 03:11
  • I think I'm being unclear here, sorry. So you see in the first hyperlinked post, in the simple linear regression case, the weight is estimated and represented as $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$. I want to know how to write out the estimated weights in a similar way for the multiple linear model I've stated above. I just don't know how to do it (translating it from the matrix form to this). I hope I'm making sense? – User_13 Jun 06 '21 at 03:16
  • 1
    It sounds like you're on the right track, writing out the linear algebra steps using summation notation. If you're stuck writing $(X^T X)^{-1}$, remember that in this problem, $X^T X$ is just a $2 \times 2$ matrix, which has a very simple inverse. – Sycorax Jun 06 '21 at 03:23

1 Answers1

3

Based on the comments above (by @Sycorax):

$$X^TX = \begin{pmatrix} x_{11} & x_{12} & ... &x_{1n} \\ x_{21} & x_{22} & ... &x_{2n} \end{pmatrix} \begin{pmatrix} x_{11} & x_{21} \\ x_{12} & x_{22} \\ \vdots &\vdots\\ x_{1n} & x_{2n} \end{pmatrix} = \begin{pmatrix} x_{11}^2 + x_{12}^2 + \:...\: x_{1n}^2 & x_{11}x_{21} + x_{12}x_{22} + \:...\: +x_{1n}x_{2n} \\ x_{11}x_{21} + x_{12}x_{22} + \:...\: +x_{1n}x_{2n} & x_{21}^2 + x_{22}^2 + \:...\: x_{2n}^2 \\ \end{pmatrix} = \begin{pmatrix} \sum x_{1i}^2 & \sum x_{1i}x_{2i}\\ \sum x_{1i}x_{2i} & \sum x_{2i}^2 \\ \end{pmatrix}$$

$$(X^TX)^{-1} = \dfrac{1}{\sum x_{1i}^2 \sum x_{2i}^2 - (\sum x_{1i}x_{2i})^2} \begin{pmatrix} \sum x_{2i}^2 & - \sum x_{1i}x_{2i} \\ - \sum x_{1i}x_{2i} & \sum x_{1i}^2 \end{pmatrix}$$

$$X^T Y = \begin{pmatrix} x_{11} & x_{12} & ... &x_{1n} \\ x_{21} & x_{22} & ... &x_{2n} \end{pmatrix} \begin{pmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{pmatrix} = \begin{pmatrix} x_{11}y_1 + x_{12}y_2 + \: ... \: + x_{1n}y_n \\ x_{21}y_1 + x_{22}y_2 + \: ... \: + x_{2n}y_n \end{pmatrix} = \begin{pmatrix} \sum x_{1i}y_i \\ \sum x_{2i}y_i \end{pmatrix} $$

$$\therefore \beta = (X^TX)^{-1}X^TY = \begin{pmatrix} \hat{\beta_1} \\ \hat{\beta_2} \end{pmatrix} = \dfrac{1}{\sum x_{1i}^2 \sum x_{2i}^2 - (\sum x_{1i}x_{2i})^2}\begin{pmatrix} \sum x_{2i}^2 \sum x_{1i}y_i - \sum x_{1i}x_{2i} \sum x_{2i}y_i \\ -\sum x_{1i}x_{2i} \sum x_{1i}y_i + \sum x_{1i}^2 \sum x_{2i}y_i \end{pmatrix} $$

User_13
  • 75
  • 8