Expectation in Linear Regression with Least Squares

Question

We know that the best predictor of $\beta$ using the least squares criteria for linear regression is $\hat{\beta} = (X^TX)^{-1}X^Ty$ and I can derive this equation by minimizing the squared error in the training sample.

But when we minimize the expected prediction error, we get: $$ \begin{aligned} \frac{\partial EPE(\beta)}{\partial \beta} = &\iint\frac{\partial}{\partial \beta}(x^T\beta - y)^2dxdy = 0\\ \iff &\iint 2(x^T\beta - y)x\, dxdy = 0 \\ \iff &2\iint x^T\beta x - yx\, dxdy = 0 \\ \end{aligned} $$

Now $x^T\beta$ is a scalar, so $x^T\beta x = x(x^T\beta)$. So: $$ \begin{aligned} \iff &\iint x(x^T\beta) - yx\, dxdy = 0 \\ \iff &E(XX^T\beta) - E(YX) = 0 \\ \iff &\beta = (E(XX^T))^{-1}E(YX) \end{aligned} $$

I dont know how this expectation leads to $\hat{\beta}$. It doesn't seem to add up since $XX^T$ is a NxN matrix. What am I missing?

Thanks!

---

Edit: since it was pointed out that if $X$ is a random variable, the expectation has to be conditioned, we have:

$$ \begin{aligned} \frac{\partial EPE(\beta)}{\partial \beta} = &\iint\frac{\partial}{\partial \beta}(x^T\beta - y)^2 \Pr(x,y) = 0\\ \iff &\iint 2(x^T\beta - y)x \Pr(x,y) = 0 \\ \iff &2\iint [x^T\beta x - yx] \Pr(y|x) \Pr(x) = 0 \\ \iff &E_XE_{Y|X} [X(X^T\beta) - YX] = 0 \\ \iff &E_X(XX^T\beta) - E_XE_{Y|X}(YX) = 0 \\ \iff &\beta = (E_X(XX^T))^{-1}E_XE_{Y|X}(YX) \end{aligned} $$

Answer 1

Indeed, $X$ is not random as @whuber mentioned in the comments. Recall formulation of the Linear Regression problem:

$$ \begin{aligned} y_j =& \beta_1 x_{j1} + ... + \beta_n x_{jn} + \epsilon_j\\ \mathbf{y} =& X \boldsymbol{\beta} + \boldsymbol{\epsilon} \end{aligned} $$

where $\mathbb{E}\epsilon_j=0$. We are given $x_{ij}$ and want to estimate $\theta_j$. Note, that $x_{ij}$ are somewhat not random. Suppose, we are trying to figure out the properties of the device. We pass $x_{ji}$ as input (we know them precisely) and measure the output $y_j$ with some error $\epsilon_j$. So by definition of the problem:

$$ \mathbb{E}y_j = \beta_1 x_{j1} + ... + \beta_n x_{jn} $$

Hence

$$ \begin{aligned} \mathbb{E}(X^TX) = X^TX\\ \mathbb{E}(X^T\mathbf{y}) = X^T\mathbf{y}\\ \end{aligned} $$

Complete derivation may look like that

\begin{aligned} &\iint\frac{\partial}{\partial \beta}(X\hat{\boldsymbol{\beta}} - \mathbf{y})^2dxdy = 0\\ \iff &2\iint X^T(X\hat{\boldsymbol{\beta}} - \mathbf{y})\, dxdy = 0 \\ \iff &2\iint X^T X\hat{\boldsymbol{\beta}} - X^T\mathbf{y}\, dxdy = 0 \\ \iff &E(X^T X\hat{\boldsymbol{\beta}}) - E(X^T\mathbf{y}) = 0 \\ \iff &X^T X\hat{\boldsymbol{\beta}}- X^T\mathbf{y} = 0 \\ \iff &\hat{\boldsymbol{\beta}} = (X^T X)^{-1} X^T\mathbf{y} \\ \end{aligned}

By the way, I am not sure that it actually makes sense to write integrals and take expectations, because we are dealing with constants. Probably it will be better to stick with something like that:

\begin{aligned} \ &(\mathbf{y} - \hat{\mathbf{y}})^T(\mathbf{y} - \hat{\mathbf{y}}) \rightarrow \min\\ \Rightarrow\ &(\mathbf{y} - \hat{\mathbf{y}})^TX = 0\qquad\text{residual}\perp\mathbf{y}\text{'s manifold}\\ \Rightarrow\ &\mathbf{y}^TX - \hat{\boldsymbol{\beta}}^TX^TX = 0\\ \Rightarrow\ &\hat{\boldsymbol{\beta}} = (X^T X)^{-1} X^T\mathbf{y} \end{aligned}

---

Edit. About taking expectation, it seems that you are right. But probably there is a small mistake connected with vector derivative. It should be:

$$ \begin{aligned} &2\iint (X^T X\hat{\boldsymbol{\beta}} - X^T\mathbf{y})\mathbb{P}(y|x)\mathbb{P}(x) dxdy = 0 \\ \iff &\mathbb{E}_X(X^TX\hat{\boldsymbol{\beta}}) - \mathbb{E}_X\mathbb{E}_{Y|X}(X^T\mathbf{y})=0 \end{aligned} $$

So

$$ \begin{aligned} \mathbb{E}(X^TX\hat{\boldsymbol{\beta}}) =& \mathbb{E}\sum_{i,j,k} x_{ji}x_{jk} \hat{b_k}\\ =& \sum_{i,j,k} \mathbb{E} x_{ji}x_{jk} \hat{b_k}\\ =& \sum_{k} \hat{b_k} \sum_{i,j} \mathbb{E} x_{ji}x_{jk}\\ =& (\mathbb{E} X^TX)\hat{\boldsymbol{\beta}} \end{aligned} $$

That is why

$$ \begin{aligned} &\mathbb{E}_X(X^TX\hat{\boldsymbol{\beta}}) - \mathbb{E}_X\mathbb{E}_{Y|X} (YX) = 0\\ \iff& \hat{\boldsymbol{\beta}} = (E_X(X^TX))^{-1}E_XE_{Y|X}(X^T\mathbf{y}) \end{aligned} $$