3

Consider a linear regression model: $Y_i = \beta_1 A_i + \beta_2 B_i + u_i$ where all variables are assumed to have mean 0, and $A_{i}$ is distributed independently of both $B_{i}$ and $u_{i}$, but $B_{i}$ and $u_{i}$ might be correlated.

I want to show that the OLS estimator $\widehat{\beta}_1$ is consistent but $\widehat{\beta}_2$ might not be due to the correlation. So I'm deducing the formula for the OLS estimators and this is what I got: $$\widehat{\beta}_1 = \left(\sum A_i A_i'\right)^{-1} \left(\sum A_iY_i\right)$$ and $$\widehat{\beta}_2 = \left(\sum B_i B_i'\right)^{-1} \left(\sum B_i Y_i \right).$$ Is this formula correct? Can any one give me some hint on the problem?

Any help is appreciated, thanks in advance!

Juan
  • 31
  • 5
  • 1
    I don't think this is a duplicate because that other question was about fitting only one parameter. A complication arises in this question because the tuples $\left( a_i : i=1,\ldots,n\right)$ and $\left( b_i : i=1,\ldots, n \right)$ may be correlated. And any answer to that other question would not say what to do about that. The answer proposed in this present posted question is wrong precisely because of that issue, which that other question cannot address. – Michael Hardy Feb 03 '19 at 18:36
  • Thank you! That's exactly what I have in mind. Actually, in the original question, $b_{i}$ are correlated with $u_{i}$ but $a_{i}$ are uncorrelated with $b_{i}, u_{i}$ – Juan Feb 03 '19 at 22:31
  • This question has a very large number of duplicates: look at any thread that discusses formulas for multiple regression. – whuber Feb 03 '19 at 22:38
  • I've looked at a lot of threads but none of them discussed the potential correlation between $b_{i}$ and $u_{i}$...I might have missed some threads – Juan Feb 03 '19 at 22:40
  • Your question doesn't posit or ask about "potential correlation." Could you be more specific about what you're looking for? If you are assuming $b$ and $u$ are random variables and you do not assume they are independent, then at the very least you need to stipulate *how* they fail to be independent, for otherwise there won't be any formula. – whuber Feb 03 '19 at 22:42
  • Sorry for the ambiguity of the description, I've edited my question now – Juan Feb 03 '19 at 22:51
  • If $(A_1, B_1), \ldots, (A_n, B_n)$ are i.i.d. and $\operatorname{cov}(A_1,B_1)=0,$ that does not imply the _sample_ correlation will be $0.$ It is only when the sample correlation is $0$ that you can treat $A$ and $B$ separately. $\qquad$ – Michael Hardy Feb 04 '19 at 20:39
  • @whuber : It this has a large number of duplicates, could you link to one? The one linked by kjetil b halvorsen, mkt, and Peter Flom is not one of which this one is a duplicate, for the reason I mentioned. – Michael Hardy Feb 04 '19 at 20:41
  • @Michael [Here's a focused search.](https://stats.stackexchange.com/search?q=multiple+regression+normal+equation*+formula) If you would like to suggest one (or more) of the hits, then we could change the reference to the duplicate. Or do you perhaps have in mind something like https://stats.stackexchange.com/questions/46151? Or https://stats.stackexchange.com/questions/352130? Or possibly something less formulaic, like https://stats.stackexchange.com/questions/17336? – whuber Feb 04 '19 at 20:49

1 Answers1

2

Your notation has interesting assumptions. Considering $Y_i$ and $y_i$ are the same, $a_i,b_i$ should be real numbers representing your data points. But,in your notation, they're represented like vectors, e.g. you have $a_ia_i'$. If they were vectors, say $m\times 1$, you couldn't also have $a_i y_i$ because in order to have $y_i=\beta_1a_i+\beta_2b_i+u_i$, $y_i$ needs to have the same dimension with $a_i$, and the vector multiplication $a_iy_i$ becomes invalid. If you just have a typo, i.e. you have something like $\sum a_i y_i'$, which is algebraically valid, then you're either trying to solve a system of multiple outputs with more than one $\beta_1,\beta_2$ (i.e. $\beta_{1,2}$ are vectors, where you have separate $\beta_1,\beta_2$ for each element of $y_i$ vector, and you can solve the systems separately) or you constraint all the system to have the same scalar $\beta_1,\beta_2$ where you can actually add each of the outputs (i.e. $y_i=[y_{i1},y_{i2},...,y_{im}]$) as more data points, in the end having $nm$ data points. Either way, there is no need to solve something like that.

But, if we assume that these are just scalars, in which the problem translates into the usual multiple regression, then you can simply write your solution as $\beta_1=\frac{\sum a_i y_i}{\sum a_i^2}$, which is correct only if you don't have the second regressor. So, your answers are valid for the individual models $y_i=\beta_1 a_i+u_i$ and $y_i=\beta_2 b_i+u_i$. However, this doesn't account for the dependence between $\beta_1,\beta_2$. The coefficients tend to affect each other when other terms are incorporated.

The general formula for multiple regression is $\beta=(X^TX)^{-1}X^Ty$, where $X$ has $n\times p$, and $y$ has $n \times 1$, $\beta$ has $p \times 1$ dimensions (here $p=2$). Each row of $X$ corresponds to data points $(a_i,b_i)$ and rows of $\beta$ corresponds to $\beta_1$ and $\beta_2$. We can also come up with the same solution by differentiating $\text{SSE}=\sum(y_i-\hat{y_i})^2$ and solving for $\beta_1,\beta_2$. Correctly formulating the problem yields the following setup:

$$X=\begin{bmatrix} a_1 & \ldots & a_n \\ b_1 & \ldots & b_n\end{bmatrix}, y=\begin{bmatrix} y_1 \\ \vdots \\ y_n\end{bmatrix},\beta=\begin{bmatrix}\beta_1 \\ \beta_2\end{bmatrix}$$ Now, we just substitute all together into the equations: $$(X^TX)^{-1}=\left(\begin{bmatrix} a_1 & \ldots & a_n \\ b_1 & \ldots & b_n\end{bmatrix}\begin{bmatrix}a_1 & b_1 \\ \vdots&\vdots \\ a_n & b_n \end{bmatrix}\right)^{-1}=\begin{bmatrix}\sum a_i^2 & \sum a_ib_i \\ \sum a_ib_i & \sum b_i^2\end{bmatrix}^{-1}=\frac{1}{D}\begin{bmatrix}\sum b_i^2 & -\sum a_ib_i \\ -\sum a_ib_i & \sum a_i^2\end{bmatrix}$$

where $D=(\sum a_i^2)(\sum b_i^2)-(\sum a_ib_i)^2$, i.e. the determinant. And, we have $X^Ty=\begin{bmatrix}\sum a_iy_i \\ \sum b_iy_i\end{bmatrix}$.

Finally, we'll have $$\beta=\frac{1}{D}\begin{bmatrix}(\sum b_i^2)(\sum a_iy_i)-(\sum a_ib_i)(\sum b_iy_i) \\ (\sum a_i^2)(\sum b_iy_i)-(\sum a_ib_i)(\sum a_iy_i) \end{bmatrix}$$

First row is $\beta_1$, second row is $\beta_2$.

gunes
  • 49,700
  • 3
  • 39
  • 75
  • Thank you for the answer! However, I guess this would only work if we assume $a_{i}, b_{i}, u_{i}$ are all uncorrelated right? In the original question, $a_{i}$ is uncorrelated to others but $b_{i}$ and $u_{i}$ might be related, so $\hat{\beta}_{1}$ is consistent but $\hat{\beta}_{2}$ might not be, and I have problem with showing that... – Juan Feb 03 '19 at 22:38
  • If you know $b$ and $u$ are correlated but don't attempt to model that correlation, you can still estimate $\hat{\beta}_2$ but it will be biased for $\beta_2$. If you attempt to model that correlation in your estimation, the model is not identified (i.e., there are more unknowns than knowns). So, there is no formula for an unbiased estimator of $\beta_2$ when $b$ and $u$ are correlated. – Noah Feb 04 '19 at 01:01