3

I am interested in how the estimate of the regression $\beta_1$ decreases with respect to sample size $n$. Does this look ok or am I missing something? $$ \begin{align} \text{Var}(\hat \beta_1) & = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar x)^2} \\ & \le \frac{\sigma^2}{\sum_{i=1}^n \min_i \{(x_i - \bar x)^2\}} \\ & = \frac{\sigma^2}{n \min_i\{(x_i - \bar x)^2\}} \\ & = O\bigg(\frac{1}{n}\bigg). \end{align} $$

Edit: It seems that the first answer in this post seems to agree with my result. However, the first answer in this post seems to say something else as it says that $\sigma^2$ is itself estimated (by the MSE) and that this estimate has a dependence on $n$.

Can someone clarify exactly how the variance decreases with $n$?

ManUtdBloke
  • 743
  • 4
  • 13
  • 1
    The answer depends on your model: exactly what marginal distribution of the explanatory variables are you assuming? This has already been explained in the answers to your previous question at https://stats.stackexchange.com/questions/491002/. – whuber Oct 08 '20 at 15:02
  • I am not making an assumption on the distribution of the explanatory variables as they are coming from real life observations. – ManUtdBloke Oct 08 '20 at 15:16
  • 1
    It would seem naturual that the more random observation we have (e.g. the larger $n$ gets), the more the variance is reduced, but I have not seen a formula that makes explicit how the variance decreases as $n$ increases. That is my motivation for asking this question. – ManUtdBloke Oct 08 '20 at 15:19
  • In kernel density estimation, I have seen proofs where the decay variance of the estimate with respect to $n$ is explicit (as the bandwidth goes to zero), so I figured it would also be possible to have similar explicit results for the decay of the variance of the coefficients in linear regression? Page 7 of http://faculty.washington.edu/yenchic/18W_425/Lec6_hist_KDE.pdf shows that the variance of a KDE estimate decays as $\frac{1}{nh}p(x_0) \sigma_K^2$, for example. – ManUtdBloke Oct 08 '20 at 15:21

1 Answers1

-1

The true variance of $\hat\beta_1$ is $\mathcal O(1/n)$, as you correctly proved. The estimated variance of $\beta_1$ is a random variable and its asymptotic behavior is better described using big O probability notation. Since the MSE is a consistent sinais of $\sigma^2$, we have $\hat\sigma^2=\sigma^2+\mathcal o_p(1)$. If we replace $\sigma^2$ by $\hat\sigma^2$ in your proof, we get that the estimated variance of $\beta_1$ is $\mathcal O_p(1/n)$.

TL;DR: the true variance is $\mathcal O(1/n)$, while the estimated variance is $\mathcal O_p(1/n)$ (that is, it is asymptotically bounded by $1/n$ with probability 1).

PedroSebe
  • 2,526
  • 6
  • 14