6

As stated in many textbooks, the Standard error of the slope in linear regression with one variable is

$\sqrt{\frac{s^2}{SSX}}$

or some rewrite, ${s^2}$ being the error variance and ${SSX}$ being the sum of the ${x}$-squares.

Can anybody help with an explicit proof?

Clyde Frog
  • 293
  • 1
  • 3
  • 11
  • 1
    see http://stats.stackexchange.com/questions/88461/derive-variance-of-regression-coefficient-in-simple-linear-regression – TooTone Mar 28 '14 at 23:19
  • It's reasonably straightforward if you start from the fact that the line goes through $(\bar x,\bar y)$ and write the slope estimator as a kind of average. – Glen_b Mar 29 '14 at 00:01
  • Thx. Answer 1 to http://stats.stackexchange.com/questions/88461/derive-variance-of-regression-coefficient-in-simple-linear-regression helped me perfectly. – Clyde Frog Apr 09 '14 at 09:50

2 Answers2

7

There are a couple of rules to start with:

If $X$ is a random vector from $N(\mu,\Sigma)$ and $A$ is a Constant matrix then $AX \sim N(A\mu, A'\Sigma A)$.

And in a regression we assume $Y = \beta X + \epsilon$ where $\epsilon \sim N(0,\sigma^2 I)$.

We estimate $\hat\beta = (X'X)^{-1}X'Y$

So: $\hat\beta = (X'X)^{-1}X'(X\beta + \epsilon)= (X'X)^{-1}(X'X)\beta + (X'X)^{-1}X'\epsilon$

So $\hat\beta \sim N(\beta, (X'X)^{-1}X'\sigma^2IX(X'X)^{-1})$.

So the variance of $\hat\beta$ is $(X'X)^{-1}\sigma^2$

When you look at what is in $(X'X)^{-1}$ this becomes $\frac{\sigma^2}{SSX}$ for the slope.

Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • When you calculate the variance of beta hat, don't you need to calculate the variance of (X'X)^{-1}X'e? can you elaborate on why you can think of (X'X)^{-1}X' as constant matrix? Aren't they random variables? thanks! – aha Dec 11 '15 at 04:05
  • @aha, The x values in regression can be considered fixed or random depending on how the data was collected and how you want to think about things. If you do an experiment where you assign different doses or treatment levels as the x-variable then it is clearly not a random observance, but a fixed matrix. The formulas all work out the same whether you treat x as fixed or random (the fixed is just a little easier to show). A linear models text will go into more detail, I suggest "Linear Models in Statistics" by Rencher and Schaalje. – Greg Snow Dec 11 '15 at 22:32
  • thanks for the clarification and for the reference book! I remember when I learnt statistics, an *estimator* was framed as a transformation/function on Random Variables( i.e $\hat{\beta} = g(x_1,x_2,\cdots))$. When we ask questions on means/variances of that estimator, we need to look at the distribution of the input RVs($x_1,x_2,\cdots)$ instead of the particular realization(i.e constant). You mentioned they work out to be the same in this example. Is there a rule specifying when we can take them as constant vs has to use the original distribution? – aha Dec 12 '15 at 04:01
  • @aha, There are lots of times that we condition on things that might otherwise be considered as random. Sample size is the most common, but we also often condition on margins for chi-squared or Fisher's exact test. I don't know of a general rule, but the reference I gave would be a good place to start. – Greg Snow Dec 14 '15 at 18:42
5

To elaborate on Greg Snow's answer: suppose your data is in the form of $t$ versus $y$ i.e. you have a vector of $t$'s $(t_1,t_2,...,t_n)^{\top}$ as inputs, and corresponding scalar observations $(y_1,...,y_n)^{\top}$.

We can model the linear regression as $Y_i \sim N(\mu_i, \sigma^2)$ independently over i, where $\mu_i = a t_i + b$ is the line of best fit. Greg's way is to use vector notation.

We can rewrite the above in Greg's notation: let $Y = (Y_1,...,Y_n)^{\top}$, $X = \left( \begin{array}{2} 1 & t_1\\ 1 & t_2\\ 1 & t_3\\ \vdots \\ 1 & t_n \end{array} \right)$, $\beta = (a, b)^{\top}$. Then the linear regression model becomes:

$Y \sim N_n(X\beta, \sigma^2 I)$.

The goal then is to find the variance matrix of of the estimator $\widehat{\beta}$ of $\beta$.

The estimator $\widehat{\beta}$ can be found by Maximum Likelihood estimation (i.e. minimise $||Y - X\beta||^2$ with respect to the vector $\beta$), and Greg quite rightly states that

$\widehat{\beta} = (X^{\top}X)^{-1}X^{\top}Y$.

See that the estimator $\widehat{b}$ of the slope $b$ is just the 2nd component of $\widehat{\beta}$ --- i.e $\widehat{b} = \widehat{\beta}_2$ . Note that $\widehat{\beta}$ is now expressed as some constant matrix multiplied by the random $Y$, and he uses a multivariate normal distribution result (see his 2nd sentence) to give you the distribution of $\widehat{\beta}$ as

$N_2(\beta, \sigma^2 (X^{\top}X)^{-1})$.

The corollary of this is that the variance matrix of $\widehat{\beta}$ is $\sigma^2 (X^{\top}X)^{-1}$ and a further corollary is that the variance of $\widehat{b}$ (i.e. the estimator of the slope) is $\left[\sigma^2 (X^{\top}X)^{-1}\right]_{22}$ i.e. the bottom right hand element of the variance matrix (recall that $\beta := (a, b)^{\top}$). I leave it as exercise to evaluate this answer.

Note that this answer $\left[\sigma^2 (X^{\top}X)^{-1}\right]_{22}$ depends on the unknown true variance $\sigma^2$ and therefore from a statistics point of view, useless. However, we can attempt to estimate this variance by substituting $\sigma^2$ with its estimate $\widehat{\sigma}^2$ (obtained via the Maximum Likelihood estimation earlier) i.e. the final answer to your question is $\text{var} (\widehat{\beta}) \approx \left[\widehat{\sigma}^2 (X^{\top}X)^{-1}\right]_{22}$. As an exercise, I leave you to perform the minimisation to derive $\widehat{\sigma}^2 = ||Y - X\widehat{\beta}||^2$.

queenbee
  • 568
  • 4
  • 7