8

In An Introduction to Statistical Learning (James et al.), in section 3.7 exercise 5, it states that the formula for $\hat{\beta}_1$ assuming linear regression without an intercept is $$\hat{\beta}_1 = \dfrac{\displaystyle\sum\limits_{i=1}^{n}x_iy_i}{\displaystyle \sum\limits_{i=1}^{n}x_i^2}\text{,}$$ where $\hat{\beta}_0 = \bar{y}-\hat{\beta}_1\bar{x}$ and $\hat{\beta}_1 = \dfrac{\displaystyle S_{xy}}{S_{xx}}$ are the usual estimates under OLS for simple linear regression ($S_{xy} = \displaystyle\sum\limits_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})$).

This is not the actual exercise; I am merely wondering how to derive the equation. Without using matrix algebra, how do I derive it?

My attempt: with $\hat{\beta}_0 = 0$, we have $\hat{\beta}_1 = \dfrac{\bar{y}}{\bar{x}} = \dfrac{S_{xy}}{S_{xx}}$.

After some algebra, it can be shown that $\displaystyle S_{xy} = \sum\limits_{i=1}^{n}x_i^2 - n\bar{x}\bar{y}$ and $\displaystyle S_{xx} = \sum\limits_{i=1}^{n}x_i^2 - n\bar{x}^2$. From here, I'm stuck.

Clarinetist
  • 3,761
  • 3
  • 25
  • 70
  • 1
    The formula is immediate from [the geometric interpretation of least squares](http://stats.stackexchange.com/questions/17336), using $x/||x||$ as a "matcher" for $y$ and recognizing the formula for $(\hat\beta_1)x$ as being $(y\cdot (x/||x||))x/||x||$. – whuber Jul 02 '15 at 21:02
  • @whuber : Rather than writing $x/||x||,$ I would write $x/\|x\|.$ If that's not conspicuous enough for you, consider the typographical difference between $||x|| ||y||,$ coded as ||x|| ||y||, and $\|x\|\|y\|,$ coded as \|x\|\|y\|. $\qquad$ – Michael Hardy Feb 03 '19 at 18:31

1 Answers1

16

This is straightforward from the Ordinary Least Squares definition. If there is no intercept, one is minimizing $R(\beta) = \sum_{i=1}^{i=n} (y_i- \beta x_i)^2$. This is smooth as a function of $\beta$, so all minima (or maxima) occur when the derivative is zero. Differentiating with respect to $\beta$ we get $-\sum_{i=1}^{i=n} 2(y_i- \beta x_i)x_i$. Solving for $\beta$ gives the formula.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
meh
  • 1,902
  • 13
  • 18