1

Let $X$ and $Y$ be mean 0 and variance 1 random variables such that we choose $\alpha$ and $\beta$ to minimise

$$\mathbb{E}(X-\beta Y)^2$$

and

$$\mathbb{E}(Y-\alpha X)^2$$

after not so difficult derivation, I arrive at $\alpha = \mathbb{E}(XY)/\mathbb{E}X^2$ and $\beta = \mathbb{E}(XY)/\mathbb{E}Y^2$, so $\alpha = \beta$.

This seems very strange, because if $y=mx$ is regression line, then surely $x = \frac{1}{m }y$.

Lost1
  • 584
  • 4
  • 18
  • (X - Y)^2 = (Y - X)^2 for each X and Y. Basically it's the same function (expand the expressions for prove). That's why you get alpha = beta – Ivan Oct 23 '15 at 12:12
  • @Ivan sorry, you are going too fast for me. How are they the same function? The coefficient is in front of $X$ in one of the expression and $Y$ in the other. – Lost1 Oct 23 '15 at 12:15
  • that's minimization problem. In best case you have E(X - \betaY)^2 = 0 expanding the expression X^2 - 2X\betaY + Y^2 = 0 \beta = (X^2 - 2xy + y^2) and E(Y-\alphaX)^2 = 0 \alpha = (Y^2 - 2xy + X^2) witch is basically the same – Ivan Oct 23 '15 at 12:19
  • @Ivan when you expand you should get $X^2+2\beta XY +\beta^2 Y^2$? why has the beta's disappeared? I do not follow this. – Lost1 Oct 23 '15 at 12:21
  • @ChristophHanck I don't get what you are saying. What are you trying to answer? – Lost1 Oct 23 '15 at 13:27
  • @ChristophHanck Homework?! lol. I am a PhD student... I have already figured out $\alpha=\beta$. My question is that why is this the case, this seems counter intuitive for the reason I have given, $y=mx$ should imply $x=\frac{1}{m} y$ Maybe, I should have deleted the sentence "have I calculated $\alpha$ and $\beta$ wrong?$ – Lost1 Oct 23 '15 at 13:32
  • 1
    What do you precisely mean by "because if $y=mx$ is regression line, then surely $x = \frac{1}{m }y$". Is it to say that, if $\hat\beta$ is the regression coefficient of a regression of $y$ on $x$, $\hat\beta^{-1}$ is the coefficient of the regression of $x$ on $y$? – Christoph Hanck Oct 23 '15 at 14:07
  • @ChristophHanck that would seem natural, since the line is chosen to minimise the sum of square of distance of 'points' to the line. This "minimislng line", to me, seems unique. so why is the coefficient not related in this simple manner? obviously, there is something wrong this logic here, but I cannot quite pin down exactly what is wrong. – Lost1 Oct 23 '15 at 14:34
  • See related discussion here: http://stats.stackexchange.com/q/22718/67799 – Christoph Hanck Oct 23 '15 at 14:40
  • Or maybe in such hypothesis that $1/m=m$ ? Like with two similar distributions on which you have very few but similar infomation, such that they would be approximated with X=Y, because $1=\frac{1}{1}$ – Anthony Martin Oct 23 '15 at 13:48
  • Here is a simple a counterexample, (X,Y) jointly Gaussian with mean 0 and variance 1, correlation $\rho$ for any $\rho$... Please do not pose comments as answers. – Lost1 Oct 23 '15 at 14:30

1 Answers1

0

So I will answer my own question. The reason the two regression lines are different is this:

The regression line of $Y$ against $X$ minimises the total squared distance of the $Y$ COORDINATES to the $y$ COORDINATES of the line, where as $X$ against $Y$ minimise the total squared distance of $X$ COORDINATES to the $x$ COORDINATES line.

The confusion I had with regards to the line should be unique comes from the mistaken assumption the regression line minimise the total squared (PERPENDICULAR) distance of each point to the regression line. If this had been the case, my assumption would have been true. However, the two measure of distance never coincide unless there is perfect correlation.

Lost1
  • 584
  • 4
  • 18