2

I have two predictors $X_1$ and $X_2$, they are positively correlated. The response $Y$ are positively corrected with both $X_1$ and $X_2$, i.e. if we fit regression $Y=\beta_1 X_1$ and $Y=\beta_2 X_2$, the estimates are both positive. Now I fit $Y=\beta_1 X_1+\beta_2 X_2$, I understand that it is well possible that one of the estimated $\beta_1$ and $\beta_2$ flips sign. I understand the discussions in this post and this post but I really want to see this effect geometrically. That is, assume all the vectors are in $\mathbb{R}^3$, I want to draw three vectors such that Cor$(Y,X_1)$ and Cor$(Y,X_2)$ are positive when I project $Y$ onto $X_1$ and $X_2$ separately, but I want to see the sign of one of the correlations flips when I project $Y$ onto the column space of $[X_1\>X_2]$ first.

Could anyone please draw me an example? I struggle to see this visually.

dynamic89
  • 539
  • 3
  • 9
  • If X1 and X2 are correlated then it may be difficult to see this geometrically as they can't be the usual orthogonal axes. – Dayne Jun 29 '20 at 03:53
  • I’m not too sure what you mean. Could you please elaborate? In fact, if x1 and x2 aren’t correlated, the sign won’t change... – dynamic89 Jun 29 '20 at 08:35
  • If X1 and X2 are correlated then you cannot plot the points on a plane with X1 and X2 being the orthogonal axes. – Dayne Jun 29 '20 at 18:02
  • X1 and X2 are any vectors in $\mathbb{R}^3$, I didn't say that they are the axes – dynamic89 Jun 29 '20 at 19:58
  • Since in usual regression we assume X1 and X2 to be uncorrelated, these vectors are taken as orthogonal and thus are represented as axes of the plane on which Y is projected. So projection of Y, say (a*X1 + b*X2) is easy to see in terms of X1 and X2 as points (a,b). So very easy. In your case we need to imagine a parallelogram with X1 and (-X2) as sides (-X2 being the smaller side) with obtuse angle at origin. Now think of Yhat as the incomplete (smaller) diagonal of this parallelogram. Now if you picture right, Yhat will have positive dot product with X1 and +X2, which is what is you want. – Dayne Jun 30 '20 at 03:59
  • To picture it better, think of b to be very small such that the smaller side of the parallelogram is a lot smaller than the bigger side. – Dayne Jun 30 '20 at 06:00

1 Answers1

0

enter image description here

I hope this is correct. You can see $\hat{Y}$ is positively correlated with X1 and X2.

Dayne
  • 2,113
  • 1
  • 7
  • 24