5

Intuitively, if X can predict Y in a multiple linear regression model,

$y$ = $\beta_0$ + $\beta_1$$X$ + $\beta_2$$Z$ + $e$

X and Y are associated. Since there's an association between them, why is it not guaranteed that I could predict X by Y? Why is the relationship asymmetric? Or is my conception incorrect?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
JetLag
  • 1,105
  • 2
  • 9
  • 22
  • 5
    Do you mean after you have fit the model? It could be because you are optimising to reduce the error defined in the $y$ dimension so it is not optimized to reduce the error in the reverse. i.e. the minimized residuals are $(\hat{y}-y)^2$ which does not optimize $(\hat{x}-x)^2$ – Dan Nov 22 '17 at 10:11
  • 6
    Risky driving makes you more likely to die. If you died, were you a risky driver? – Tim Nov 22 '17 at 12:52
  • 4
    Related: [What is the difference between linear regression on y with x and x with y?](https://stats.stackexchange.com/q/22718/) – gung - Reinstate Monica Nov 22 '17 at 15:51
  • Take the case of $Z = X^2$, then you may have two values of $X$, or none, for a particular value of $y$. – James Nov 22 '17 at 17:02
  • @Tim You're comparing two different things. First you say risky driving makes you *more likely* to die, and then you ask whether dying means that one is a risky driver. You switched from "more likely to be" to "is". Dying does in fact mean that one is more likely a risky driver. – Acccumulation Nov 22 '17 at 17:35
  • The symmetric prediction would be predicting X on Y and Z, not X on just Y. – Acccumulation Nov 22 '17 at 17:46
  • @Acccumulation no I'm not. I'm thinking here of logistic regression model with binary dependent and independent variables. While risky driving would be a pretty good predictor of dying, dying would be a very bad predictor of risky driving since people die for many different reasons, and even the drivers that die in the car accidents die for many different reasons, with risky driving being only one of them. – Tim Nov 22 '17 at 18:09
  • This may sound like a very general answer, but the question seems as a restatement of "correlations do not imply causality" in terms of linear regression. – Roger Vadim Nov 22 '17 at 11:58

3 Answers3

11

The answer depends on what you mean by “predict”. If you imply any kind of causation then obviously it is one way road. Suppose sunrise causes you to wake up. If I wake you up in the middle of the night sun will not rise suddenly.

On the other hand, if you mean by predicting an explanatory power of X in a multiple regression Y~1+X+Z or explanatory power of Y in a regression X~1+Y+Z, then it is a different story. You can certainly invert the relationship algebraically as long as it is strong. If I know what time you wake up I can predict when the sun rises.

I conditioned on relationship being "strong" because when you invert the equation your optimization problem changes. Instead of minimizing squares $(\hat y_i-y_i)^2$ we minimize squares $(\hat x_i-x_i)^2$. It is a different equation that may not render a significant relationship if the relationship was weak to start with. For instance, you can find that $\beta_1$ is significant, i.e. X predicts Y in this narrow definition. Yet, when you invert the equation you may end up with a model where coefficient of Y is not significant, so in this sense Y does not predict X.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • How can you invert the relationship without also knowing values of $Z$? This is mathematically impossible. – whuber Nov 22 '17 at 14:58
  • 1
    @whuber OP means to regress X on Y and Z, so you do know Z. OP explicitly wrote it's a multiple regression setup – Aksakal Nov 22 '17 at 15:10
  • 3
    Here's how I am interpreting it: the question explicitly asks "predict X by Y," with no mention of Z. If we do assume Z given, then it effectively disappears from the model leaving us with a simple OLS situation, not multiple regression. That's why I felt that some clarification of your assertion that one can "certainly inverse the relationship" is needed. – whuber Nov 22 '17 at 15:21
  • @whuber I clarified language to make it explicit that I'm talking about multiple regression both ways – Aksakal Nov 22 '17 at 15:26
  • 3
    Thank you. I'm unsure what you mean by "strong," though. The question explicitly concerns a *model* rather than an *estimate* derived from data. Because the assumption is that $X$ is related to $y$, I understand that to mean $\beta_1\ne 0$. We could then observe that a simple algebraic transformation yields the multiple regression model $$X = -\frac{\beta_0}{\beta_1}-\frac{1}{\beta_1}y-\frac{\beta_2}{\beta_1}Z+\delta$$ where $$\delta=\frac{1}{\beta_1}e$$ is zero-mean and homoscedastic if and only if $e$ is. The idea of "strong" does not seem to apply. – whuber Nov 22 '17 at 15:31
  • @whuber, what I meant was that it's possible to get $\beta_1$ significant and $\delta$ not significant, because our optimization objectives are different. Maybe I need to re-word this – Aksakal Nov 22 '17 at 15:34
4

When you use the regression equation to make a prediction by plugging in a value of $x$, you are not predicting the value of $y$ for that value of $x$. You are predicting the mean of the $y$-values for that value of $x$. In detail:

The regression equation $$y = \beta_0 + \beta_1 x + \epsilon$$ says that $y$ is equal to a linear function of $x$ plus some random scatter. If you set $x=3$, say, you have $$y = \beta_0 + 3\beta_1 + \epsilon$$ and there is still some random scatter there. In other words, you are saying that "my prediction is that $y$ is normally distributed with mean equal to $\beta_0 + 3\beta_1$". To get an actual value for $y$, you need to take the expectation. So you are saying that "The mean of all the $y$-values for which $x=3$ is $\beta_0 + 3\beta_1$".

If you make a prediction by inverting the regression equation, say by plugging in $y=4$, then you are saying "The $x$-value for which the mean of all the corresponding $y$-values is equal to $4$ is $(4-\beta_0)/\beta_1$", which isn't usually the kind of prediction that you want.

Statistical courses often don't help by talking about "the line of best fit", which makes it sound like the situation is symmetrical in $x$ and $y$, which is very not the case. Recently there was a debate on the ANZSTAT mailing list, and someone posted a link to a good introductory course which explains it well:

https://www.stat.berkeley.edu/~stark/SticiGui/Text/regression.htm

Flounderer
  • 9,575
  • 1
  • 32
  • 43
  • 1
    Interesting point, it contradicts a little with my answer and I'd like to clarify/understand more. Are you saying that a linear relationship is not symmetric if there is noise over one dimension? – A.D Nov 22 '17 at 16:16
0

This is an interesting question. In the case of single variable linear regression, there is a (assumption) symmetric relationship but the same is not true for multiple linear regression.

A symmetric relationship can exist for some problems, a contrived example is training a linear regression model to behave like an AND gate.

If you ask why is this not "guaranteed" then there is a clear counterexample (proof by construction) which is training an OR gate using linear regression. You can learn a model that can predict the output of the gate $Y$ given 2 inputs $X_1, X_2$ accurately but the reverse is not possible.

Another way to think about it is that the multivariable linear regression model learns a many-to-one dimension mapping. And since many different points on the input can map to the same point on the output the reverse mapping is ambiguous.

UPDATE: The other answers seems to explain why the linear-regression objective (or loss function) is not symmetric. But I think you are asking a different question which is (paraphrasing), once I learn a linear relationship between $Y$ and $X_1, X_2$, why can't I use it in reverse?

A.D
  • 2,114
  • 3
  • 17
  • 27
  • The *relationship* between $X$ and $Y$ can have strong symmetries, but *regression analysis itself* does not treat the variables symmetrically. A standard model is that the data $(x_i,y_i)$ are a sample from a bivariate Normal distribution of unknown parameters: that's a symmetric assumption. However, one asks an *asymmetric* question: namely, as a function of possible values $x$, what is the conditional expectation of $Y$ when $X=x$? Algebraic inversion of that function does *not* answer the analogous question when the roles of $X$ and $Y$ are switched. – whuber Nov 23 '17 at 14:29
  • I agree that regression analysis does not treat the problem symmetrically - that is what I meant by "linear-regression objective is not symmetric". But a linear relationship (once obtained) is symmetric for some class of problems. (1d case or cases of highly correlated input dimensions). – A.D Nov 23 '17 at 15:00
  • It's a limited class of problems. Apart from the multivariate normal case, there aren't too many situations where the conditional expectation is a linear function regardless of the variables that one conditions on. That's not usually the case even for ordinary regression (which I believe you might mean by "1d"). – whuber Nov 23 '17 at 15:03
  • Let's forget linear regression for a moment. Suppose someone tells me a relation between $Y$ and $X$ is defined by $y = mx + c$. I can derive $x = (y -c) / m$ and use it as a predictor, in doing so, I assume symmetry. (Yes, I know that the value of $m,c$ would be different if I regress $x$ on $y$ vs. $y$ on $x$, but I _can_ still derive an inverse). On the other hand, if someone tells me $y = m_1x_1 + m_2x_2 + c$ there is nothing I can do to invert the relationship unless I'm working with some special case, $x_1 = x_2$ etc. Are my statements true? If not what is incorrect? – A.D Nov 23 '17 at 17:17
  • You appear to be overlooking the statistical aspect of this issue, which is the error term. To make some progress, you need to include it explicitly in your models. For one example of that, see my [comment to Aksakal's answer.](https://stats.stackexchange.com/questions/315072/if-x-can-predict-y-in-regression-why-isnt-y-guaranteed-to-predict-x/315151?noredirect=1#comment598132_315118) – whuber Nov 24 '17 at 13:17