12

Suppose are three time series, $X_1$, $X_2$ and $Y$

Running ordinary linear regression on $Y$ ~ $X_1$ ($Y = b X_1 + b_0 + \epsilon$ ), we get $R^2 = U$. The ordinary linear regression $Y$ ~ $X_2$ get $R^2 = V$. Assume $U < V$

What's the minimum and maximum possible values of $R^2$ on regression $Y$ ~ $X_1 + X_2$ ($Y = b_1 X_1 + b_2 X_2 + b_0 + \epsilon$ )?

I believe the minimum $R^2$ should be $V$ + a small value, since adding new variables always increases $R^2$, but I don't know how to quantify this small value, and I don't know how to obtain the maximum range.

Ferdi
  • 4,882
  • 7
  • 42
  • 62
CuriousMind
  • 2,133
  • 5
  • 24
  • 32

3 Answers3

10

1) EDIT: Cardinal's comment below shows that the correct answer to the min $R^2$ question is $V$. Hence I'm deleting my "interesting", but ultimately incorrect, answer to that part of the OP's post.

2) The maximum $R^2$ is 1. Consider the following example, which fits your case.

x1 <- rnorm(100)
x2 <- rnorm(100)
y <- x1 + 2*x2

> summary(lm(y~x1))$r.squared
[1] 0.2378023                 # This is U
> summary(lm(y~x2))$r.squared
[1] 0.7917808                 # This is V; U < V
> summary(lm(y~x1+x2))$r.squared
[1] 1

Here we are fixing the variance of $\epsilon$ at 0. If you want $\sigma^2_\epsilon > 0$, though, things change a little. You can get the $R^2$ arbitrarily close to 1 by making $\sigma^2_\epsilon$ smaller and smaller, but, as with the minimum problem, you can't get there, so there is no maximum. 1 becomes the supremum, since it's always greater than $R^2$ but it's also the limit as $\sigma^2_\epsilon \to 0$.

jbowman
  • 31,550
  • 8
  • 54
  • 107
  • 2
    (+1) Some comments: This is a good answer; it's interesting that you've taken an asymptotic approach whereas it's not clear whether the OP was interested in that or, possible, a fixed-$n$ one (or both). This answer is a little inconsistent with the OP's constraint that $U < V$, though, and if $X_1 = 0$ or $X_1 = a \mathbf{1}$ for some $a \in \mathbb R$, for example, then the minimum $R^2$ for all fixed sample sizes *is* exactly $V := V(n)$. (Excuse the pathology of these examples.) Also, OLS is *not* necessarily consistent absent additional constraints on the predictors. :) – cardinal Jul 14 '12 at 15:28
  • @cardinal - on rereading, I can't figure out why I took that approach to the min problem, when $V$ now seems like the obviously correct answer and, as you've implicitly observed, I could have constructed an example that achieves it in the vein of the max part... oh well, maybe my espresso this morning was accidentally decaf. (Maybe I should review my answers more thoroughly before posting, too!) – jbowman Jul 15 '12 at 02:24
  • I don't think you should remove what you've written, which I *did* find an interesting approach to answering the question! While the pathologies I mention certainly allow for a minimum $R^2$, one might wonder what is really meant by $X_1 = 0$. The other example is perhaps not quite as pathological since in a general version of this problem, it extends to the case where any additional $X_i$ is in the column space of the other predictors. :) – cardinal Jul 15 '12 at 02:31
  • 1
    @cardinal - thanks! I'll reconstruct it, maybe a little more formally, and put it back in at the bottom in a while. – jbowman Jul 15 '12 at 02:35
7

With no constraints on $U$ and $V$, then the minimum is $V$, and then maximum is the smaller $\min(V + U, 1)$. This is because two variable could be perfectly correlated (in which case adding the second variable does not change the $R^2$ at all) or they could be orthogonal in which case including both results in $U + V$. It was rightly pointed out in the comments that this also requires that each be orthogonal to $\mathbf{1}$, the column vector of 1s.

You added the constraint $U < V \implies X_{1} \neq X_{2}$. However, it is still possible that $U = 0$. That is, $X_{1} \perp Y$, in which case, $\min = \max = V + 0$. Finally, it is possible that $X_{1} \perp X_{2}$ so the upper bound is still $\min(V + U, 1)$.

If you knew more about the relationship between $X_{1}$ and $X_{2}$, I think you could say more.

cardinal
  • 24,973
  • 8
  • 94
  • 128
Joshua
  • 1,913
  • 15
  • 16
  • 1
    (+1) But, note that it is not (quite) true that if $X_1$ and $X_2$ are orthogonal, then their individual $R^2$ values will sum when including both in the model. We *also* need them to be orthogonal to the all-ones vector $\mathbf 1$. Note that you can use $\LaTeX$ on this site for marking up the math. :) – cardinal Jul 15 '12 at 22:00
  • That is true. Thanks very much for the comments, and for pointing out that $\LaTeX$ can be used. I thought it might but had tried mathjax style escapes \( and \[ for inline/equations. Writing just like I would in TeX worked like a charm :) – Joshua Jul 16 '12 at 03:35
  • From @Magot's answer below, what happens if $r_{1,Y}$ is negative, and $r_{1,2}$ is non-zero? We can find some setting here when $R^2$ is greater than $U+V$ and close to $1$, but $U$ and $V$ are less than say $0.3$ each. – newbie Oct 27 '21 at 04:04
6

Let $r_{1,2}$ equal the correlation between $X_1$ and $X_2$, $r_{1,Y}$ equal the correlation between $X_1$ and $Y$, and $r_{2,Y}$ the correlation between $X_2$ and $Y$. Then $R^2$ for the full model divided by $V$ equals

$$\left(\frac{1}{(1 - r_{1,2}^2)}\right) \left(1 - \frac{2 \cdot r_{1,2} \cdot r_{1,Y}}{r_{2,Y}} + \frac{U}{V}\right).$$

So $R^2$ for the full model equals $V$ only if $r_{1,2} = 0$ and $r_{1,Y}^2 = U = 0$ or

$$r_{1,2}^2 = \frac{2\cdot r_{1,2} \cdot r_{1,Y}}{r_{2,Y}} - \frac{U}{V}.$$

If $r_{1,2} = 0$, $R^2$ for the full model equals $U + V$.

cardinal
  • 24,973
  • 8
  • 94
  • 128
Margot
  • 61
  • 1
  • (+1) Cute. Welcome to the site. Please consider registering your account so you can participate more fully. I'll have to look at this expression a little more closely later on. :) – cardinal Jul 18 '12 at 12:54