How can the sum of two variables explain more variance than the individual variables?

Question

I am getting some perplexing results for the correlation of a sum with a third variable when the two predictors are negatively correlated. What is causing these perplexing results?

Example 1: Correlation between the sum of two variables and a third variable

Consider formula 16.23 on page 427 of Guildford's 1965 text, shown below.

Perplexing finding: If both variables correlate .2 with the third variable and correlate -.7 with each other, the formula results in a value of .52. How can the correlation of the total with the third variable be .52 if the two variables each correlate only .2 with the third variable?

Example 2: What is the multiple correlation between two variables and a third variable?

Consider formula 16.1 on page 404 of Guildford's 1965 text (shown below).

Perplexing finding: Same situation. If both variables correlate .2 with the third variable and correlate -.7 with each other, the formula results in a value of .52. How can the correlation of the total with the third variable be .52 if the two variables each correlate only .2 with the third variable?

I tried a quick little Monte Carlo simulation and it confirms the results of the Guilford formulas.

But if the two predictors each predict 4% of the variance of the third variable, how can a sum of them predict 1/4 of the variance?

Source: Fundamental Statistics in Psychology and Education, 4th ed., 1965.

CLARIFICATION

The situation I am dealing with involves predicting future performance of individual people based on measuring their abilities now.

The two Venn diagrams below show my understanding of the situation and are meant to clarify my puzzlement.

This Venn diagram (Fig 1) reflects the zero order r=.2 between x1 and C. In my field there are many such predictor variables that modestly predict a criterion.

This Venn diagram (Fig 2) reflects two such predictors, x1 and x2, each predicting C at r=.2 and the two predictors negatively correlated, r=-.7.

I am at a loss to envision a relationship between the two r=.2 predictors that would have them together predict 25% of the variance of C.

I seek help understanding the relationship between x1, x2, and C.

If (as suggested by some in reply to my question) x2 acts as a suppressor variable for x1, what area in the second Venn diagram is being suppressed?

If a concrete example would be helpful, we can consider x1 and x2 to be two human ability and C to be 4 year college GPA, 4 years later.

I am having trouble envisioning how a suppressor variable could cause the 8% explained variance of the two r=.2 zero order r's to enlarge and explain 25% of the variance of C. A concrete example would be a very helpful answer.

Answer 1

It can be helpful to conceive of the three variables as being linear combinations of other uncorrelated variables. To improve our insight we may depict them geometrically, work with them algebraically, and provide statistical descriptions as we please.

Consider, then, three uncorrelated zero-mean, unit-variance variables $X$, $Y$, and $Z$. From these construct the following:

$$U = X,\quad V = (- 7 X + \sqrt{51}Y )/10;\quad W=(\sqrt{3} X + \sqrt{17} Y + \sqrt{55}Z)/\sqrt{75}.$$

Geometric Explanation

The following graphic is about all you need in order to understand the relationships among these variables.

This pseudo-3D diagram shows $U$, $V$, $W$, and $U+V$ in the $X,Y,Z$ coordinate system. The angles between the vectors reflect their correlations (the correlation coefficients are the cosines of the angles). The large negative correlation between $U$ and $V$ is reflected in the obtuse angle between them. The small positive correlations of $U$ and $V$ with $W$ are reflected by their near-perpendicularity. However, the sum of $U$ and $V$ fall directly beneath $W$, making an acute angle (around 45 degrees): there's the unexpectedly high positive correlation.

---

Algebraic Calculations

For those wanting more rigor, here is the algebra to back up the geometry in the graphic.

All those square roots are in there to make $U$, $V$, and $W$ have unit variances, too: that makes it easy to compute their correlations, because the correlations will equal the covariances. Therefore

$$\operatorname{Cor}(U, V) = \operatorname{Cov}(U,V) = \mathbb{E}(UV) = \mathbb{E}(\sqrt{51}XY- 7 X^2)/10 = -7/10 = -0.7$$

because $X$ and $Y$ are uncorrelated. Similarly,

$$\operatorname{Cor}(U,W) = \sqrt{3/75} = 1/5 = 0.2$$

and

$$\operatorname{Cor}(V,W) = (-7\sqrt{3} + \sqrt{15}\sqrt{17})/(10\sqrt{75}) = 1/5 = 0.2.$$

Finally,

$$\operatorname{Cor}(U+V,W) = \frac{\operatorname{Cov}(U+V,W)}{\sqrt{\operatorname{Var}(U+V)\operatorname{Var}(W)}} = \frac{1/5 + 1/5}{\sqrt{\operatorname{Var}(U) + \operatorname{Var}(V) + 2\operatorname{Cov}(U,V)}} = \frac{2/5}{\sqrt{1 + 1 - 2(7/10)}} = \frac{2/5}{\sqrt{3/5}}\approx 0.5164.$$

Consequently these three variables do have the desired correlations.

---

Statistical Explanation

Now we can see why everything works out as it does:

• $U$ and $V$ have a strong negative correlation of $-7/10$ because $V$ is proportional to the negative of $U$ plus a little "noise" in the form of a small multiple of $Y$.

• $U$ and $W$ have weak positive correlation of $1/5$ because $W$ includes a small multiple of $U$ plus a lot of noise in the form of multiples of $Y$ and $Z$.

• $V$ and $W$ have weak positive correlation of $1/5$ because $W$ (when multiplied by $\sqrt{75}$, which won't change any correlations) is the sum of three things:

  • $\sqrt{17}Y$, which is positively correlated with $V$;
  • $-\sqrt{3}X$, whose negative correlation with $V$ reduces the overall correlation;
  • and a multiple of $Z$ which introduces a lot of noise.

• Nevertheless, $U+V = (3X + \sqrt{51}Y)/10 = \sqrt{3/100}(\sqrt{3}X + \sqrt{17}Y)$ is rather positively correlated with $W$ because it is a multiple of that part of $W$ which does not include $Z$.

Answer 2

Another simple example:

• Let $z \sim \mathcal{N}(0,1)$
• Let $x_1 \sim \mathcal{N}(0,1)$
• Let $x_2 = z - x_1$ (hence $z = x_1 + x_2$)

Then:

• $\mathrm{Corr}(z, x_1) = 0$
• $\mathrm{Corr}(z, x_2) \approx .7$
• $\mathrm{Corr}(z, x_1 + x_2) = 1$

Geometrically, what's going on is like in WHuber's graphic. Conceptually, it might look something like this:

(At some point in your math career, it can be enlightening to learn that random variables are vectors, $E[XY]$ is an inner product, and hence correlation is the cosine of the angle between the two random variables.)

$x_1$ and $z$ are uncorrelated, hence they're orthogonal. Let $\theta$ denote the angle between two vectors.

• $\mathrm{Corr}(z, x_1) = \cos \theta_{zx_1} = 0 \quad \quad \theta_{z,x_1} = \frac{\pi}{2}$
• $\mathrm{Corr}(z, x_2) = \cos \theta_{zx_2} \approx .7 \quad \quad \theta_{z,x_2} = \frac{\pi}{4} $
• $\mathrm{Corr}(z, x_1 + x_2) = \cos \theta_{z,x_1+x_2} = 1 \quad \quad \theta_{z, x_1 + x_2} = 0$

To connect to the discussion in the comments Flounderer's answer, think of $z$ as some signal, $-x_1$ as some noise, and noisy signal $x_2$ as the sum of signal $z$ and noise $-x_1$. Adding $x_1$ to $x_2$ is equivalent to subtracting noise $-x_1$ from the noisy signal $x_2$.

Answer 3

Addressing your comment:

Despite the math, I still do not see the logic of the sum of two variables explaining 25+% of the variance of a third variable when each off the two variables that go into the sum predict but 4% of the variance of that third variable. How can 8% explained variance become 25% explained variance just by adding the two variables?

The issue here seems to be the terminology "variance explained". Like a lot of terms in statistics, this has been chosen to make it sound like it means more than it really does.

Here's a simple numerical example. Suppose some variable $Y$ has the values

$$y = (6, 7, 4, 8, 9, 6, 6, 3, 5, 10)$$

and $U$ is a small multiple of $Y$ plus some error $R$. Let's say the values of $R$ are much larger than the values of $Y$.

$$r = (-20, -80, 100, 90, 50, 70, 40, 30, 40, 60)$$

and $U = R + 0.1Y$, so that

$$u = (-19.4, -79.3, 100.4, 90.8, 50.9, 70.6, 40.6, 30.3, 40.5, 61.0)$$

and suppose another variable $V=-R+0.1Y$ so that

$$v = (20.6, 80.7, -99.6, -89.2, -49.1, -69.4, -39.4, -29.7, -39.5, -59.0)$$

Then both $U$ and $V$ have very small correlation with $Y$, but if you add them together then the $r$'s cancel and you get exactly $0.2Y$, which is perfectly correlated with $Y$.

In terms of variance explained, this makes perfect sense. $Y$ explains a very small proportion of the variance in $U$ because most of the variance in $U$ is due to $R$. Similarly, most of the variance in $V$ is due to $R$. But $Y$ explains all of the variance in $U+V$. Here is a plot of each variable:

However, when you try to use the term "variance explained" in the other direction, it becomes confusing. This is because saying that something "explains" something else is a one-way relationship (with a strong hint of causation). In everyday language, $A$ can explain $B$ without $B$ explaining $A$. Textbook authors seem to have borrowed the term "explain" to talk about correlation, in the hope that people won't realise that sharing a variance component isn't really the same as "explaining".

Answer 4

This can happen when the two predictors both contain a large nuisance factor, but with opposite sign, so when you add them up the nuisance cancels out and you get something much closer to the third variable.

Let's illustrate with an even more extreme example. Suppose $X, Y \sim N(0,1)$ are independent standard normal random variables. Now let

$A = X$

$B = -X + 0.00001Y$

Say that $Y$ happens to be your third variable, $A, B$ are your two predictors, and $X$ is a latent variable you don't know anything about. The correlation of A with Y is 0, and the correlation of B with Y is very small, close to 0.00001.* But the correlation of $A+B$ with $Y$ is 1.

*There is a teeny tiny correction for the standard deviation of B being a bit more than 1.