Creating Correlations

Question

I have three variables, A, B, and C. It seems obvious that if X = A-B and Y = C-B that there should be a correlation between X and Y. When I've done this in matlab with random numbers, this seems to be the case with a mean r of about 0.5. The fact that there is a correlation makes intuitive since because X and Y are sharing the variance in B; but when I've tried to figure out why it's 0.5, I've had less luck. Any hints as to why this is the case would be greatly appreciated.

Answer 1

So calculating this by hand we have the following (I will assume independence of $A,B$ and $C$):

\begin{align*} \text{Cor}(X,Y)&=\frac{\text{Cov}(X,Y)}{\sqrt{\text{Var(X)}\text{Var}(Y)}}\\ &\\ &=\frac{\text{Cov}(A-B,C-B)}{\sqrt{\text{Var}(A-B)\text{Var}(C-B)}}\\ &\\ &=\frac{E[(A-B)(C-B)]-E[A-B]E[C-B]}{\sqrt{[\text{Var}(A)+\text{Var}(B)]\times[\text{Var}(C)+\text{Var}(B)]}}\\ &\\ &=\frac{E[AC-AB -BC +B^2]-(E[A]-E[B])(E[C]-E[B])}{\sqrt{[\text{Var}(A)+\text{Var}(B)]\times[\text{Var}(C)+\text{Var}(B)]}}\\ &\\ &=\frac{E[AC]-E[AB] -E[BC] +E[B^2]-(E[A]-E[B])(E[C]-E[B])}{\sqrt{[\text{Var}(A)+\text{Var}(B)]\times[\text{Var}(C)+\text{Var}(B)]}}\\ &\\ &=\frac{E[A]E[C]-E[A]E[B] -E[B]E[C] +E[B^2]-(E[A]-E[B])(E[C]-E[B])}{\sqrt{[\text{Var}(A)+\text{Var}(B)]\times[\text{Var}(C)+\text{Var}(B)]}}\\ &\\ &=\frac{E[B^2]-(E[B])^2}{\sqrt{[\text{Var}(A)+\text{Var}(B)]\times[\text{Var}(C)+\text{Var}(B)]}}\\ &\\ &=\frac{\text{Var(B)}}{\sqrt{[\text{Var}(A)+\text{Var}(B)]\times[\text{Var}(C)+\text{Var}(B)]}}\\ &=\frac{1}{\sqrt{2\times2}}\\ &=\frac{1}{2} \end{align*}

Answer 2

It will depend on what the variables are. E.g. (R code, should be pretty clear though)

set.seed(1234)
A <- rnorm(100)
B <- rnorm(100)
C <- rnorm(100)

cor(A-C, B-C) #0.40

A <- runif(100)
B <- runif(100)
C <- runif(100)

cor(A-C, B-C)  #0.54

A <- rnorm(100,1,10)
B <- rnorm(100,10,100)
C <- rnorm(100,1,200)

cor(A-C, B-C) #0.89

And, of course, if A and B are related, you would get other values

Answer 3

Essentially what happens is that you have two sources of variance in your subtracted score and one of those sources is shared between each of the subtracted scores. When the initial random variables of A, B, and C are all independent but you then add in the variance from B to both A, and C then the proportion of variance that's shared is going to be your correlation. You had equal variances in each condition, that's why it's 0.5, half of the variance is shared.

Note that, if the correlation between A and B is 0 then the var(A-B) == var(A) + var(B). All of the following equations show similar results (I'll get to why not exact later).

a <- rnorm(10000, 0, 1)
b <- rnorm(10000, 0, 1)
var(a)
var(b)
var(a) + var(b)
var(a + b)
var(a - b)

So, your X, and Y variables you created include the variance from B. Correlations are about shared variance so your intuitions are correct.

I imagine that you've already done something like the following but perhaps through brute force. I'm using R and the mvrnorm function because it allows me to set my initial correlations to 0.

library(MASS) #so I can use mvrnorm and insure 0 correlation

# The following is a covariance matrix with the variance of each condition
# on the diagonal and the covariances among the conditions of of it.
sigma <- matrix(c(1.0, 0.0,  0.0,
                  0.0, 1.0,  0.0,
                  0.0, 0.0,  1.0), 3, byrow = TRUE)
mat <- mvrnorm(100, c(0,0,0), sigma, empirical = TRUE)
cor(mat)
a <- mat[,1]
b <- mat[,2]
c <- mat[,3]

x <- a - b
y <- c - b

cor(x, y)

You can see the final correlation of x and y is 0.5, just as you found. You can predict in advance what would happen if you changed the variances of the conditions (keep in mind variance is what matters here, not standard deviation). Let's say the variance of b was raised from 1 to 2. Now, the proportion of variance that comes from b in x is going to be var(b) / (var(a) + var(b)) or 2/3. That's going to be the same for y so the geometric mean shared variance is going to be equal to both of those and the correlation is 0.66 (2/3). To generate the data you'd just change the original variance-covariance matrix I made above and then proceed as above.

# note the variance of the second condition is now 2
sigma <- matrix(c(1.0, 0.0,  0.0,
                  0.0, 2.0,  0.0,
                  0.0, 0.0,  1.0), 3, byrow = TRUE)
mat <- mvrnorm(100, c(0,0,0), sigma, empirical = TRUE)
....

And, indeed, what happens is you get a correlation of 0.66. So, it's easy to work out what the correlation should be based on the initial variances. If you've got correlations among a, b, and c, and you almost always do, then it becomes a bit trickier. I imagine you've come across this in your simulations when you said that what you found was approximately 0.5. Sometimes your random variables happened to be correlated. If I take my current variables that are uncorrelated then the following will all produce the same results.

var(a) + var(b)
var(a - b)
var(a + b)

They all show the sum of the variances. But the equations are different if there is a correlation. The first one should be...

var(a) + var(b) - 2*cov(a,b)

And it would be the same as var(a) + var(b) when the correlation is 0. It also comes out the same as var(a - b). As an aside, you actually have to add the covariance term instead of the subtracting to match var(a + b).

So, hopefully that explains it in enough detail that you could derive the expected correlation of X and Y based on the covariances among A, B, and C, and their respective variances.

There's a graphical explanation using geometry that's very pretty but it escapes me right now. Also, this is excellent stuff to understand if you ever want to get a grasp on multiple regression (or even unbalanced ANOVAs).

Answer 4

Nope. Let $B=C$, let $A=B$, or let $C=A$.