Different coefficient values from multiple vs. bivariate regression under orthogonality

Question

I wonder how to generate such data, so that in single variable regression feature coefficient would be positive, and in multiple regression would be negative.

So I read several related questions on Cross Validated. As I understood, there are 2 possible cases for this desired situation: when features are correlated (collinearity) or when feature values are not orthogonal.

---

So I made a code that generates data for collinearity case:

set.seed(1)
xPositive = rnorm(100)
xNegative = xPositive / 3 + rnorm(100, 0, .15)
y = 5 * xPositive - 3 * xNegative + rnorm(100)
fitXNegative = glm(y~xNegative)
fixBothX = glm(y~xNegative+xPositive)
print(paste("Coefficient for xNegative when fitting only xNegative:", coef(fitXNegative)['xNegative']))
print(paste("Coefficient for xNegative when fitting both x'es:", coef(fixBothX)['xNegative']))
print(paste("That is because of correlation between x'es:", cor(xNegative, xPositive)))

The output is:

Coefficient for xNegative when fitting only xNegative: 9.57682149821626
Coefficient for xNegative when fitting both x'es: -2.85321980961991
That is because of correlation between x'es: 0.911618247307253

So the example works.

---

Similarly, I was trying to come up with non-orthogonality example. Here is my best try so far:

set.seed(1)
xPositive = runif(100, 100, 120)
xNegative = runif(100, 100, 120)
y = 13 * xPositive - 11 * xNegative
fitXNegative = glm(y~0+xNegative)
fitBothX = glm(y~0+xNegative+xPositive)
print(paste("Coefficient for xNegative when fitting only xNegative:", coef(fitXNegative)['xNegative']))
print(paste("Coefficient for xNegative when fitting both x'es:", coef(fixBothX)['xNegative']))

The output is:

Coefficient for xNegative when fitting only xNegative: 1.97024384743124
Coefficient for xNegative when fitting both x'es: -2.85321980961991

But as you may have noticed, I manually excluded the intercept from the model, because otherwise the coefficient doesn't change its sign. Also, if you center the data in this example:

xPositive = xPositive - 110
xNegative = xNegative - 110

then you effectively get rid of non-orhogonality and the coefficient then doesn't change its sign.

---

So, regarding this second non-orthogonality example I have 2 questions:

1. I would like to come up with a better mechanism of data generation so that I could observe the effect of changing coefficient sign without the necessity of setting the intercept to zero.

2. I am checking for non-orthogonality between features using the following formula:

   nonOrthogonalityMeasure = mean(xPositive * xNegative)

If that value approximately equals zero then I conclude that features are orthogonal. Is this a correct method?

3. Are there other situations when the coefficient may change its sign, except for the aforementioned situations?

Answer 1

Uncorrelated:

• In population: $\text{Cov}=\mathbb{E}XY-\mathbb{E}X \mathbb{E}Y=0$
• In sample: $\widehat{\text{Cov}}=\frac{1}{n-1}\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})=0$

Orthogonal:

• In population: $\text{M}_{XY}=\mathbb{E}XY=0$
• In sample: $\hat{\text{M}}_{XY}=\frac{1}{n}\sum_{i=1}^n x_i y_i=0$

The two coincide in population when $\mathbb{E}X=0$ and/or $\mathbb{E}Y=0$; they coincide in sample when $\bar{x}=0$ and/or $\bar{y}=0$.

The relevant feature for your question is correlation rather than orthogonality. When regressors are correlated (have non-zero correlation), their estimated coefficients in a multiple regression do not match their estimated coefficients from bivariate regressions; when they are uncorrelated, the coefficients match.

You may notice that centering the data by subtracting their sample means effectively turns an orthogonal pair of vectors to an uncorrelated pair of vectors; the non-central cross moment $\hat{\text{M}}_{XY}=0$ turns into the central cross moment $\widehat{\text{Cov}}=0$.

Regarding Question 1, here is a tip for generating orthogonal variables: running a regression of $y$ on $x$ will generate a residual $\varepsilon$ that is uncorrelated and also orthogonal (since it has a mean of zero) to $x$.

Regarding Question 2, the in-sample orthogonality measure defined above suggests that your approach is sensible (if I interpret correctly that xPositive and xNegative are two vectors and you want to check whether they are orthogonal).