Correlations between explanatory variables in regression

Question

I'm just starting to learn about linear regression models and time series analysis and came upon the following doubt.

Suppose we have a variable $Y$ that we're trying to model using $p$ explanatory variables $X_i$, maybe using a simple linear model such as:

$$Y = a_0 + a_1X_1 +...+a_pX_p$$

My question is the following: what happens if the explanatory variables are correlated with each other. In the assumptions for this model, I see no mention of this fact, but clearly there has to be some kind of qualitative difference depending on the degree to which the variables are correlated with each other. For example if $X_1 = X_3 ^2$, then what is the point of including both $X_1$ and $X_3$ in the model, intuitively it seems like the other one carries no additional 'information'?

Perhaps depending on how correlated the variables are one would use a different model or approach?

Answer 1

If $$Y = a_0 + a_1X_1 + a_2X_2 +a_3X_3 +...+a_pX_p\;\&$$ $$X_1=X_3^2$$ then eliminate $X_1$ as it is unnecessary and regress $$Y = a_0 +a_3X_3 + a_1X_3^2 + a_2X_2 +...+a_pX_p$$ However, the example used for the question of how to treat correlated variables was not relevant as the answer to the example is based on the assumption of a perfect correlation, i.e., an equality, that lets us eliminate $X_1$ entirely as perfectly redundant.

Covariates in the analysis of covariance context, i.e., as per the ANCOVA procedure is as follows, assuming that a linear relationship between the response (DV) and covariate (CV) exists: $$y_{ij} = \mu + \tau_i + \beta (x_{ij} - \overline{x_i}) + \epsilon_{ij}.$$

In this equation, the DV, $y_{ij}$ is the $j^{th}$ observation under the $i^{th}$ categorical group; the CV, $x_{ij}$ is the $j^{th}$ observation of the covariate under the $i^{th}$ group. Variables in the model that are derived from the observed data are $\mu$ (the grand mean) and $\overline{x_i}$ (the $i^{th}$ group mean). The variables to be fitted are $\tau_i$ (the effect of the $i^{th}$ level of the IV), $\beta$ (the slope of the line) and $\epsilon_{ij}$ (the associated unobserved error term for the $j^{th}$ observation in the $i^{th}$ group).

This answer is only partial in the sense that the variations on the theme are numerous, and each circumstance requires special treatment.

Answer 2

Regressions are pretty good at handling correlations between predictors. In fact that is WHY we use multiple regression instead of doing many many simple correlations.

However. If the correlation is REALLY high, you get something called collinearity. Which means that the two predictors are so correlated that they are essentially the same thing. A real life example of collinearity might be something like waist circumference in inches and jean-size. These are so similar to each other that they are basically measuring the same thing (but not quite).

In the case of collinearity it's preferable to remove one of them. Usually you remove the one with less theoretical reasons to be included.

Answer 3

In the linear regression, it's preferable to remove correlated variables, otherwise your model would have a very high variance. adding by the correlated variable (X3 in your exemple) will result of opposite estimates forcing your predictions to highly vary : the absolute value of the parameters a1 and a3 would be very close but the signs of these two paramter would be opposite.

Many ways are possible to remove these correlation :

1. Keep the variable having the most meaning (in the studied subject)
2. Create a third variable : which represent a linear combination of the two first variables ...