0

I am fitting regression models using closely related and highly collinear critical variables.

I have used a few collinearity-reduction techniques, such as:

  • PCA,
  • variable selection.

Nevertheless, there is scope for further reduction.

Since my primary goal is to perform do inference on the variables, dimensionality reduction and variable selection are not the best options for me, because they make the variables and the models harder to interpret or even outright uninterpretable. Sadly, regularization is not an option for me given the specific type of regression model and the specific software package I am using.

Are there any other collinearity reduction methods that preserve as much information as possible while keeping the information contained in each of the critical variables distinct?

Des Grieux
  • 189
  • 11
  • How would scaling have any effect whatsoever on collinearity? – Tim Nov 21 '17 at 22:16
  • There might be people on this forum who understand the reasons better than I do. There is a big reduction in the kappa value when I scale my variables. – Des Grieux Nov 21 '17 at 22:31
  • See this post. https://stats.stackexchange.com/questions/16710/does-standardising-independent-variables-reduce-collinearity – Des Grieux Nov 21 '17 at 22:36
  • See also this post https://www.researchgate.net/post/What_is_the_difference_among_multicollinearity_covariance_confounding_variables_and_interaction_term_in_a_multiple_linear_regression_model – Des Grieux Nov 21 '17 at 22:39
  • Part of the content in the second link is also accessible here: https://www.quora.com/In-statistics-what-is-the-difference-between-collinearity-and-correlation – Des Grieux Nov 21 '17 at 22:45
  • If you read the article carefully you will see that the answer is more nuanced. High collinearity implies high correlation, but high correlation does not necessarily imply high collinearity. – Des Grieux Nov 21 '17 at 22:46
  • In your comment you only talk about scaling. I think the answer might be to compare the effect of manipulating lambda in the collinearity and scaling equations. I grant you that this is a controversial topic but I don't think this is an appropriate forum for discussing this because my question is not directly related to scaling. I have edited my question and removed scaling to make it uncontroversial. – Des Grieux Nov 21 '17 at 22:56
  • You may set $\lambda_0,\lambda_1,\mu,\sigma$ to *whatever* you want and it won't have any effect on collinearity. I agree that this is not a place for discussing this, yet I wanted to correct your misconception. – Tim Nov 21 '17 at 22:59
  • 1
    I moved what I said in the comments to the answer I gave for the thread you linked, see here for more details on standardization and collinearity/correlation: https://stats.stackexchange.com/questions/16710/does-standardising-independent-variables-reduce-collinearity/315089#315089 – Tim Nov 23 '17 at 10:49

1 Answers1

1

I am not sure what you mean by "collinearity reduction". Scaling will not affect it. If you want to keep your original variables, then the usual methods of dealing with collinearity are to used some form of biased regression.

Ridge regression is, I think, the most usual tool for dealing with colinearity while keeping the original variables. It doesn't lower collinearity, per se (nothing can do that and keep the original variables) but it reduces the inflation of variance of the estimators that is one of the main negative consequences of collinearity.

Another option, of course, is to drop some variables. If the collinearity consists of a relation between two variables, then you could drop either. If it consists in one variable being close to a linear combination of others, then you might want to drop the single variable.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • The problem with coefficient regularization methods such as Lasso and Ridge regression is that currently there are no R packages that implement Lasso and Ridge penalties into mixed-effects regression with more than 1 random effect. – Des Grieux Nov 22 '17 at 02:50
  • By collinearity reduction methods I am referring to techniques that distribute and extract information that is (to the extent that this is plausible given the data) uniquely explained by a given variable. – Des Grieux Nov 22 '17 at 02:53
  • In that case, you will have to drop some variables. – Peter Flom Nov 22 '17 at 13:07