24

When working with many input variables, we are often concerned about multicollinearity. There are a number of measures of multicollinearity that are used to detect, think about, and / or communicate multicollinearity. Some common recommendations are:

  1. The multiple $R^2_j$ for a particular variable

  2. The tolerance, $1-R^2_j$, for a particular variable

  3. The variance inflation factor, $\text{VIF}=\frac{1}{\text{tolerance}}$, for a particular variable

  4. The condition number of the design matrix as a whole:

    $$\sqrt{\frac{\text{max(eigenvalue(X'X))}}{\text{min(eigenvalue(X'X))}}}$$

(There are some other options discussed in the Wikipedia article, and here on SO in the context of R.)

The fact that the first three are a perfect function of each other suggests that the only possible net advantage between them would be psychological. On the other hand, the first three allow you to examine variables individually, which might be an advantage, but I have heard that the condition number method is considered best.

  • Is this true? Best for what?
  • Is the condition number a perfect function of the $R^2_j$'s? (I would think it would be.)
  • Do people find that one of them is easiest to explain? (I've never tried to explain these numbers outside of class, I just give a loose, qualitative description of multicollinearity.)
Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • I have posted a related follow up question, with answers to that supplementing what is already on here: http://stats.stackexchange.com/questions/173665/what-are-the-merits-of-different-approaches-to-detecting-collinearity – kyrenia Sep 22 '15 at 21:45

1 Answers1

17

Back in the late 1990s, I did my dissertation on collinearity.

My conclusion was that condition indexes were best.

The main reason was that, rather than look at individual variables, it lets you look at sets of variables. Since collinearity is a function of sets of variables, this is a good thing.

Also, the results of my Monte Carlo study showed better sensitivity to problematic collinearity, but I have long ago forgotten the details.

On the other hand, it is probably the hardest to explain. Lots of people know what $R^2$ is. Only a small subset of those people have heard of eigenvalues. However, when I have used condition indexes as a diagnostic tool, I have never been asked for an explanation.

For much more on this, check out books by David Belsley. Or, if you really want to, you can get my dissertation Multicollinearity diagnostics for multiple regression: A Monte Carlo study

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • 1
    So is the idea here that looking at VIFs, you might mistakenly conclude that multicollinearity isn't a problem, but if you had looked at the condition number, you would have been more likely to draw the right conclusion? Perhaps something like a test w/ greater statistical power? – gung - Reinstate Monica Jan 29 '13 at 23:17
  • 4
    +1. Fortunately, for explaining the condition number we already have [an outstanding thread](http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues) on this site: it's the maximum distortion found in the second-order description of the design variables as a point cloud. The greater the distortion, the more the points tend to lie within a subspace. This geometric insight also shows why the conditioning of a centered design matrix is better than that of the raw design matrix itself. – whuber Jan 29 '13 at 23:18
  • 1
    Well, it's hard to define exactly what the "right" conclusion is; but it should have something to do with small changes in the data producing large changes in the output. As I recall, condition indexes were more directly related to this. But the big thing was getting the variance proportions, which let you see sets of variables and the degree of their collinearity. (Of course, all that was 14 years ago.... but I don't think things have changed. The measures are the same. But my memory may not be perfect). – Peter Flom Jan 29 '13 at 23:20
  • 3
    Gung, one key point here is that the condition number is independent of coordinates: it remains unchanged under (orthogonal) linear recombinations of the data. Thus it cannot possibly express anything about individual variables but it must capture a property of the entire collection. Using it thereby partially insulates you from being misled by how your variables happen to be expressed. – whuber Jan 29 '13 at 23:21
  • @whuber Nice finding that thread! I will have to take a closer look later. – Peter Flom Jan 29 '13 at 23:22
  • @whuber, if you have time at some point & want to contribute an answer, it might be nice to elaborate on what "it thereby partially insulates you from being misled by how your variables happen to be expressed" means for this issue. – gung - Reinstate Monica Jan 29 '13 at 23:28
  • 1
    I've been too swamped to finish your dissertation yet, but it's been really helpful thus far. Thanks again. – gung - Reinstate Monica Feb 01 '13 at 21:02