1

I have a correlation matrix where by for a good chunk of the data set the correlations for all pairs of variables are more correlated than +/-0.5 Infact many of these are more correlated than +/-0.75

The program i have reports the correlation matrix is not positive definite. Therefore principle components (although generated) seem to be collinear when examined in regression

I have provided a sample of the correlation matrix

http://imgur.com/44ddFIW

Samuel
  • 121
  • 10
  • 1
    The collinearity is not just "seeming;" it is real. I see some "1.0000" values on the off-diagonals and some other numbers very close to that, so at least to this level of precision you have perfectly redundant data. Any algorithm to process this matrix will produce eigenvalues extremely close to zero; any machine-precision algorithm will be as likely to represent such near-zero values as tiny negative numbers as tiny positive numbers. – whuber Feb 18 '14 at 17:35
  • if i remove the extremely high values (I can justifiably remove these) i am still left with a matrix that is almost entirely above 0.5 correlation, the NPD report is still apparent. This is fine if i can just use the initial PCs – Samuel Feb 20 '14 at 15:27
  • 1
    That only means there is still real collinearity. Not all collinearity is as evident as the presence of ones in the off-diagonal entries. Anyway, what is your question? There's nothing unusual or wrong with collinear independent variables in regression problems. What exactly do you mean, then, by "proceed"? – whuber Feb 20 '14 at 15:32
  • 1
    I am really attempting to try and use PCA to reduce the number of variables i use to model my dependent response variable by using a number of methods from linear to neural nets. My other option really is to remove variables by judgement. – Samuel Feb 20 '14 at 16:50
  • 1
    You might be very interested, then, in the recent answer at http://stats.stackexchange.com/a/87231, which discusses exactly this issue. It points out that because PCA on the independent variables *has nothing to do with how they might be related to the dependent variable,* you risk throwing out important variables. Your case is probably not quite that bad, because some of the near-zero eigenvalues really are just "noise" due to floating point rounding error. But after you identify and remove those components you will need to think hard about whether to remove any others. – whuber Feb 20 '14 at 17:10
  • 1
    Thanks for the link. Yes the lack of relationship to the dependent var is a problem. I have seen the same effect as in the references. where, say, PC 8 of 10 may be the most significant predictor out of all PCs. The current data has PCs with eigenvalues ~0 have no loadings of more than 0.1-0.2 in them. Therefore i thought it safe to disregard these. Often the more PCs i generate the easier they are to interpret where one or two variables may load highly onto one PC. To sum up my current thought track..does it seem acceptable therefore to pick those with high var loadings? – Samuel Feb 21 '14 at 10:30

1 Answers1

1

The heading of your question says the matrix is not invertible; in the text you state that the program you use reports it is not positive definite. It is not the same problem, although both may co-exist.

If that is the case --both problems at the same time-- I would think that because of ill-conditioning of your matrix the last (or last few) eigenvalues are within machine error of zero, perhaps with negative value. This does not preclude the use of principal components (PC): the only problem is that you will only have as many uncorrelated PC's as you have non-zero eigenvalues. If you keep only this many PC's you can regress on them without problems.

F. Tusell
  • 7,733
  • 19
  • 34
  • I will try this..i have edited the title according to your comments – Samuel Feb 18 '14 at 14:32
  • So 13 components in this case were generated with eigenvalue above zero (16 components in total). Even with the problems i can use 13 or less and my results will be valid? – Samuel Feb 18 '14 at 15:03
  • Yes. You may want to investigate why you have three exact or nearly exact linear dependences among your regressors. That may point to redundand variables in your design, as @whuber points in his comment. – F. Tusell Feb 19 '14 at 08:14
  • With those removed (they are redundant) the extreme correlation is still causing a problem. After trying what you suggested i seem to have no collinearity. Also the last few (out of the 13) do not have any loadings from the original variables to any significant degree. I was not aware that NPD matrix result still allowed utilization of resulting factors – Samuel Feb 20 '14 at 15:34
  • Are you sure you are picking the scores of the 13 PC's associated with non-zero eigenvalues? These should be orthogonal by construction. One check: if you got the eigenvalues from the correlation matrix, are you centering and standardizing the data when you compute the PC's scores? – F. Tusell Feb 20 '14 at 15:47
  • I use a number not larger than 13 (if 13 are non 0) . I tend to make a judgement call on how many to use based on where using more adds little to the variation explained – Samuel Feb 20 '14 at 16:54
  • Well, check they are correctly computed. As I said, even with 13, they should be orthogonal by construction. – F. Tusell Feb 20 '14 at 16:56
  • the are being centered and standardized, and appear orthogonal..VIFs for the PCsselected when regressed appear as less than 2 – Samuel Feb 20 '14 at 17:00
  • Then, what is the problem? If they are orhogonal, no more multicollinearity to fear. – F. Tusell Feb 20 '14 at 17:01