8

Can anyone kindly give me some information about the statement (last sentence) at the end of below definition. What does it mean by "It can be used when a correlation matrix is singular"? This quote is from SPSS help menu on factor analysis.

Principal Components Analysis. A factor extraction method used to form uncorrelated linear combinations of the observed variables. The first component has maximum variance. Successive components explain progressively smaller portions of the variance and are all uncorrelated with each other. Principal components analysis is used to obtain the initial factor solution. It can be used when a correlation matrix is singular.

I already know that different from other factor extraction methods, PCA uses total variance in data while extracting factors. Is it somehow related to this?

amoeba
  • 93,463
  • 28
  • 275
  • 317
kzmlbyrk
  • 183
  • 1
  • 4

2 Answers2

8

The citation and its last sentence says of the following.

Singular matrix is a one where rows or columns are linearly interdependent. Most of factor analysis extraction methods require that the analyzed correlation or covariance matrix be nonsingular. It must be strictly positive definite. The reasons for it is that at various stages of the analysis (preliminary, extraction, scores) factor analysis algorithm addresses true inverse of the matrix or needs its determinant. Minimal residuals (minres) method can work with singular matrix at extraction, but it is absent in SPSS.

PCA is not iterative and is not true factor analysis. Its extraction phase is single eigen-decomposition of the intact correlation matrix, which doesn't require the matrix to be full rank. Whenever it is not, one or several last eigenvalues turn out to be exactly zero rather than being small positive. Zero eigenvalue means that the corresponding dimension (component) has variance 0 and therefore does not exist. That's all; it doesn't hamper the extraction, nor it precludes computing component scores. So, you can do PCA with singular, multicollinear data. Sometimes PCA is used specifically for the purpose to get rid of the multicollinearity.

Community
  • 1
ttnphns
  • 51,648
  • 40
  • 253
  • 462
  • 1
    +1. But can't "principal factor" extraction method (iterating updates of uniquenesses $\Psi$ and updates of loadings via PCA of reduced covariance matrix $C-\Psi$) be applied to low-rank covariance matrices as well? I don't see anything in this simple iterative procedure that would fail when $C$ is low rank. – amoeba Mar 23 '15 at 14:54
  • @amoeba, the OP did not ask about FA. In regard to your comment, I might notice two things: (1) we cannot use images as (the best and default) initial communality estimates because we cannot compute them from a singular matrix. (2) Moreover, multicollinearities imply some partial correlations approaching 1. But common model FA assumption is low or negligible partial correlations. So, you can try to do it as you said, but it won't be a correct FA. I expect (I haven't check it) that you are doomed to encounter Heywood case on nearest iterations. – ttnphns Mar 23 '15 at 18:46
1

It's related to the mathematical property of the correlation matrix. In simple words, PCA is essentially an orthogonal transformation of the given correlation matrix, which can be either invertible or singular (I.e. doesn't have an inverse matrix).

fred
  • 125
  • 6