3

What is the relationship between SVD and factor analysis? How can use singular values and other matrices from SVD to perform factor analysis or cluster document-term matrix without using other clustering techniques?

amoeba
  • 93,463
  • 28
  • 275
  • 317
dan24
  • 71
  • 1
  • 4
  • Welcome to CV. This question has been addressed in a number of prior threads. For instance, if you drill into the link to your question, in the right hand column are a large number of related threads. – Mike Hunter Mar 06 '16 at 18:30
  • 1
    Thanks DJohnson. I went through those questions before posting this. I couldn't find an answer to my questions from those. – dan24 Mar 06 '16 at 18:41
  • 2
    Are you sure? This one looks almost redundant with your question... http://stats.stackexchange.com/questions/134282/relationship-between-svd-and-pca-how-to-use-svd-to-perform-pca?rq=1 – Mike Hunter Mar 06 '16 at 18:49
  • 1
    May be first I need to clarify the relationship between PCA and FA. From what I know, PCA and FA are different. PAC helps to combine instances into components to reduce dimension. But FA, finds related features (of the instances) and extract latent factors. Isn't it correct? Thanks – dan24 Mar 06 '16 at 18:56
  • Dude, there are more than a few threads that speak to that question...poke around. – Mike Hunter Mar 06 '16 at 19:06
  • 2
    If you are specifically about FA, not PCA. Some methods of FA have to deal with not positive definite _correlation_ matrix on iterations; eigendecomposition can detect negative eigenvalues while svd cannot - therefore usually the former is preferred to the latter ([Footnote 1](http://stats.stackexchange.com/a/50758/3277)). As for svd of rectangular _data_ matrix - it is not needed and not done in FA. – ttnphns Mar 06 '16 at 19:28
  • `factor analysis or cluster document-term matrix without using other clustering techniques` - that is unclear. What is your interest ultemately - FA or Cluster analysis? – ttnphns Mar 06 '16 at 19:30
  • ttnphns I want to find factors combining related terms in the data matrix. – dan24 Mar 06 '16 at 19:35
  • @ttnphns: What do you mean by "eigendecomposition can detect negative eigenvalues while SVD cannot"? I don't understand this. – amoeba Mar 06 '16 at 20:57
  • @amoeba, if a square symm. matrix is p.s.d. its eigen- and svd- decompositions return the same result, in out current interest - the eigenvalues, which are all nonnegative. But consider a negative s.d. matrix, such as any from [here](http://stats.stackexchange.com/a/69206/3277) examples or - let us - this one: `(10) 9 (15) -3 6 (6) 18 12 -7 (23) -7 5 8 -4 (8)` [lower triangle shown, diag. entries are parenth.). [to cont.] – ttnphns Mar 06 '16 at 21:38
  • [cont.] Eigenvalues of that matrix: `44.7361 22.5044 2.3908 -2.1720 -5.4592`. But sing. values are: `44.7361 22.5044 5.4592 2.3908 2.1720`. You see that svd fails to recognize the negative eigenvalues and takes (and sorts) them as if they were positive. – ttnphns Mar 06 '16 at 21:39
  • [cont.] that is why eigendecomposition is preferable in cases of non p.s.d. matrices. You _see_ negative eigenvalues and therefore are able to manage them differently in some way from positive ones. In many analyses fitting will win. For example, Torgerson's MDS (PCoA) based on eigen() is usually better reconstruction of (noneuclidean) distaces than when based on svd(). – ttnphns Mar 06 '16 at 21:46
  • @ttnphns: Thanks a lot, this was a helpful. I've never thought about this difference before. (Related math.SE thread: http://math.stackexchange.com/questions/28036.) – amoeba Mar 06 '16 at 22:03

3 Answers3

2

Google brought me here, and I dislike how the comments just assume everyone knows that FA and PCA are related. So to answer your question: yes. See Tipping and Bishop, 1996. This paper is great because:

  1. It discusses the connection between FA and PCA (Section 2.2)
  2. It discusses using the SVD to compute the ML parameters (Appendix A)
jds
  • 1,402
  • 1
  • 13
  • 24
1

Google brought me here too, and I found that the implementation of Scikit-learn library, a famous repository for data science in Python, uses SVDs with a small tweak to fit the data points and perform factor analysis.

Hence the answer is a big YES you can use SVD.

If you're keen with code implementation, I suggest you can read the Factor Analysis source code of Scikit-learn here at github. They implement the SVD algorithm using Scipy library and tweak the output for shape adjustment.

In addition to that, I want to add some reference on top of Probabilistic Principal Component Analysis paper PPCA paper suggested by @gwg:

  • David Barber, Bayesian Reasoning and Machine Learning, Algorithm 21.1 (textbook) textbook here
  • Christopher M. Bishop: Pattern Recognition and Machine Learning, Chapter 12.2.4 pdf here (paper, same author of PPCA). Scikit-learn referenced this paper for their alogrithm
Daniel Kurniadi
  • 11
  • 2
0

Idea, in some cases (aka objective functions), Factor Analysis problem will be reformulated as Low Rank Matrix Approximation Problem. And, SVD can be used to solved Low Rank Matrix Approximation Problem.

Benjaminliupenrose
  • 11
  • 1