Refers to techniques for reducing a large number of variables or dimensions spanned by data to a smaller number of dimensions while preserving as much information about the data as possible. Prominent methods include PCA, MDS, Isomap, etc. The two main subclasses of techniques: feature extraction and feature selection.
Questions tagged [dimensionality-reduction]
993 questions
516
votes
3 answers
Relationship between SVD and PCA. How to use SVD to perform PCA?
Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix $\mathbf X$. How does it work? What is the…
amoeba
- 93,463
- 28
- 275
- 317
156
votes
1 answer
How to reverse PCA and reconstruct original variables from several principal components?
Principal component analysis (PCA) can be used for dimensionality reduction. After such dimensionality reduction is performed, how can one approximately reconstruct the original variables/features from a small number of principal…
amoeba
- 93,463
- 28
- 275
- 317
118
votes
4 answers
PCA and proportion of variance explained
In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a precise mathematical definition of what "variance…
user9097
- 2,973
- 7
- 18
- 11
103
votes
2 answers
Why do we need to normalize data before principal component analysis (PCA)?
I'm doing principal component analysis on my dataset and my professor told me that I should normalize the data before doing the analysis. Why?
What would happen If I did PCA without normalization?
Why do we normalize data in general?
Could…
jjepsuomi
- 5,207
- 11
- 34
- 47
103
votes
11 answers
Explain "Curse of dimensionality" to a child
I heard many times about curse of dimensionality, but somehow I'm still unable to grasp the idea, it's all foggy.
Can anyone explain this in the most intuitive way, as you would explain it to a child, so that I (and the others confused as I am)…
Kobe-Wan Kenobi
- 2,437
- 3
- 20
- 33
74
votes
2 answers
What is the difference between ZCA whitening and PCA whitening?
I am confused about ZCA whitening and normal whitening (which is obtained by dividing principal components by the square roots of PCA eigenvalues). As far as I know,
$$\mathbf x_\mathrm{ZCAwhite} = \mathbf U \mathbf x_\mathrm{PCAwhite},$$ where…
RockTheStar
- 11,277
- 31
- 63
- 89
46
votes
4 answers
Would PCA work for boolean (binary) data types?
I want to reduce the dimensionality of higher order systems and capture most of the covariance on a preferably 2 dimensional or 1 dimensional field. I understand this can be done via principal component analysis, and I have used PCA in many…
Alvin Nunez
- 647
- 1
- 7
- 8
41
votes
3 answers
Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?
In a recent assignment, we were told to use PCA on the MNIST digits to reduce the dimensions from 64 (8 x 8 images) to 2. We then had to cluster the digits using a Gaussian Mixture Model. PCA using only 2 principal components does not yield distinct…
willk
- 583
- 1
- 7
- 12
39
votes
1 answer
What is the intuitive reason behind doing rotations in Factor Analysis/PCA & how to select appropriate rotation?
My Questions
What is the intuitive reason behind doing rotations of factors in factor analysis (or components in PCA)?
My understanding is, if variables are almost equally loaded in the top components (or factors) then obviously it is difficult to…
GeorgeOfTheRF
- 5,063
- 14
- 42
- 51
37
votes
2 answers
When is t-SNE misleading?
Quoting from one of the authors:
t-Distributed Stochastic Neighbor Embedding (t-SNE) is a (prize-winning) technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.
So it sounds…
Lyndon White
- 2,744
- 1
- 19
- 35
36
votes
1 answer
How would PCA help with a k-means clustering analysis?
Background: I want to classify the residential areas of a city into groups based on their social-economic characteristics, including housing unit density, population density, green space area, housing price, number of schools / health centers / day…
enaJ
- 567
- 1
- 6
- 11
33
votes
1 answer
Dimensionality reduction (SVD or PCA) on a large, sparse matrix
/edit: Further follow up now you can use irlba::prcomp_irlba
/edit: following up on my own post. irlba now has "center" and "scale" arguments, which let you use it to calculate principle components, e.g:
pc <- M %*% irlba(M, nv=5, nu=0,…
Zach
- 22,308
- 18
- 114
- 158
29
votes
4 answers
How to do dimensionality reduction in R
I have a matrix where a(i,j) tells me how many times individual i viewed page j. There are 27K individuals and 95K pages. I would like to have a handful of "dimensions" or "aspects" in the space of pages which would correspond to sets of pages which…
laramichaels
- 1,119
- 3
- 12
- 12
29
votes
2 answers
What is "reduced-rank regression" all about?
I have been reading The Elements of Statistical Learning and I could not understand what Section 3.7 "Multiple outcome shrinkage and selection" is all about. It talks about RRR (reduced-rank regression), and I can only understand that the premise is…
cgo
- 7,445
- 10
- 42
- 61
29
votes
5 answers
How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?
Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$?
I understand that from dimensionality-reduction/feature-selection point of view, if $v_1, v_2,…
CuriousMind
- 2,133
- 5
- 24
- 32