13

I am a beginner in PCA, I am trying to apply it on a dataset I have. The features are different geometrical parameters with different units and variability, I do standardize the features matrix by subtracting the mean and dividing by the standard deviation.

For the PCA, I use the PCA() method in sklearn wich is based on the singular value decomposition (SVD). Once I fit the model I get the following results if I choose just 4 components:

Is this telling me something about my features? or there is something fundamentally wrong in my approach? Thank you!

cfd_aero
  • 133
  • 7

3 Answers3

17

Results suggest that your features are mutually orthogonal. Accounting for total variance means accounting for both variance and covariance. Orthogonality limits covariance. Standardization equates variance across features. Put together, each feature makes a roughly equal contribution to total variance, as do your components.

Ed Rigdon
  • 1,341
  • 4
  • 11
  • 1
    Thank you for the answer, the thing is I don't think all the features are mutually orthogonal as I know for sure that some have no impact on the target therefore I don't trust the current result I have, would the way the normalization is done impact the result of the PCA? – cfd_aero Feb 27 '20 at 19:03
  • 2
    Resolve this issue by computing a correlation matrix among the features. Are the correlations large or small, and are there any visually discernable patterns, or not? – Ed Rigdon Feb 27 '20 at 20:12
  • 6
    @cfd_aero: It doesn't matter what effect the features have on the target. The definition of orthogonality doesn't involve a target at all. – user2357112 supports Monica Feb 28 '20 at 02:38
  • 7
    Another important thing to note is that orthogonality does not imply independence. Some features could be redundant, but their relationship could be non-linear, and thus beyond the reach of PCA. – jpa Feb 28 '20 at 05:59
  • 2
    @jpa that makes sense actually, are you aware of another method that could take account of possible non-linear relationships? – cfd_aero Feb 28 '20 at 07:29
  • 1
    Are you absolutely sure those features have no impact on the target, even when interacting with 1+ other features? Try a corrplot to see their pairwise impact on the target. Try using a tree for exploratory data analysis, do those features show up with zero or near-zero feature importance? – smci Feb 28 '20 at 08:45
  • 1
    @cfd_aero It is possible to expand PCA into arbitrary level polynomial relationships by expanding input dimension by adding x^2, y^2, z^2 etc. terms before running it through. But I don't see that used very often, so I'm not sure if it is very useful in practice. – jpa Feb 28 '20 at 09:45
  • 3
    @cfd_aero There are many nonlinear dimensionality reduction techniques, tSNE/HSNE is a particularly popular one in the area I work in. It's mainly useful for visualizing highdimensional datasets in few dimensions. If you'd explain the goal behind reducing the number of dimensions, we could more easily indicate which technique would be appropriate. – Erik A Feb 28 '20 at 13:09
  • @ErikA basically what I want is to reduce the dimensionality of the features in order to use those variables in a geometric optimization task. – cfd_aero Feb 28 '20 at 17:26
  • @smci I am sure that some features are very influential on the target and some are not, but PCA does not use the target so it doesn't matter as pointed in the answers here, I will try your suggestions and see what I get. – cfd_aero Feb 28 '20 at 17:28
  • @jpa I guess you are referring to kernal PCA here? – cfd_aero Feb 28 '20 at 17:28
  • @cfd_aero Please clarify. That doesn't sound like anything that requires a limited amount of dimensions upfront, unless you want to reduce the computational load or something, and reducing dimensionality before optimizing something often makes the result harder to interpret or reproduce. – Erik A Feb 28 '20 at 18:22
  • @ErikA Indeed I want to reduce the computational load, or even use the reduced dimensions to create a surrogate model . – cfd_aero Feb 28 '20 at 18:49
  • @cfd_aero The formulation I'm familiar with is more similar to this: https://ieeexplore.ieee.org/document/8646515 but I think that might just be a special case of kernel-PCA. – jpa Feb 28 '20 at 18:57
  • 1
    @cfd_aero That sounds like you want an [autoencoder](https://en.wikipedia.org/wiki/Autoencoder#Dimensionality_Reduction), if you don't have initial information about how the dimensions are related (they're not linearly related, that much we know). However, that might come at a computational increase instead of a decrease, since, well, creating one certainly isn't free. – Erik A Feb 28 '20 at 20:20
  • *"I want to reduce the dimensionality of the features in order to use those variables in a geometric optimization task."* But does the target have anything to do with that or not? What sort of geometric optimization task? – smci Feb 28 '20 at 23:15
  • the target is an efficiency ratio, and optimizing the geometry using the features would increase that efficiency, some features have more impacts than others. – cfd_aero Feb 29 '20 at 18:10
8

PCA can be used to scale and rotate data, if we select all transformed features instead of a subset of them. My answer here gives an example of scaling and rotating the data, but without dimension reduction. How to decide between PCA and logistic regression?

Your plot shows that if we use all 12 features, the variance explained is 100%, i.e., without information loss. But if you select number of features smaller than 12, there will be information loss.

Note that in most cases, PCA can reduce dimension but at the cost of losing information. If you want to keep 99% variance, unless you have highly correlated (redundant features), PCA will not able to help.

In other words, your plot shows there are not too much redundancies in your data set.

Here are examples of both cases (with 5 features).

set.seed(0)
x1=matrix(rnorm(1000),ncol=5)

x2 = matrix(rnorm(600),ncol=3)
x2=cbind(x2,x2[,3]*runif(200)*0.01)
x2=cbind(x2,x2[,3]*runif(200)*0.01)

you may run PCA on x1 and x2 to see the difference on variance explained respect to number of features selected.

You would see for x2, 3 features will explain most of the variance, because other two features are highly correlated with the third column of x2.

Haitao Du
  • 32,885
  • 17
  • 118
  • 213
  • I am not sure how your answer relates to the problem I have? could you please expand more? – cfd_aero Feb 27 '20 at 13:36
  • thanks for the answer. yes, I a aware that the plot says that, however, what I would have expected is that the first 2 or 3 PC woud carry the majority of the variance something the PCA is useful for, but it is not happening in my case, even though I know that some of the features in the dataset have a big impact on the target and others don't. – cfd_aero Feb 27 '20 at 13:44
  • 1
    @cfd_aero note PCA is unsupervised. Please read my linked post , "having impact on target" is not a good statement. – Haitao Du Feb 27 '20 at 13:46
  • Thanks for the edit, it is helping me understand more the problem, but the fact is I know that some of the features are redundant, but if the PCA is unsupervised like you suggest in the post you linked, how is it possible that here I am finding out that there isn't redundancy in my data? – cfd_aero Feb 27 '20 at 14:05
1

Well, clearly, if you were to perform PCA on a dataset, and then perform PCA on the result, you wouldn't get any benefit over just performing PCA once. So if your original dataset already had the properties of a PCA result (i.e. orthogonality), then applying PCA to it won't produce any further benefit. The more orthogonal a dataset is, the more it's already "PCA optimized", and the less further PCA helps. You should check the correlation matrix of the original dataset and see how much correlation there is between the variables.

Acccumulation
  • 3,688
  • 5
  • 11