2

Is that correct that LDA/FDA can only generate 2 outputs as a dimensional reduction method?

Suppose I have 100 features, I want to reduce to 5 features. Is LDA/FDA not usable?

AdamO
  • 52,330
  • 5
  • 104
  • 209
aan
  • 75
  • 8

2 Answers2

2

Applying LDA to data with $K$ classes allows you to project the data onto a $K-1$ dimensional surface in a way that separates the data by class. You cannot arbitrarily choose the number of "features" that the data gets transformed into.

  • so Let say my original dataset has `2 classes, the output will be 1 dimensionality ( 2 – 1 =1 )`, likewise, if my original dataset has 5 classes, the output will be 4 dimensionality. – aan May 07 '20 at 16:11
  • 1
    Yes! That's right. If you want to do dimensionality reduction, consider applying PCA to get the first 5 principal components of your data, then project the data onto the subspace spanned by those 5 principal components. – Alex Van de Kleut May 07 '20 at 16:26
  • Van de Kluet Thanks. Because also `eigenvalues for dimensions > c−1 will be 0 or imaginary (where c is the classes).` – aan May 07 '20 at 16:35
  • I'm not sure what you mean by that, but the eigenvalues in PCA for real-valued data are always real-valued since PCA does not use the classes of the data. Unless two explanatory variables are perfectly correlated, PCA should give you $d$ eigenvalues/eigenvectors where $d$ is the dimensionality of the data. – Alex Van de Kleut May 07 '20 at 16:38
  • sorry, I mean because `in LDA/FDA`, the `eigenvalues for dimensions > c−1 will be 0 or imaginary (where c is the classes).` That is why the output is “c-1” where “c” is the number of classes and the dimensionality of the data is n with “n>c” – aan May 07 '20 at 16:43
  • His thoughts are related to [that question](https://stats.stackexchange.com/questions/198437/lda-for-dimensionality-reduction-usage) where you can also use LDA for dimensionality reduction (like a "supervised PCA"). – Matthieu May 08 '20 at 10:29
1

Also, note that you need labels to perform LDA, which is not always available (or relevant). That's why PCA is usually preferred because it's class-agnostic.

The number of output dimensions in LDA is linked to the number of degrees of freedom in the dataset, which is linked to the number of classes $c$: eigenvalues above $c$ will be zero and bear no information (same as with PCA and explained variance).

Matthieu
  • 318
  • 3
  • 13
  • thanks. I actually have a dataset `2 classes/labels (c), 50 features/dimensions/attributes (d), 150 samples/instances (n)`. I want to compare `PCA and LDA`. LDA only produce `1 feature` as output because `c-1 = 2-1 =1`. – aan May 08 '20 at 22:30
  • @aan compare PCA and LDA for what? Classification? Data distribution? Variance analysis? If possible, you should ask questions about the *goal*, not about the *tools*... – Matthieu May 09 '20 at 07:43
  • I want to compare the output of both methods. Just to make sure the output of LDA/FDA is `1 feature` from my `2 classes` of dataset. – aan May 09 '20 at 08:37
  • are you familiar with standarised a data in PCA? https://stats.stackexchange.com/questions/466460/what-is-the-meaning-of-standardization-in-lda-fda – aan May 14 '20 at 13:49