1

Assume that we have two classes, $X$ and $Y$ and we find the mean $X_\mu$ and $Y_\mu$ and variance $X_\sigma$ and $Y_\sigma$.

With that, we could use linear discriminant analysis to expend the distanse between $X$ and $Y$.

But when I look at the images of linear discriminant analysis, it seems only that the data has been "rotated".

Here is a good example how to interpret linear discriminant analysis, where one axis is the mean and the other one is the variance.

Question:

What's the idea of using linear discriminant analysis if I already know the average mean and variance of each class?

I mean, if I have three classes, X, Y, Z and I know the average mean of them and the average variance. Then I get a unknown class U. If I want to find out which class U belongs to, I could just use a simple pythagorean theorem. I don't need to rotate the data.

It seems that linear discriminant analysis only rotates the data. I don't find the use case in linear discriminant analysis.

Or could it be that linear discriminant analysis also expands the distance between the classes too?

enter image description here

ttnphns
  • 51,648
  • 40
  • 253
  • 462
Nazi Bhattacharya
  • 203
  • 1
  • 8
  • Your picture clearly shows that in the case shown only one discriminant, the one on the right, is sufficient to discriminate the classes & to classify to them. So, instead of 2 features you have 1. Isn't that a major reason we use LDA for? – ttnphns Dec 04 '21 at 16:30
  • @ttnphns I thougth LDA was supposted to maximize the distance between classes, not rotate them for a better visualization. – Nazi Bhattacharya Dec 05 '21 at 13:44
  • True. LDA created new space directions which are not necessarily orthogonal to each other, so it is not a "rotation" in the narrow sense. And yes, it maximizes separation. Your right axis (discriminant) maximizes it. – ttnphns Dec 05 '21 at 15:09
  • Showing that discriminants are not necessarily orthogonal axes in the original space: https://stats.stackexchange.com/a/22889/3277. Comparing LDA and PCA (a rotation) visually: https://stats.stackexchange.com/q/12861/3277. And numerically on the Iris dataset: https://stats.stackexchange.com/a/83114/3277. – ttnphns Dec 05 '21 at 15:12
  • @ttnphns So the image above, is only a geometrical interpretion? – Nazi Bhattacharya Dec 05 '21 at 17:30
  • @ttnphns Will the distance between two classes become larger with LDA? – Nazi Bhattacharya Dec 05 '21 at 17:31
  • Look at the pic in my 1st link, it is better. The discriminant 1 "drawn" so as the group centroids projected on it is maximally apart. Discriminant 2 is the remaining one (only two are possible there), it is drawn so as to be uncorrelated with the 1st one. – ttnphns Dec 05 '21 at 18:06
  • @ttnphns Yes, I understand. But if we compute the absolute distance as it was a coordinate system, then LDA is useless here. – Nazi Bhattacharya Dec 05 '21 at 18:16
  • Because LDA is not a rotation of original axes (like PCA is), it does not preserve original euclidean distances between data points or between group centroids. LDA just has no such aim. Note that LDA decomposes not the covariance matrix of the data, but the "B/W" matrix which represents a _ratio_, thence it is unitless. – ttnphns Dec 05 '21 at 19:36
  • @ttnphns Ok. Then I know that all the LDA plots are just geometrical intepretions and the distance between the mean of the classes/objects/data will expand after finding the $W$ matrix from LDA. (I'm talking about multiclass LDA). – Nazi Bhattacharya Dec 05 '21 at 19:52
  • 1
    I think that overall, the distances will expand. If one computes squared euclidean distances (between points or between classes) in the original variables, and does so in the space of the discriminants, and then compare sums of the distances, the sum in the second matrix will be greater. – ttnphns Dec 05 '21 at 20:16
  • @ttnphns Can you prove it in this question? Then I can accept it as an answer. Let's say five classes. MATLAB is a OK tool to use, or Python. – Nazi Bhattacharya Dec 05 '21 at 21:16

0 Answers0