5

There is a cluster criterion defined as:

$$\mathcal{C} = \operatorname{tr}(S_W^{-1}S_B) = \sum_{i=1}^d \lambda_i,$$

where $\operatorname{tr}$ is the trace, $S_W$ is the pooled within-group scatter matrix, and $S_B$ is the between-group scatter matrix; $d$ is the number of features (or dimensions of the scatter matrices). I have two sources for it, one here (eq 103) and the other here (p.22).

The R package clusterCrit computes this as the matrix inverse of $S_W$ (I compared their output with a "manual" calculation), but the slides, based on the book by Duda, call this "the ratio of between to within-cluster scatter in the direction of eigenvectors".

My questions:

  1. Is it the matrix inverse or a division? I find a similar term in linear discriminant analysis which is clearly a division. EDIT: Clearly this is the matrix inverse (no "matrix division").

  2. In the direction of which eigenvectors? What does that direction represent? What do the eigenvectors of $S_W^{-1}S_B$ represent?

  3. If this is the matrix inverse, then I understand that $S_W^{-1}$ can be interpreted as a precision matrix. What is the intuitive interpretation of the trace of $S_W^{-1}S_B$?

amoeba
  • 93,463
  • 28
  • 275
  • 317
Benjamin
  • 217
  • 1
  • 7
  • Re (1): What distinctions are you drawing between "inverse," "ratio," and "fraction"? Re (2): since $d$ is the dimension of the matrices, necessarily all eigenspaces are represented in the sum on the right hand side. – whuber Feb 29 '16 at 19:28
  • @whuber: I updated the terminology; I meant "inverse" as in "matrix inverse". As you politely point out, there is no "matrix division", (element-wise division makes no sense either), so that makes it the matrix inverse. Forget question 1), but 2) and 3) still stand... – Benjamin Feb 29 '16 at 20:01
  • 3
    That trace is known as Hotteling trace and is the principal statistic computed in MANOVA and related procedures. The eigenvalues of $S_W^{-1}S_B$ are the canonical discriminant roots obtained in LDA. The most convenient trick to solve this (asymmetric) eigenproblem is via Cholesky - [see](http://stats.stackexchange.com/a/48859/3277), with further links. – ttnphns Feb 29 '16 at 20:20
  • ...Hotelling (pardon for misspelling). – ttnphns Mar 01 '16 at 07:26

1 Answers1

6

$S_W^{-1}S_B$ can be interpreted as multivariate signal-to-noise ratio.

The between-class scatter matrix $S_B$ tells us how far from each other class means are located. The within-class scatter matrix $S_W$ tells us how much variability is inside each class. If classes correspond to the "signal" and within-class variability can be seen as noise, then $S_W^{-1}S_B$ can be interpreted as multivariate signal-to-noise ratio.

If the classes are well separated, then signal-to-noise ratio should be "large" ($S_W\ll S_B$). If they are completely overlapping, then the signal-to-noise ratio should be "small" ($S_W \approx S_B$). The problem is that $S_W^{-1}S_B$ is not a number but a matrix; so what does "large" and "small" really mean?

There are several reasonable ways to quantify how "big" $S_W^{-1}S_B$ is. One way is to add up its eigenvalues, i.e. to compute the trace. As @ttnphns mentioned, this is called Hotelling's trace and is used as one of the test statistics in MANOVA. So the interpretation is that it is one possible way to quantify the signal-to-noise ratio $S_W^{-1}S_B$.

In turn, the eigenvectors of $S_W^{-1}S_B$ represent the directions in space along which the class discriminability is the highest. The eigenvector corresponding to the largest eigenvalue is the axis of the best class separation. In linear discriminant analysis (LDA) the eigenvectors of $S_W^{-1}S_B$ are called "discriminant axes".

Once the data are projected on the $i$-th discriminant axis, the standard univariate signal-to-noise ratio defined as between-class sum of squares divided by within-class sum of squares, will equal $\lambda_i$. This explains the quote from Duda.

Further reading:

Community
  • 1
amoeba
  • 93,463
  • 28
  • 275
  • 317