3

I am studying Linear Discriminant Analysis (LDA). According to the formula for LDA, we are supposed to get the inverse of within group covariance. However, if $p\gg n$ (i.e., the dimension is much larger than the number of samples), what should I do?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
coolcat
  • 235
  • 1
  • 5

1 Answers1

4

One possibility is to regularize your estimate of the covariance matrix. See Regularization parameter to generate inverse covariance matrix.

Regularization uses assumed prior-information to turn the covariance inversion into a well-posed problem (this is also mentioned in the wikipedia article)

In practice, you can use a shrinkage estimator of the covariance matrix, such that:

$$\Sigma \rightarrow (\lambda-1)\Sigma+\lambda I_m$$

Where $\lambda$ is a hyperparameter that must be set in advance (and possibly optimized).

Firebug
  • 15,262
  • 5
  • 60
  • 127
  • Thank you! That is a good strategy to get the inverse of a matrix. But in LDA, do people also use this method to solve this problem? – coolcat Oct 05 '18 at 17:50
  • @coolcat Yes, it's even mentioned in the article linked. There are other ways to achieve regularization though. I only presented an alternative. – Firebug Oct 05 '18 at 18:12