1

First of all sorry but I am a little bit of a newbie to advanced stats hence why my question may sound silly.

I am trying to perform a classification task where I assume that my data is generated by a multivariate Gaussian distribution. For that purpose I estimate the covariance matrices from my sample data, but for some of the variables I am getting a non positive definite matrix. Could anyone provide me with a sound explanation to this (I understand I have some non linearly independent variables in my data probably?) and maybe a workaround so I can proceed?

PL-RL
  • 111
  • 1
  • 2
    Could you explain what you mean by "proceed"? What exactly is the matter with obtaining a singular covariance matrix? – whuber Oct 11 '16 at 16:24
  • Hi @whuber, in order to "proceed" with my classification task I need to estimate probabilities that depend on these estimated covariance matrices. What I mean is I am using a multivariate Gaussian to calculate the probabilities of an observation given each class and then just using the argmax of that to classify, hence why I need those covariance matrices to work for the probability density function... Any idea how I could work around my data to avoid singular covariance matrices? – PL-RL Oct 11 '16 at 23:16
  • It's unclear how the singularity of a covariance matrix could prevent you from estimating a probability. – whuber Oct 12 '16 at 12:50
  • Any data generated by a multivariate Gaussian function must have a positive definite covariance matrix. I am reversing that, and estimating the covariance matrices from my data set, having as a starting point the assumption that my data is Gaussian. If my covariance matrix is not positive definite, I cannot use it to calculate probabilities with a multivariate Gaussian. But maybe I am missing something in this approach...? – PL-RL Oct 12 '16 at 14:06
  • 1
    Any data generated by *any* distribution--including Gaussians--only need have a positive *semidefinite* covariance. You definitely can use your estimate to compute probabilities. After all, your covariance estimate (along with an estimated multivariate mean) completely determines the distribution! – whuber Oct 12 '16 at 14:14
  • Ok then maybe the problem is I am using native matlab functions to determine the pdf value at point x, but the truth is when I apply the Cholesky covariance decomposition (`cholcov` in Matlab) I get the result that my matrix has at least one negative eigenvalue... Should I approach this the hard coded way and ditch the prebuilt functions? BTW thanks a lot for your help @whuber, I needed this discussion to move forward! – PL-RL Oct 12 '16 at 14:42
  • The value of the PDF is not a probability--and therein lies your difficulty. The approaches described in http://stats.stackexchange.com/questions/159313 and its duplicate might point you in a good direction. The negative eigenvalue arises from floating point roundoff error, so you will want to deal with that, too. – whuber Oct 12 '16 at 14:45
  • I am aware of that (PDF vs. probability), but I need the pdf value to make my estimates. Thanks again for the help, I will check out the link and the roundoff error (which I believe may be the source of my problem here now)! – PL-RL Oct 12 '16 at 14:51

1 Answers1

1

You might have a singular multivariate normal distribution. See Multivariate normal with singular covariance. Such a singular distribution do not have a density (in the full space ... ) Can a multivariate distribution with a singular covariance matrix have a density function? But we can still work with it, for instance using the Moore-Penrose generalized inverse of the covariance matrix.

But, maybe the software you are using do not do so. If you tell us with more details what you want to do, we can tell you ...

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467