Since Covariance matrix is symmetric it is Hermitian (self adjoint) and always diagonalizable. If the matrix has all non zero eigen-values its is a full rank matrix. But what is the importance of the covariance matrix being positive definite?
I am asking this question in context of principle component analysis. We often come across the statement that covariance matrix is positive semi-definite and also the proof for it. The proof and information are nice to know. But, by not being able to figure out its importance in given context i feel I am missing some important point. For example the importance of positive definiteness of a kernel is often stated clearly that a positive definite kernel corresponds to a RKHS. But what is the importance of positive definiteness of covariance matrix.
