Questions tagged [covariance-matrix]

A $k\times k$ matrix of covariances between all pairs of $k$ random variables. It is also called variance-covariance matrix or simply the covariance matrix.

Given a random vector $\mathbf x$ with $k$ elements, its covariance matrix (also called variance-covariance matrix or simply variance matrix) is a $k \times k$ matrix $\mathbf C$ of covariances between all pairs of elements: $C_{ij} = \operatorname{Cov}(x_i, x_j)$. Diagonal elements of the covariance matrix are variances of each $x_i$. Covariance matrix is always symmetric and positive semi-definite.

The sample covariance matrix, for a random vector $\mathbf{x}$ is also a $k \times k$ matrix:

$$Q = \frac{1}{n-1} \sum_{i=1}^{n}(x_{i} - \bar{x})(x_{i} - \bar{x})^T$$

The sample covariance matrix is an unbiased estimate of the covariance matrix. Both covariance and sample-covariance matrices are positive semi-definite.

1008 questions
129
votes
6 answers

Is there an intuitive interpretation of $A^TA$ for a data matrix $A$?

For a given data matrix $A$ (with variables in columns and data points in rows), it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical solution of ordinary least squares. Or, for PCA, its…
Alec
  • 2,185
  • 4
  • 17
  • 14
86
votes
5 answers

How to interpret an inverse covariance or precision matrix?

I was wondering whether anyone could point me to some references that discuss the interpretation of the elements of the inverse covariance matrix, also known as the concentration matrix or the precision matrix. I have access to Cox and Wermuth's…
Vinh Nguyen
  • 1,031
  • 1
  • 9
  • 4
60
votes
5 answers

Is every covariance matrix positive definite?

I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?
Jingjings
  • 1,173
  • 1
  • 9
  • 13
50
votes
3 answers

Why does correlation matrix need to be positive semi-definite and what does it mean to be or not to be positive semi-definite?

I have been researching the meaning of positive semi-definite property of correlation or covariance matrices. I am looking for any information on Definition of positive semi-definiteness; Its important properties, practical implications; The…
45
votes
4 answers

Why does inversion of a covariance matrix yield partial correlations between random variables?

I heard that partial correlations between random variables can be found by inverting the covariance matrix and taking appropriate cells from such resulting precision matrix (this fact is mentioned in http://en.wikipedia.org/wiki/Partial_correlation,…
37
votes
4 answers

Measures of similarity or distance between two covariance matrices

Are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimensions)? I am thinking here of analogues to KL divergence of two probability distributions or the Euclidean distance between vectors…
34
votes
3 answers

Why is a sample covariance matrix singular when sample size is less than number of variables?

Let's say I have a $p$-dimensional multivariate Gaussian distribution. And I take $n$ observations (each of them a $p$-vector) from this distribution and calculate the sample covariance matrix $S$. In this paper, the authors state that the sample…
user34790
  • 6,049
  • 6
  • 42
  • 64
33
votes
4 answers

How to create an arbitrary covariance matrix

For example, in R, the MASS::mvrnorm() function is useful for generating data to demonstrate various things in statistics. It takes a mandatory Sigma argument which is a symmetric matrix specifying the covariance matrix of the variables. How would…
rsl
  • 845
  • 2
  • 9
  • 15
33
votes
1 answer

Is there a way to use the covariance matrix to find coefficients for multiple regression?

For simple linear regression, the regression coefficient is calculable directly from the variance-covariance matrix $C$, by $$ C_{d, e}\over C_{e,e} $$ where $d$ is the dependent variable's index, and $e$ is the explanatory variable's index. If one…
David
  • 433
  • 1
  • 5
  • 8
32
votes
2 answers

Generating data with a given sample covariance matrix

Given a covariance matrix $\boldsymbol \Sigma_s$, how to generate data such that it would have the sample covariance matrix $\hat{\boldsymbol \Sigma} = \boldsymbol \Sigma_s$? More generally: we are often interested in generating data from a density…
Kees Mulder
  • 1,414
  • 1
  • 10
  • 10
31
votes
7 answers

Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-variance matrix is SPD. Co-variance matrix is a important…
29
votes
5 answers

A measure of "variance" from the covariance matrix?

If the data is 1d, the variance shows the extent to which the data points are different from each other. If the data is multi-dimensional, we'll get a covariance matrix. Is there a measure that gives a single number of how the data points are…
dontloo
  • 13,692
  • 7
  • 51
  • 80
29
votes
5 answers

How to generate a large full-rank random correlation matrix with some strong correlations present?

I would like to generate a random correlation matrix $\mathbf C$ of $n \times n$ size such that there are some moderately strong correlations present: square real symmetric matrix of $n \times n$ size, with e.g. $n=100$; positive-definite, i.e.…
amoeba
  • 93,463
  • 28
  • 275
  • 317
26
votes
1 answer

Different covariance types for Gaussian Mixture Models

While trying Gaussian Mixture Models here, I found these 4 types of covariances. 'full' (each component has its own general covariance matrix), 'tied' (all components share the same general covariance matrix), 'diag' (each component has its own…
Bee
  • 423
  • 1
  • 4
  • 5
24
votes
3 answers

Unbiased estimation of covariance matrix for multiply censored data

Chemical analyses of environmental samples are often censored below at reporting limits or various detection/quantitation limits. The latter can vary, usually in proportion to the values of other variables. For example, a sample with a high…
whuber
  • 281,159
  • 54
  • 637
  • 1,101
1
2 3
67 68