12

Say $X \in \mathbb{R}^n$ is a random variable with covariance $\Sigma \in \mathbb{R}^{n\times n}$. By definition, entries of the covariance matrix are covariances: $$ \Sigma_{ij} = Cov( X_i,X_j). $$ Also, it is known that entries of the precision $\Sigma^{-1}$ satisfy: $$ \Sigma^{-1}_{ij} = Cov(X_i,X_j| \{X_k\}_{k=1}^n \backslash X_i,X_j\}), $$ where the right hand side is the covariance of $X_i$ with $X_j$ conditioned on all other variables.

Is there a statistical interpretation to the entries of a square root of $\Sigma$ or $\Sigma^{-1}$? By square root of a square matrix $A$ I mean any matrix $M$ such that $M^tM = A$. An eigenvalue decomposition of said matrices does not give such entry-wise interpretation as far as I can see.

Yair Daon
  • 2,336
  • 16
  • 29
  • 1
    @kjetilbhalvorsen added explanation. Swept a couple of details under the rugh, though. What is the value conditioned on? None given, so it has to be averaged over all values. – Yair Daon Jun 28 '17 at 17:00
  • 1
    The account I gave of regression, correlation, and conditional distributions at https://stats.stackexchange.com/questions/71260/what-is-the-intuition-behind-conditional-gaussian-distributions/71303?r=SearchResults&s=2|0.0000#71303 provides explicit geometric constructions of two different square roots of the inverse covariance matrix. These geometric ideas generalize to higher dimensions, thereby supplying at least two distinct, well-known statistical interpretations (namely, PCA and multiple regression). – whuber Dec 21 '18 at 16:37
  • what is the size of the square root of the empirical covariance matrix? is it rectangular? – Charlie Parker Oct 24 '21 at 14:07

3 Answers3

3

I will write matrix square roots of $\Sigma$ as $\Sigma=A A^T$, to be consistent with the Cholesky decomposition which is written as $\Sigma = L L^T$ where $L$ is lowtri (lower triangular). So let $X$ be a random vector with $\DeclareMathOperator{\E}{\mathbb{E}} \DeclareMathOperator{\Cov}{\mathbb{Cov}}\DeclareMathOperator{\Var}{\mathbb{Var}} \E X=\mu$ and $\Var X =\Sigma$. Let now $Z$ be a random vector with expectation zero and unit covariance matrix.

Note that there are (except for the scalar case) infinitely many matrix square roots. If we let $A$ be one of the, then we can find all the others as $A \mathcal{O}$ where $\mathcal{O}$ is any orthogonal matrix, that is, $\mathcal{O} \mathcal{O}^T = \mathcal{O}^T \mathcal{O} =I$. This is known as unitary freedom of square roots.

Let us look at some particular matrix square roots.

  1. First a symmetric square root. Use the spectral decomposition to write $\Sigma = U \Lambda U^T = U\Lambda^{1/2}(U\Lambda^{1/2})^T$. Then $\Sigma^{1/2}=U\Lambda^{1/2}$ and this can be interpreted as the PCA (principal component analysis) of $\Sigma$.

  2. The Cholesky decomposition $\Sigma=L L^T$ and $L$ is lowtri. We can represent $X$ as $X=\mu + L Z$. Multiplying out to get scalar equations, we get a triangular system in $Z$, which in the time series case can be interpreted as a MA (moving average) representation.

  3. The general case $A= L \mathcal{O}$, using the above we can interpret this as a MA representation after rotating $Z$.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • 1
    +1 Re square root calculation through spectral decomposition, I've seen $\Sigma^{1/2}=U\Lambda^{1/2}U^{T}$ as well. But this would be different from the one you mentioned, yet both are valid, is that true? – NULL Nov 01 '19 at 14:32
  • @NULL I believe that is valid, and is called the principle square root. If I'm not mistaken, $\Sigma^{\frac{1}{2}} = U\Lambda^{\frac{1}{2}}$ would not actually be symmetric, so maybe that is an error. – Brian Feb 16 '21 at 10:46
  • what is the size of the square root of the empirical covariance matrix? is it rectangular? – Charlie Parker Oct 24 '21 at 14:07
  • 1
    @Charlie Parker: The square root is of the same size as the covariance matrix. – kjetil b halvorsen Oct 24 '21 at 18:57
  • @kjetilbhalvorsen thanks for the response! If the covariance matrix $\Sigma_{X, Y}$ is of size $[D_1, D_2]$ and of rank $r$, for some reason I'd expect it to be of size $[r, r]$ (similar to the SVD). But that is wrong. Why is that? – Charlie Parker Oct 25 '21 at 17:32
  • @Charlie Parker: In this question and answers it is assumed that $\Sigma$ is square, $n\times n$. If you are interested in a more general case, you should ask a new question. But I don't know what a square root would be in the rectangular case. But maybe it can be defined. – kjetil b halvorsen Oct 25 '21 at 17:38
  • @kjetilbhalvorsen what about in this square case $[n, n]$? Would the square root be $[n, n]$ or $[r, r]$? Thanks for your time. – Charlie Parker Oct 25 '21 at 17:46
2

An nxn matrix can have many square roots as you mention. However a covariance matrix must be positive semi-definite and a positive semi-definite matrix has only one square root that is also positive semi-definite. Take a look at the wikipedia article titled "Square root of a matrix".

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • 3
    I did not ask for a symmetric square root. For example, a Cholesky decomposition is enough for the purpose of this question. – Yair Daon Jan 16 '17 at 22:03
  • 1
    I didn't say that a term in the covariance matrix couldn't be negative. I am talking about a different property that a covariance matrix is at least positive semi-definite. – Michael R. Chernick Feb 26 '19 at 23:07
  • The wikipedia page about the square root of a matrix and why positive definite and semi-definite matrices have a unique positive (semi-)definite square root are in: https://en.wikipedia.org/wiki/Square_root_of_a_matrix#Positive_semidefinite_matrices. Specifically you can refer to the sections: 2 Positive semidefinite matrices 3 Matrices with distinct eigenvalues – user96265 Oct 08 '21 at 20:04
  • what is the size of the square root of the empirical covariance matrix? is it rectangular? – Charlie Parker Oct 24 '21 at 14:07
0

Sometimes people are interested in estimating the locations of zeros in the precision matrix for the same reason you describe above. If $M$ is your square root matrix, i.e. $M'M = \Sigma^{-1}$, then for nodes $i \neq j$ $$ \Sigma^{-1}_{i,j} = 0 \iff M_i'M_j = 0 $$ so I imagine that looking at the inner product between columns of your estimated square root matrix will give you some number akin to how close to conditionally independent two nodes are. Just an idea.

The square root of the covariance matrix, that's the scale. I imagine simulating a standard normal random vector, and then pre-multiplying by the square root matrix. If this matrix is lower-triangular, then I always imagine doing all the little multiplications and additions out.

There are also individual cases you can think of where certain elements of the square root matrix are just square roots of individual elements of the covariance matrix. This isn't that interesting, though, so I figure you have already thought of this.

Taylor
  • 18,278
  • 2
  • 31
  • 66
  • what is the size of the square root of the empirical covariance matrix? is it rectangular? – Charlie Parker Oct 19 '21 at 21:07
  • @CharlieParker https://en.wikipedia.org/wiki/Square_root_of_a_matrix in this case, since $\Sigma^{-1}$ is full rank positive definite symmetric, $M$ is rectangular – Taylor Oct 19 '21 at 21:20
  • so is the shape of the square root of a covariance matrix $\Sigma_{X, Y}$ (of size `[D1, D2]`) of size $[r, r]$ for $\Sigma^{1/2}$ where $r = rank(\Sigma_{X, Y})$? – Charlie Parker Oct 24 '21 at 14:10
  • actually for $\Sigma_{X} = X^T X$ covariance of (centered) X with itself. – Charlie Parker Oct 24 '21 at 14:11