Intuitive meaning of vector multiplication with covariance matrix

Question

I often see multiplications with covariance matrices in literature. However I never really understood what is achieved by multiplication with the covariance matrix. Given $\Sigma * r = s$ with $\Sigma$ being the covariance matrix of $n$ random variables $X_i$, can someone give me an intuitive explanation what $s$ gives me?

What I already (at least think) to understand is the principle of covariance in general and a meaning of the covariance matrix in terms of a linear basis with the $i$th basis vectors being the covariance between random variable $X_i$ and $X_j$ for $1 \leq j \leq n$.

Some intuition I already gathered is as follows: By multiplying $\Sigma * r$ we weight the samples $X_i$ according to $r$. With fixed $i$, $y_i$ gives us then the sum over the weighted covariances with $X_i$ and $X_j$ for $1 \leq j \leq n$, which means a value of how well $X_i$ "covaries" in the direction of $r$.
But what would mean "in the direction of $r$" and what is this result really useful for? Often I see a value like this: $r^T * \Sigma^{-1} * r$
What would this value be useful for?
(And I know about the nice properties of the Eigenvalues and Eigenvectors for $\Sigma$)

Answer 1

Your answer is good. Note that since $\Sigma$ is symmetric and square so is $\Sigma^{-1}$. The matrix, its transpose, or inverse all project your vector $\Sigma r$ in the same space.

Since $\Sigma$ and $\Sigma^{-1}$ are positive definite, all eigenvalues are positive. Thus a multiplication with a vector always ends up in the same halfplane of the space.

Now if $\Sigma$ or $\Sigma^{-1}$ would be a diagonal matrix, then the multiplication would reweigh (or undo the reweigh) of only the lengths of the target vector in each dimension (as you noticed). If they are full matrices, then indeed the matrix is full rank as it is PSD, the eigendecomposition exists and $\Sigma = V \Lambda V^{-1}$, here $V$ is an orthonormal eigenvector matrix by the virtue of $\Sigma$ being PSD, and $\Lambda$ the diagonal with eigenvalues. Thus $r$ is first rotated by $V^{-1}$, and then reweighed by $\Lambda$, then rotated back by $V$. The same thing goes for $\Sigma^{-1}$, but then $r$ is rotated the other way around and the scaled by the diagonal of reciprocals $\Lambda^{-1}$ and rotated back with $V^{-1}$. It is easy to see they are opposite processes.

Additionally, you may think of $$ r^T \Sigma^{-1} r = (\Lambda^{-1/2} V^T r)^T(\Lambda^{-1/2} V^T r) = \big\|\Lambda^{-1/2} V^T r\big\|^2 $$ as the length of your vector $r$ reweighed by the "standard deviations" after correction for cross-correlations.

Hope that helps.

Answer 2

After some further thinking I now came to a (sort of) intuitive interpretation myself. It is actually not that hard if one is familiar with PCA in the context of covariance and variance.

We denote by $X$ a dataset $X_1 \dots X_n$ with $X_i$ being a random vector of dimension $d$ (this is a slightly different notation than I used in my question).
If we look at the eigendecomposition of $\Sigma$ one observes that multiplying $r$ with $\Sigma$ just stretches $r$ in the directions of the principle components of the dataset $X$ by the according eigenvalues component (the variances of the dataset in the directions of the principle components).
A nice property of this linear mapping is, that $r^T \Sigma r$ returns the variance of the dataset in direction of $r$ (assuming $|r| = 1$, otherwise it returns the variance times $|r|^2$).

With this being said, lets look at the term $r^T \Sigma^{-1} r$. The term $\Sigma^{-1} r$ now retracts the vector $r$ to $r' = \Sigma^{-1} r$ again according to the principle components (scales by the reciprocal value of the eigenvalues of $\Sigma$ along the principle components (eigenvectors)), as this is just the reversed operation. This means, that we receive with the previous term just the variance along $r'$ (again assuming $|r'| = 1$.
A small proof of this statement: $$r^T \Sigma^{-1} r = \left( \Sigma^{-1} r \right)^T r = \left( \Sigma^{-1} r \right)^T \Sigma \Sigma^{-1} r = r'^T \Sigma r'$$ which is equal to the variance of $X$ along $r'$.

Answer 3

Note that $x^TAx$ is a just number, in many cases, we want this number to be small, i.e., try to solve the optimization problem. The nice property of this term is if $A$ SPD, then $\frac 1 2 x^TAx - b^Tx$ will have the same solution with the linear system $Ax=b$.

More discussions can be found here.

Why are symmetric positive definite (SPD) matrices so important?

I strongly recommend following 2 tutorials that helped me a lot.

essence of linear algebra

An Introduction to the Conjugate Gradient Method Without the Agonizing Pain

Answer 4

1. Eigenvectors of the empirical covariance matrix are directions where data has maximal variance.
2. We know that the eigenvector basis of a linear operator is the basis where the operator has diagonal representation.

combining these two facts above I conclude:

If the vector $x$ were drawn from normal distribution $\mathcal{N}(0, I)$ then multiplication with the covariance matrix would result in a vector drawn from normal distribution but according to the empirical covariance matrix.

I believe there are other intuitions. Please share it with us.