2

Suppose $\mathbf{Y}$ is a $n$-dimensional random vector, $A$ is a fixed $r \times n$ matrix, and $b$ is a fixed vector in $\mathbb{R}^n$. I have proven already that $$\mathbb{E}\left[A\mathbf{Y}+b\right] = A\mathbb{E}\left[\mathbf{Y}\right]+b\text{, }$$ $$\mathbb{E}\left[A\mathbf{Y}b\right] = A\mathbb{E}[\mathbf{Y}]b\text{,} $$ and $$\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}\text{,}$$ where $\text{Cov}(\mathbf{Y})$ denotes the (variance-)covariance matrix of $\mathbf{Y}$.

The book says that using $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$, we can show that $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite for any random vector $\mathbf{Y}$.

So we have for $v \in \mathbb{R}^n$, $$v^{\prime} \text{Cov}\left(\mathbf{Y}\right) v = \text{Cov}\left(v^{\prime}\mathbf{Y}\right)\text{.}$$ Just looking at the matrix dimensions, I know that $v^{\prime}\mathbf{Y}$ is a $1 \times 1$ summation of terms. In particular, $$v^{\prime}\mathbf{Y} = \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}\begin{bmatrix} y_{1} \\ y_2 \\ \vdots \\ y_n\end{bmatrix} = \sum\limits_{i=1}^{n}y_iv_i\text{.}$$ Just given my machinery above, I'm not sure how to find $\text{Cov}\left(\sum\limits_{i=1}^{n}y_iv_i\right)$ without it being very messy.

Note that I already asked a very similar question here, but I would like to know what the author of the book is thinking by suggesting using $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$.

Community
  • 1
Clarinetist
  • 3,761
  • 3
  • 25
  • 70
  • 1
    I must be missing something. You're asking how to show that the covariance of any random vector is non-negative definite. What does this have to do with $A$ or $b$? They seem to be irrelevant. Moreover, the demonstration is immediate (and has appeared in other threads on this site): because for any $x\in\mathbb{R}^n$, $x^\prime\text{cov}(Y)x$ is the variance of a random vector $Yx$, it is non-negative, *QED*. Is there perhaps some confusion in the text concerning what you're trying to prove? – whuber Jun 01 '15 at 21:23
  • @whuber The text states $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$ implies that $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite. The variance of a random vector in this text has not yet been defined. The machinery I have been given so far is that $\mathbb{E}\left[A\mathbf{Y}+b\right] = A\mathbb{E}\left[\mathbf{Y}\right]+b$, $\mathbb{E}\left[A\mathbf{Y}b\right] = A\mathbb{E}[\mathbf{Y}]b$, and $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$, as shown in my post. – Clarinetist Jun 01 '15 at 22:05
  • @whuber I also know that a symmetric matrix $A$ is nonnegative definite if and only if $A = QQ^{\prime}$ for some square matrix $Q$ and that I can decompose $A$ as $PD(\lambda_i)P^{\prime}$, where $P$ is orthogonal. – Clarinetist Jun 01 '15 at 22:06
  • @whuber What I am struggling with is using what I have so far to prove that $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite using my machinery. The answer below assumes that I have already shown that any covariance matrix is nonnegative definite, which isn't the case here. I am still trying to dissect that claim. – Clarinetist Jun 01 '15 at 22:07
  • 1
    Sorry, the thing I termed "random vector" is just a random variable. Since you have a definition of covariances you *certainly* have a definition of variances. – whuber Jun 01 '15 at 22:15
  • @whuber Actually not. For $\mathbf{Y} = [y_j]$, The textbook defines $$\mathbb{E}\left[\mathbf{Y}\right] = \begin{bmatrix} \mathbb{E}[y_1] \\ \mathbb{E}[y_2] \\ \vdots \\ \mathbb{E}[y_n]\end{bmatrix}$$ and $\text{Cov}\left(\mathbf{Y}\right) = \mathbb{E}\left[\left(\mathbf{Y}-\boldsymbol{\mu} \right)\left(\mathbf{Y}-\boldsymbol{\mu} \right)^{\prime}\right]$. It assumes knowledge of elementary probability, so component-wise variances (i.e., of the $y_j$) have been defined, but of the random vector $\mathbf{Y}$, they haven't been defined. – Clarinetist Jun 01 '15 at 22:33
  • 2
    Clarinetist, you just have to follow whuber's original suggestion.cov of a scalar=variance>=0. And take A=v ie 1xn 'matrix' – seanv507 Jun 01 '15 at 22:33

3 Answers3

3

A square matrix $C = [C_{i,j}]$ is said to be positive semi-definite if and only if $$\sum_i \sum_j a_ia_j C_{i,j} \geq 0 ~ \forall a_i, a_j \in \mathbb R. \tag{1}$$ It is called positive definite if the inequality in $(1)$ is a strict inequality.

Now choose $A$ to be the row vector (a.k.a. $1\times n$ matrix) $[a_1, a_2, \ldots, a_n]$ so that $A\mathbf Y$ is the univariate random variable $Z= \sum_i a_iY_i$ and so $$\operatorname{cov}\left(A\mathbf Y+b\right) = \operatorname{var}(Z) \geq 0.\tag{2}$$ But, we are given that $$\operatorname{cov}\left(A\mathbf{Y}+b\right) = A\operatorname{cov}\left(\mathbf{Y}\right)A^{T}\tag{3}$$ where, for our chosen $A$, the right side of $(3)$ is just $\sum_i \sum_j a_i a_j \operatorname{cov}\left(\mathbf Y\right)_{i,j}$. We conclude from $(2)$ and $(3)$ that $$\sum_i \sum_j a_i a_j \operatorname{cov}\left(\mathbf Y\right)_{i,j}\geq 0~ \forall a_i, a_j \in \mathbb R,\tag{4}$$ that is, $\operatorname{cov}\left(\mathbf Y\right)$ is a positive semi-definite matrix.

Adapted from this previous answer of mine.

Community
  • 1
Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200
  • It took me a while to realize that $\text{Cov}\left(A\mathbf{Y}+b\right) = \text{Var}(Z)$, but it made sense once I realized that the covariance matrix of a vector in $\mathbb{R}^1$ is just the variance of the random variable in the vector. Thank you. By the way, $b$ isn't necessary for this proof at all, is it? – Clarinetist Jun 02 '15 at 04:05
  • No, $b$ was not needed except that we did use the special case $\operatorname{var}(Z+b)=\operatorname{var}(Z)$ without saying so in arriving at $(2)$. – Dilip Sarwate Jun 02 '15 at 04:20
1

Recall any valid covariance matrix is nonneg def. So for any random vector $Y$ if $\mbox{Cov}(Y)$ is defined, it has $\mbox{Cov}(Y) \geq 0$. Furthermore, another property of NND matrices is that, for any real vector $A$, and any NND matrix $\Sigma$, $A^\prime \Sigma A$ is also NND.

AdamO
  • 52,330
  • 5
  • 104
  • 209
  • Unfortunately your sentence starting with "recall" isn't familiar to me. How would one prove this? – Clarinetist Jun 01 '15 at 20:07
  • @Clarinetist that would have been the ideal starting point for such a question as yours. If going a more axiomatic route, the full proof of my "recall" claim is in Thm 3.1 of [these lecture notes](http://www.stat.washington.edu/jaw/COURSES/580s/581/LECTNOTES/ch1.pdf). The rough sketch is using the singular value decomposition of the covariance matrix. – AdamO Jun 01 '15 at 20:16
  • I am looking at Theorem 3.1 right now - it looks like it is dependent on $\Sigma = AA^{\prime}$ for jointly normal distributions. The factorization $AA^{\prime}$ I know is a necessary and sufficient condition for nonnegative definite matrices, but is there a way to prove that an *arbitrary* covariance matrix can be factored as such (i.e., not just for jointly normal distributions)? – Clarinetist Jun 01 '15 at 20:22
  • @Clarinetist they're all twoway implications. All the nonprobabilistic claims should have been covered in a basic linear algebra class. If you haven't covered them, you would do well to study this more closely. For instance, none of the assumptions in Thm 3.1 involve jointly normal RVs. An arbitrary cov mx is symmetric (verify from def), so it's SVD gives rise to AA' as a factorization. – AdamO Jun 01 '15 at 20:27
1

In the result, plug $A = U'$, where $U$ is the matrix of eigenvectors. Writing the eigendecomposition $S = U \Lambda U^{T}$ reveals $Cov(UY)$ =$\Lambda$ which is a diagonal matrix

Now, the eigenvalues of a diagonal matrix are the diagonal entries. But, the diagonal entries are the variances and hence non-negative (this result can also be proved trivially) and therefore, the covariance matrix of $UY$ is positive definite.

Now use $Y_0 =UY$ and $A_{0} = U^{T}$ in the equation to get your result

Sid
  • 2,489
  • 10
  • 15