2

I appreciate the help I have been getting on this site today. I had help on a proof that $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite for any random vector $\mathbf{Y}$. According to the textbook I have, it follows from this statement that $\mathbf{Y}$ is nonsingular if and only if $\text{Cov}\left(\mathbf{Y}\right)$ is positive definite.

Now this doesn't make sense to me, since $\mathbf{Y} \in \mathbb{R}^n$, and I thought that only square matrices were nonsingular? Also, I know that $\text{Cov}\left(\mathbf{Y}\right)$ is positive definite if and only if there exists a nonsingular (square matrix) $Q$ such that $$\text{Cov}\left(\mathbf{Y}\right) = QQ^{\prime}\text{.}$$

So are they perhaps actually saying that $\text{Cov}\left(\mathbf{Y}\right) = \mathbf{Y}\mathbf{Y}^{\prime}$, where $\mathbf{Y}$ is a nonsingular square matrix rather than vector?

Community
  • 1
Clarinetist
  • 3,761
  • 3
  • 25
  • 70
  • Do you mean $Y \in \mathbb{R}^n$, i.e. $Y$ is a random vector? – Matthew Drury Jun 02 '15 at 05:21
  • @MatthewDrury Oh yes, will edit – Clarinetist Jun 02 '15 at 05:24
  • 1
    Given the focus of the book you are reading on *linear models*, I am wondering whether you have revealed all the assumptions it is making about $\mathbf{Y}$. Is this truly an arbitrary random vector, or is it perhaps the dependent variable in a linear model? And if so, what additional assumptions might have been made about it? – whuber Jun 02 '15 at 18:48
  • @whuber I'm really quite sure there haven't been any other assumptions made. I can post the statement here: "It is easy to see that if $\mathbf{Y}$ is a $n$-dimensional random vector, $A$ is a fixed $r\times n$ matrix, and $b$ is a fixed vector in $\mathbb{R}^r$, then $$\mathbb{E}[A\mathbf{Y}+b] = A\mathbb{E}[\mathbf{Y}] + b$$ and $$\text{Cov}(A\mathbf{Y} + b) = A\text{Cov}(\mathbf{Y})A^{\prime}\text{.}$$... – Clarinetist Jun 02 '15 at 19:12
  • @whuber "... This last equality can be used to show that for any random vector $\mathbf{Y}$, $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite. It follows that $\mathbf{Y}$ is nonsingular if and only if $\text{Cov}\left(\mathbf{Y}\right)$ is positive definite." – Clarinetist Jun 02 '15 at 19:12
  • 2
    @whuber It looks like I must have skimmed over this page too much because there is one sentence I haven't seen before that basically clarifies everything: *A random vector is referred to as singular or nonsingular depending on whether its covariance matrix is singular or nonsingular.* Ugh. Apologies and thank you for your efforts. – Clarinetist Jun 02 '15 at 19:15
  • @Dougal see above comment – Clarinetist Jun 02 '15 at 19:15
  • 1
    When your last comment appeared I was just in the process of asking whether this text defines "singular" in an unexpected way! (@Dougal guessed as much in a comment two hours ago.) That sentence resolves everything. – whuber Jun 02 '15 at 19:16

2 Answers2

4

Your textbook may be referring to the distribution of $Y$ being a singular distribution: a distribution which doesn't have any point masses, but still its whole probability mass lies in a region of measure zero. This is kind of like $Y$ being in a lower-dimensional subset of $\mathbb R ^n$.

This is what I'd expect the statement to mean, but (as we'll see in a moment) this isn't an if and only if, just an only if. There may be some other conditions going on, or you may have misread it?

Let's call $\mathrm{Cov} Y$ just $\Sigma$ for brevity, and let $\mu = \mathbb E Y$.

First, if $\Sigma$ is singular, the distribution of $Y$ is singular.

As an example, suppose that $Y_1 \sim \mathcal{N}(0, 1)$ but $Y_2$ is always $0$. Then $\Sigma = \begin{bmatrix}1 & 0 \\ 0 & 0\end{bmatrix}$, a singular matrix, and $Y$ is concentrated on the $x$ axis of the plane (which is of measure zero).

In general, $\Sigma$ being singular means that there's some nonzero $q \in \mathbb R ^n$ with $\Sigma q = 0$. But note then that $\mathrm{Var}[ Y^T q] = q^T \Sigma q - (\mu^T q)^2 = -(\mu^T q)^2$, and since both variances and squared numbers must be nonnegative, we have both that $Y^T q$ has zero variance and $\mu^T q = 0$. Thus $Y$ lies only in the hyperplane $\{ y \mid q^T y = 0\}$, an $(n-1)$-dimensional subset of $\mathbb R ^n$.

This should make it clear that it's possible to have a singular distribution with a nonsingular covariance matrix, if $Y$ lies on any measure-zero subset other than a hyperplane through the origin.

For example, consider $Y$ to be uniformly distributed over the unit circle in $\mathbb{R}^2$; this has measure zero, but $$\mathrm{Cov}(Y_1, Y_2) = \mathbb{E}[Y_1 Y_2] = \mathbb{E}_{Y_1}\left[ \mathbb{E}\left[ Y_2 \mid Y_1 \right] \right] = \mathbb{E}_{Y_1}\left[ 0 \right] = 0,$$ and clearly $\mathrm{Var}(Y_1) = \mathrm{Var}(Y_2) = v > 0$ for some $v$, so $$\mathrm{Cov}[Y] = \begin{bmatrix}v & 0 \\ 0 & v\end{bmatrix}$$ which is nonsingular.

Danica
  • 21,852
  • 1
  • 59
  • 115
  • Thank you for the explanation. There are no other conditions imposed on the statement in my post other than $\mathbf{Y} \in \mathbb{R}^n$ being a random vector. ("Singular" in this textbook other than the matrix non-invertibility definition has not been defined.) The author even goes on to say that "By definition, $\mathbf{Y}$ nonsingular means precisely that $\text{Cov}\left(\mathbf{Y}\right)$ is positive definite," and I have always been told that definitions are iff statements. (and of course, this "by definition" statement isn't justified by any definition that has appeared so far.) – Clarinetist Jun 02 '15 at 15:07
  • 1
    @Clarinetist Yep, that sure sounds like an if and only if. I suppose the author is giving here a nonstandard definition of a singular distribution. What book is this? – Danica Jun 02 '15 at 17:17
  • This is [Plane Answers to Complex Questions](http://link.springer.com/book/10.1007%2F978-1-4419-9816-3), in chapter 1. Is it possible that the author is just flat out wrong? – Clarinetist Jun 02 '15 at 17:34
0

Matrices can be singular/non singular. Singular matrix basically means that the matrix is not full rank. For a $N \times P$ matrix with $N>P$, the matrix is non-singular if the rank of the matrix is $P$. By this notion I guess you can say vectors are never singular but that is a rather peculiar way of saying it.

In terms of singular values, the result means you have $P$ non-zero singular values. Rank of ${\bf{YY}}'$ is equal to the number of non-zero singular values of ${\bf{Y}}$

When ${\bf{Y}}$ is a vector, ${\bf{YY}}'$ is always rank 1. If ${\bf{Y}}$ is a 2 $\times$ N on the other hand, based on the rank of ${\bf{Y}}$ you can tell if the covariance matrix is singular or not

Sid
  • 2,489
  • 10
  • 15
  • It isn't at all clear how this post addresses the question, which concerns the sense in which a *vector-valued random variable* could be termed "singular." Could you perhaps edit it to make the connection evident? – whuber Jun 02 '15 at 13:08
  • The statement suggests $Y$ is a vector. Vectors are never singular in my opinion. They are 1-D objects in 1-D spaces. Hence, I discuss the result for the general $N \times P$ case. When $P = 1$, we get a vector – Sid Jun 02 '15 at 15:39
  • [Dougal's answer](http://stats.stackexchange.com/a/155094/919) suggests this interpretation may be misleading. – whuber Jun 02 '15 at 15:41
  • The question says (if and only if bit) that Cov(${\bf{Y}}$) is positive definite. Thus, it is not singular. If ${\bf{Y}}$ is a $N \times 1$ vector, ${\bf{YY}}'$ is always rank 1 and therefore singular. Dougal's answer considers the possible explanation that the covariance matrix is singular which though not the case in question, may be hinting at some point the author missed – Sid Jun 02 '15 at 16:05
  • If ${\bf{Y}}$ is a non-singular 2 $\times$ N matrix, my result shows that ${\bf{YY}}'$ is non-singular – Sid Jun 02 '15 at 16:06
  • Thank you for your comments, Sid. The place to make them is in the answer itself, though: many more people will read them there. – whuber Jun 02 '15 at 16:10
  • 1
    Thank you. I am puzzled, though. Since $\mathbf{Y}$ is a random vector (or random matrix as your "$2\times N$" remark suggests), $\mathbf{YY^\prime}$ is a random matrix, whence its rank is a random variable, too. Whether or not it indicates non-singularity will depend on the probability distribution. And when $\mathbf Y$ is a random matrix, exactly what do you mean by its covariance matrix? That is ambiguous, but ordinarily it would just be the covariance of the *components*. However, its singularity--or lack thereof--has no apparent connection to the rank of $\mathbf{Y}$ itself. – whuber Jun 02 '15 at 17:25
  • You are absolutely right. It is not exactly the covariance matrix of ${\bf{Y}}$ but of the components . What this means is that each of the multivariate samples is stacked as columns of ${\bf{Y}}$ and then the covariance matrix of the components is found by ${\bf{YY}}'$. As far as your rank comment goes, there is a very direct connection between the rank of ${\bf{Y}}$ and then rank of ${\bf{Y}}'$, If I write the SVD of ${\bf{Y}} = UWV'$ then, ${\bf{YY}'} = UW^{2}U'$ and therefore, rank of both matrices is the same – Sid Jun 04 '15 at 01:40
  • What we do not know is the rank of Y i.e., the dependence between the columns of Y. If Y has one (not all zeros) column then for this Y, U=Y, W=1 and V=1, therefore ${\bf{YY}}′=UW^{2}U′$ has rank 1.The only way in which a vector can produce a rank 0 YY′ is when W=0 i.e., Y is the 0 vector – Sid Jun 04 '15 at 02:09