4

Consider the $p \times p$ sample covariance matrix:

$$\mathbf{S} = \frac{1}{n-1} \cdot \mathbf{Y}_\mathbf{c}^\text{T} \mathbf{Y}_\mathbf{c} \quad \quad \quad \mathbf{Y}_\mathbf{c} = \mathbf{C} \mathbf{Y},$$

where $\mathbf{C} = \mathbf{I}-\frac{1}{n} \mathbf{1} \mathbf{1}^\text{T}$ is the $n \times n$ centering matrix and $\mathbf{Y}$ is an $n \times p$ matrix. How can it be proved that if the variables are continuos, not linearly related and $n-1> p$ then the sample covariance matrix is ​​positive definite?

The following clue is found in Ranchera's book:

enter image description here

user.
  • 211
  • 1
  • 6
  • No, this isn't my question. – user. Sep 15 '20 at 01:58
  • 2
    You've asked how to prove that a sample covariance matrix is positive definite. The answers provide proofs that the sample covariance matrix is positive *semi*-definite. How is this not an answer to your question? – Sycorax Sep 15 '20 at 02:00
  • 1
    I want to know how the condition $ - 1> $ makes the matrix positive definite. I know covariance matrix is positive semi-definite. – user. Sep 15 '20 at 02:12
  • 2
    It's possible that the misunderstanding arises because the claim is false. $S$ is positive **semi**-definite whenever the columns of $Y$ are linearly dependent -- for example, if each row of $Y$ sums to the same number. This circumstance is addressed in the duplicate. – Sycorax Sep 15 '20 at 02:16
  • Your question about $n-1\gt p$ is asked and answered at https://stats.stackexchange.com/questions/60622. – whuber Sep 15 '20 at 12:51
  • The claim is false whenever the columns of $Y_c$ are linearly dependent. For example: $$ Y=\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}, \quad\text{rk}(Y)=2$$ but $$Y_c=CY=\begin{bmatrix} 0 & -1 \\ 0 & 0 \\ 0 & 1 \end{bmatrix},\quad\text{rk}(Y_c)=1,\quad Y_c^TY_c=\begin{bmatrix} 0 & 0 \\ 0 & 2 \end{bmatrix}$$ which doesn't look positive definite. – Sergio Sep 15 '20 at 13:55
  • Moreover, if $\text{rk}(Y_c)=p$, then the proof is trivial: if $M$ is an $n\times p$ matrix, $n\ge p$, $\text{rk}(M)=p$, then $\text{rk}(M^TM)=p$, therefore if it is positive semidefinite, it must be positive definite. – Sergio Sep 15 '20 at 14:13

1 Answers1

4

First, let's simplify the equation for your sample covariance matrix. Using the fact that the centering matrix is symmetric and idempotent you get the $p \times p$ form:

$$\begin{align} \mathbf{S} &= \frac{1}{n-1} \cdot \mathbf{Y}_\mathbf{c}^\text{T} \mathbf{Y}_\mathbf{c} \\[6pt] &= \frac{1}{n-1} \cdot (\mathbf{C} \mathbf{Y})^\text{T} (\mathbf{C} \mathbf{Y}) \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{Y}^\text{T} \mathbf{C}^\text{T} \mathbf{C} \mathbf{Y} \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{Y}^\text{T} \mathbf{C} \mathbf{Y}. \\[6pt] \end{align}$$

This is a simple quadratic form in $\mathbf{Y}$. I will show that this matrix is non-negative definite (or "positive semi-definite" if you prefer) but it is not always positive definite. To do this, consider an arbitrary non-zero column vector $\mathbf{z} \in \mathbb{R}^p - \{ \mathbf{0} \}$ and let $\mathbf{a} = \mathbf{Y} \mathbf{z} \in \mathbb{R}^n$ be the resulting column vector. Since the centering matrix is non-negative definite (it has one eigenvalue equal to zero and the rest are equal to one) you have:

$$\begin{align} \mathbf{z}^\text{T} \mathbf{S} \mathbf{z} &= \frac{1}{n-1} \cdot \mathbf{z}^\text{T} \mathbf{Y}^\text{T} \mathbf{C} \mathbf{Y} \mathbf{z} \\[6pt] &= \frac{1}{n-1} \cdot (\mathbf{Y} \mathbf{z})^\text{T} \mathbf{C} \mathbf{Y} \mathbf{z} \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{a}^\text{T} \mathbf{C} \mathbf{a} \geqslant 0. \\[6pt] \end{align}$$

This shows that $\mathbf{S}$ is non-negative definite. However, it is not always positive definite. To see this, take any $\mathbf{z} \neq \mathbf{0}$ giving $\mathbf{a} = \mathbf{Y} \mathbf{z} \propto \mathbf{1}$ and substitute into the quadratic form to get $\mathbf{z}^\text{T} \mathbf{S} \mathbf{z} = 0$.

Update: This update is based on the additional information you have added in your edit to the question and your comments. In order to get a positive definite sample variance matrix you need $\mathbf{a}^\text{T} \mathbf{C} \mathbf{a} > 0$. If $n-1>p$ and all $n$ rows of $\mathbf{Y}$ are linearly independent then $\mathbf{Y} \mathbf{z} \propto \mathbf{1}$ implies $\mathbf{z} = \mathbf{0}$. The contrapositive implication is that $\mathbf{a}^\text{T} \mathbf{C} \mathbf{a} > 0$ for all $\mathbf{z} \neq 0$, which establishes that the sample covariance matrix is positive definite. Presumably this is what you are looking for.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • ok, I understand this, but How does the condition $ - 1> $ make the matrix positive definite? – user. Sep 15 '20 at 02:06
  • As far as I can see, it doesn't necessarily do this. However, under some additional conditions on $\mathbf{Y}$ you might be able to ensure that $\mathbf{a} \neq \mathbf{0}$. – Ben Sep 15 '20 at 02:45
  • There were two other conditions that I forgot, the variables are not linearly related and they are continuous. I placed a clue to solve the proof in the question. – user. Sep 15 '20 at 02:50
  • @JavierMariño Ben's answer provides that $S$ is positive semidefinite in general. You've edited the question to specify that $Y$ has linearly independent columns (equiv. $Y$ has rank $p$). By the definition of linear independence, we know that there is no $z\in \mathbb{R}^p \setminus 0$ such that $Yz=0$. This completes Ben's proof. – Sycorax Sep 15 '20 at 02:59
  • And the n-1

    – user. Sep 15 '20 at 03:04
  • 1
    It's not necessary. It's implied by the linear independence, since you can't have $p$ linearly independent columns if $p\geq n$. Finding a linear dependence involves solving $p$ equations in $n$ unknowns, which you can do if $p\geq n$ – Thomas Lumley Sep 15 '20 at 03:36
  • mm, a lot of books have this proposition. – user. Sep 15 '20 at 03:59
  • I have updated the question to deal with the additional conditions you have specified, to show how these imply positive definiteness. – Ben Sep 15 '20 at 05:26