Quadratic form and Chi-squared distribution

Question

It's about the demostration of the quadratic forms and chi-squared distribution.

Let's split the problem:

1. We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2...z_n]}$. Obviously $z'z$ is a $\chi^2_n$
2. If we have a symmetric, idempotent matrix (i.e. called $A$) then $z'Az \thicksim \chi^2_p $ where $p$ is the rank of $A$. I got this splitting the $A$ matrix with the spectral theorem into: $Q'\Lambda Q$ hence: $z'Az = z'Q'\Lambda Qz$ we call $w= Qz$ hence $z'Az =w'\Lambda w$. Because it's a idempotent matrix we have only 1s or 0s as eigenvalues and the number of 1's is exactly the rank of $A$. Then we have $w'w$ with with different length now as the number of the rank of $A$. We backward the process and we get: $z'Q'Qz$ but $Q'Q= I_{rank(A)}$ (they are orthogonal since we consider the eigenvectors) so we have the summation of $p$ $Z^2$.
3. Now the question comes: if we have a symmetric, positive definite matrix $\Sigma_{nxn}$ my teacher told me that $z'\Sigma z$ is a $\chi^2_n $ distribution. I was trying to find a proof, but I can't find one.

4. My efforts are: use the spectral theorem again, hence we have: $z'Q' \Lambda Qz$ but now $\Lambda$ is a diagonal matrix with real positive numbers (cause it's positive definite).

   $\Lambda$= \begin{bmatrix} \lambda_{1} & 0 & 0 & \dots & 0 \\ 0 & \lambda_{2} & 0 & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \dots & \lambda_{n} \end{bmatrix}

We have called, as before, $w=Qz$ hence we have: $w'\Lambda w$. Let's suppose, that w is 3-dimensional vector with 3 elements called a,b and c. So $w=$ \begin{bmatrix} a \\ b \\ c \\ \end{bmatrix}

so $w'\Lambda w= a^2 \lambda_1+b^2 \lambda_2+c^2 \lambda_3$ And here I'm stuck. Maybe this last point is wrong. Therefore I'm asking for clarification and a clear proof of what my teacher said.

[EDIT]

I found this:

Let $X$ be a $ K \times 1$ standard multivariate normal random vector, i.e., $X \thicksim N(0,I)$. Let $A$ be an orthogonal $K \times K$ real matrix. Define $Y=AX$ Then also $Y$ has a standard multivariate normal distribution, i.e., $Y \thicksim N(0,I)$. I'm okay with this, the proof is here: https://www.statlect.com/probability-distributions/normal-distribution-quadratic-forms So my problem would come something like this: $w'\Lambda w$ where $w=QZ$ and by this theorem $w \thicksim N(0,I)$ since Q is an orthogonal matrix. So, if w is a 3 dimensional vector I'll have something like this:

$\lambda_1 \chi^2_1 + \lambda_2 \chi^2_1 + \lambda_3 \chi^2_1 $ A lineare combination of chi-squared distribution with weights the value of the eigenvalues. Am I right? If I am, does this sum has a particular distribution?

Answer 1

In general, the quadratic form is a weighted sum of $\chi_1^2$

It is not true in general that $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi^2_p$ for any symmetric positive-definite (variance) matrix $\mathbf{\Sigma}$. Breaking this quadratic form down using the spectral theorem you get:

$$\begin{equation} \begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} = \mathbf{z}^\text{T} \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^\text{T} \mathbf{z} &= (\mathbf{Q}^\text{T} \mathbf{z})^\text{T} \mathbf{\Lambda} (\mathbf{Q}^\text{T} \mathbf{z}) \\[6pt] &= \sum_{i=1}^p \lambda_i ( \mathbf{q}_i \cdot \mathbf{z} )^2, \\[6pt] \end{aligned} \end{equation}$$

where $\mathbf{q}_1,...,\mathbf{q}_p$ are the eigenvectors of $\mathbf{\Sigma}$ (i.e., the columns of $\mathbf{Q}$). Define the random variables $y_i = \mathbf{q}_i \cdot \mathbf{z}$. Since $\mathbf{\Sigma}$ is a real symmetric matrix, the eigenvectors $\mathbf{q}_1,...,\mathbf{q}_p$ are orthonormal, which means that $y_1,...,y_p \sim \text{IID N}(0,1)$. Thus, we have:

$$\begin{equation} \begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} &= \sum_{i=1}^p \lambda_i ( \mathbf{q}_i \cdot \mathbf{z} )^2 \\[6pt] &= \sum_{i=1}^p \lambda_i \cdot y_i^2 \\[6pt] &\sim \sum_{i=1}^p \lambda_i \cdot \chi_1^2. \\[6pt] \end{aligned} \end{equation}$$

We can see that the distribution of the quadratic form is a weighted sum of $\chi_1^2$ random variables, where the weights are the eigenvalues of the variance matrix. In the special case where these eigenvalues are all one we do indeed obtain $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi_n^2$, but in general this result does not hold. In fact, we can see that in general, the quadratic form is distributed as a weighted sum of chi-squared random variables each with one degree-of-freedom. The general distribution for this form is complicated and its density function does not have a closed form representation. Bodenham and Adams (2015) examine some approximations to this distribution, and provide comparisons with simulations.

Answer 2

Regarding 3. above, suppose $\Sigma_{n\times n}$ is diagonal with diagonal elements $\{3, 2, 1\}$, then $\Sigma$ is symmetric positive definite. ...but $z'\Sigma z$ cannot be $\chi^2_n$ (chi-squared with 3 df), because $E\left[z'\Sigma z\right]=3E(z_1^2)+2E(z_2^2)+E(z_3^2)=6$, but the mean of a chi-squared with 3 df is 3.