11

Let $Z_1,\cdots,Z_n$ be independent standard normal random variables. There are many (lengthy) proofs out there, showing that

$$ \sum_{i=1}^n \left(Z_i - \frac{1}{n}\sum_{j=1}^n Z_j \right)^2 \sim \chi^2_{n-1} $$

Many proofs are quite long and some of them use induction (e.g. Casella Statistical Inference). I am wondering if there is any easy proof of this result.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
3x89g2
  • 1,366
  • 1
  • 11
  • 26
  • For an intuitive geometric (coordinate-free) approach, look at Section 1.2 of the excellent text *The Coordinate-Free Approach to Linear Models* by Michael J. Wichura (the technical details are filled in Theorem 8.2), where the author actually compared the traditional matrix proof (provided by whuber's answer) and his projection approach, showing that his geometric approach is more natural and less obscure. Personally, I think this proof is insightful and succinct. – Zhanxiong Dec 17 '17 at 04:25

2 Answers2

11

For $k=1, 2, \ldots, n-1$, define

$$X_k = (Z_1 + Z_2 + \cdots + Z_k - kZ_{k+1})/\sqrt{k+k^2}.$$

The $X_k$, being linear transformations of multinormally distributed random variables $Z_i$, also have a multinormal distribution. Note that

  1. The variance-covariance matrix of $(X_1, X_2, \ldots, X_{n-1})$ is the $n-1\times n-1$ identity matrix.

  2. $X_1^2 + X_2^2 + \cdots + X_{n-1}^2 = \sum_{i=1}^n (Z_i-\bar Z)^2.$

$(1)$, which is easy to check, directly implies $(2)$ upon observing all the $X_k$ are uncorrelated with $\bar Z.$ The calculations all come down to the fact that $1+1+\cdots+1 - k = 0$, where there are $k$ ones.

Together these show that $\sum_{i=1}^n(Z_i-\bar Z)^2$ has the distribution of the sum of $n-1$ uncorrelated unit-variance Normal variables. By definition, this is the $\chi^2(n-1)$ distribution, QED.

References

  1. For an explanation of where the construction of $X_k$ comes from, see the beginning of my answer at How to perform isometric log-ratio transformation concerning Helmert matrices.

  2. This is a simplification of the general demonstration given in ocram's answer at Why is RSS distributed chi square times n-p. That answer asserts "there exists a matrix" to construct the $X_k$; here, I exhibit such a matrix.

Community
  • 1
whuber
  • 281,159
  • 54
  • 637
  • 1,101
  • 2
    This construction has a simple geometric interpretation. (1) The variables $Z_i$ are distributed on a n-dimensional spherically symmetric distribution (thus we can rotate it any way we like). (2) The $\overline{Z}$ is found as a solution to the linear problem $Z_i = \overline{Z} + \epsilon_i$, which is effectively a projection of the vector $\mathbf{Z}$ onto $\mathbf{1}$. (3) If we rotate the coordinate space such that one of the coordinates coincides with this projection vector, $\mathbf{1}$, then the remainder is a (n-1)-multinomial distribution representing the residual space. – Sextus Empiricus Nov 13 '17 at 22:17
  • You show that the $X_i$'s are uncorrelated with each other. But as far as I understand, to say that a sum of squared standard normal variables is $\chi^2$, we need independence, which is a much stronger requirement than uncorrelated? EDIT: oh wait, if we know that two variables are normally distributed, then uncorrelated implies independence. – user56834 Nov 14 '17 at 06:09
  • Also, I don't understand how you go from the fact that the $X_i$'s are uncorrelated with $\bar Z$ (which I do understand), to (2). Could you elaborate? – user56834 Nov 14 '17 at 06:14
  • @Programmer Sorry; I didn't mean to imply it's a logical deduction--(1) and (2) are two separate observations. (2) is merely a (straightforward) algebraic identity. – whuber Nov 14 '17 at 14:39
  • Ok. I was misled by the "directly implies". But I'm sorry to say, after working at it for a while, I can't see why the identity is true? Could you show it? – user56834 Nov 14 '17 at 15:18
  • @Programmer It really is just algebra and it really is simple. The easiest way to make the demonstration is as a matrix identity. One way to convince yourself is to work it out for small $n$. (I trust that I have recorded the result correctly--there's always the chance of typos--but checking with small $n$ will clarify the relationship, regardless.) – whuber Nov 14 '17 at 15:23
  • @whuber, This may be disconcerting but I really can't seem to see it. I worked out the algebra and got to the wrong answer. Could you maybe just quickly give me the matrix formula as a hint? – user56834 Nov 14 '17 at 15:48
  • 2
    Programmer, note the link to the other answer that Whuber gave ( https://stats.stackexchange.com/questions/259208/how-to-perform-isometric-log-ratio-transformation/259223#259223 ) The $X_k$ are constructed based on a matrix, $H$, with orthogonal rows. So you can evaluate in a more abstract (less fallible) way $\sum K_i^2$ as $K \cdot K = (H Z) \cdot (H Z) = (HZ)^T (HZ) = Z^T (H^TH) Z= Z^T I Z = Z \cdot Z$, (note we have to extend K by the vector 1111 to make it n by n) – Sextus Empiricus Nov 17 '17 at 17:16
6

Note you say $Z_is$ are iid with standard normal $N(0,1)$, with $\mu=0$ and $\sigma=1$

Then $Z_i^2\sim \chi^2_{(1)}$

Then \begin{align}\sum_{i=1}^n Z_i^2&=\sum_{i=1}^n(Z_i-\bar{Z}+\bar{Z})^2=\sum_{i=1}^n(Z_i-\bar{Z})^2+n\bar{Z}^2\\&=\sum_{i=1}^n(Z_i-\bar{Z})^2+\left[\frac{\sqrt{n}(\bar{Z}-0)}{1}\right ]^2 \tag{1} \end{align}

Note that the left hand side of (1), $$\sum_{i=1}^n Z_i^2\sim\chi^2_{(n)}$$ and that the second term on the right hand side $$\left[\frac{\sqrt{n}(\bar{Z}-0)}{1}\right ]^2 \sim\chi^2_{(1)}.$$

Furthermore $\operatorname{Cov}(Z_i-\bar Z,\bar Z)=0$ such that $Z_i-\bar Z$ and $\bar Z$ are independent. Therefore the two last terms in (1) (functions of $Z_i-\bar Z$ and $Z_i$) are also independent. Their mgfs are therefore related to the mgf of the left hand side of (1) through $$ M_n(t) = M_{n-1}(t)M_1(t) $$ where $M_n(t)=(1-2t)^{-n/2}$ and $M_1(t)=(1-2t)^{-1/2}$. The mgf of $\sum_{i=1}^n(Z_i-\bar{Z})^2$ is therefore $M_{n-1}(t)=M_n(t)/M_1(t)=(1-2t)^{-(n-1)/2}$. Thus, $\sum_{i=1}^n(Z_i-\bar{Z})^2$ is chi-square with $n-1$ degrees of freedom.

Jarle Tufto
  • 7,989
  • 1
  • 20
  • 36
Deep North
  • 4,527
  • 2
  • 18
  • 38
  • 1
    The last "Therefore" is too careless – Zhanxiong Nov 07 '17 at 03:09
  • The independent can be seen from standard deivation is in dependent of $\bar{X}$ – Deep North Nov 07 '17 at 03:35
  • 2
    "standard deviation independent of $\bar{X}$"? Maybe what you wanted to say is $\sum Z_i^2$ is independent of $\bar{Z}$. Unfortunately, this is not true. What really holds is $\sum (Z_i - \bar{Z})^2$ is independent of $\bar{Z}$, which is also a part of the proof we need to complete (instead of using it when showing this proposition). – Zhanxiong Nov 07 '17 at 03:54
  • I think I used Cochran’s Theorem – Deep North Nov 07 '17 at 04:26
  • 3
    @DeepNorth If filled in some missing pieces in your proof – Jarle Tufto Nov 07 '17 at 16:45