8

I am having some problem trying to prove that the diagonal elements of the hat matrix $h_{ii}$ are between $1/n$ and $1$.

Suppose that $Range(X_{n,k})=K $ the number of columns of our matrix of data with a constant.⇒$H_{k,k}$

$H=X(X' X)^{-1}X' ⇒ H'=H ;H^{2}=H $

If $y = \beta x + \epsilon ⇒ HY =\hat Y; (I-H)=\epsilon $

$\mathbf{H}=\begin{bmatrix}h_{11} &... &h_{1n}\\⋮ & ⋱ &⋮\\ h_{n1} & ... & h_{nn}\end{bmatrix}$

If $\boldsymbol 1=(1, \ldots, 1) ∈ X ⇒ 1H^2=1H=1 ⇒ ∑ h_{i1}^2=∑ h_{i1}=1 , h_{ii}≤1$

So $∑h_{i1}^2∑h_{i2}^2...∑h_{in}^2=1 ⇒h_{11}^2h_{22}^2...h_{nn}^2<=1 $

How to prove that $h_{ii} >=(1/n)$?

(exercise 3.4 from Meyer "Classical and modern regression with applications") Let $h_{ii}$ be the ith diagonal of the Hat matrix H. (a) prove that for a multiple regression model with a constant term, $h_{ii} >=(1/n)$ ; (b) show that $h_{ii}≤1$ (Hint: make the use of the fact that H is idempotent)

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
EAguirre
  • 117
  • 1
  • 1
  • 13
  • 3
    The result is not generally true: the diagonal elements can be less than $1/n$ when $X'X$ is not of full rank (and the generalized inverse is used). – whuber Jun 17 '13 at 14:33
  • 1
    $H$, as a projection matrix, is rarely of full rank. Even when $X'X$ is invertible, your conclusion is incorrect. Consider $X=(1,2)'$, where $X'X=(5)$, $(X'X)^{-1}=(1/5)$, and $H=X(X'X)^{-1}X'=\left( \begin{array}{cc} \frac{1}{5} & \frac{2}{5} \\ \frac{2}{5} & \frac{4}{5} \end{array} \right)$ has a diagonal entry less than $1/n=1/2$. – whuber Jun 17 '13 at 14:47
  • 2
    Please don't put your assumptions into comments: edit the question to include all the assumptions you wish to make. – whuber Jun 17 '13 at 14:52
  • 1
    $H$ will be of full rank only when $X$ is square: in that case you will no longer be doing least squares, but merely solving a completely determined set of linear equations. In general, the rank of $H$ does not exceed the number of columns of the design matrix $X$. In my counterexample, $X$ has one column and the rank of $H$ is one: as large as possible. – whuber Jun 17 '13 at 16:53
  • @whuber you are right my apollogies. When I said full range I mean that the range(X)= the number of regressor – EAguirre Jun 17 '13 at 17:33
  • let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/9271/discussion-between-eaguirre-and-whuber) – whuber Jun 17 '13 at 22:09

3 Answers3

7

This is several years later, but I found the notation very difficult in the asker's question and self-answer, so here's a cleaner solution.

We have $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ where $(1,...,1)^T$ is a column of $\mathbf{X}$. We want to show that the diagonals $h_{ii}$ of $\mathbf{H}$ have $h_{ii} \geq 1/n$.

Define $\mathbf{P} = \mathbf{H} - \mathbf{C}$, where

$$\mathbf{C} = \frac{1}{n}\begin{pmatrix}1 & \dots & 1 \\ \vdots & \ddots & \vdots \\ 1 & \dots & 1 \end{pmatrix}$$

the matrix consisting of only $1/n$. This is the projection matrix onto the space spanned by $(1, ..., 1)$. Then

$$\mathbf{P}^2 = \mathbf{H}^2 - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}^2 = \mathbf{H} - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}$$

However, $\mathbf{H}$ orthogonally projects onto $\text{Col}(\mathbf{X})$, and $\mathbf{C}$ orthogonally projects onto $\text{span}\{(1,...,1)\} \subset \text{Col}(\mathbf{X})$, so obviously $\mathbf{H}\mathbf{C} = \mathbf{C}$. Still intuitively, but less obviously, $\mathbf{C}\mathbf{H} = \mathbf{C}$. To see this, we can compute $\mathbf{C} = \mathbf{C}\big(\mathbf{H} + (\mathbf{I} - \mathbf{H})\big)$, and note that $\mathbf{C}(\mathbf{I} - \mathbf{H}) = 0$ because $\mathbf{I} - \mathbf{H}$ projects onto $\text{Col}(\mathbf{X})^\perp$.

Therefore we have $\mathbf{P}^2 = \mathbf{H} - \mathbf{C} = \mathbf{P}$. So $\mathbf{P}$ is also a projection matrix.

So $h_{ii} = p_{ii} + c_{ii} = p_{ii} + 1/n$. Since projection matrices are always positive semidefinite, the diagonals of $\mathbf{P}$ satisfy $p_{ii} \geq 0$. (In fact, you can show that since $\mathbf{P}$ is symmetric and idempotent, it satisfies $0 \leq p_{ii} \leq 1$.)

Then $h_{ii} \geq 1/n$ as needed.

Drew N
  • 490
  • 3
  • 10
3

For prove that $h_{ii} >=(1/n)$, we can center $H_c=X(X_c' X_c)^{-1}X_c'$ , $\mathbf{H_c}=\begin{bmatrix}x_{11}-\bar x_1 &... &x_{1n}-\bar x_1 \\⋮ & ⋱ &⋮\\ x_{n1}-\bar x_n & ... & x_{nn}-\bar x_n\end{bmatrix}$

$y=\alpha1+ X_c'\beta +\epsilon⇒ \hat y=\hat \alpha1+ X_c'\hat\beta ⇒ \hat y=\bar y+ X_c'\hat\beta= \bar y+ X_c'(X_c' X_c)^{-1}X_c'y⇒ \hat y=[(1/n) 1'y]1+H_cy$ $=[1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c ] y=Hy $

Then $ H=1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c $⇒ $h_{ii} >=(1/n)$ because $H_c$ is a positive definite matrix

EAguirre
  • 117
  • 1
  • 1
  • 13
3

Here is another answer that that only uses the fact that all the eigenvalues of a symmetric idempotent matrix are at most 1, see one of the previous answers or prove it yourself, it's quite easy.

Let $H$ denote the hat matrix. The $i$th diagonal element of the hat matrix is given by

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i,$$

where $\mathbf{e}_i^{t}$ is the vector whose $i$th element is 1 and the rest are 0s. Consider the quadratic form on the unit sphere given by

$$ f(\mathbf{x}) = \frac{\mathbf{x}^{t} \mathbf{H} \mathbf{x}}{\mathbf{x}^{t} \mathbf{x}}. $$

It is well known that the maximum of this expression is $\lambda_n$, the largest eigenvalue of the matrix $\mathbf{H}$. Returning to the diagonal elements of the hat matrix, one therefore has

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i = \frac{\mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i}{\mathbf{e}_i^{t} \mathbf{e}_i} \underbrace{\mathbf{e}_i^{t} \mathbf{e}_i}_{ = 1} \leq \lambda_n \leq1 $$

and this gives us what we need.

JohnK
  • 18,298
  • 10
  • 60
  • 103