0

I was going through this answer which gives total sum of squares as

$TSS=\sum_i(Y_i-\hat{Y}_i)^2+\sum_i(\hat{Y}_i-\bar{Y})^2+2\sum_i(Y_i-\hat{Y}_i)(\hat{Y}_i-\bar{Y})$

It then says:

In a model with an intercept, it can be shown that the third term on the right hand side is zero, but this can not be shown for a model without an intercept (see e.g. D.N. Gujaratti, "Basic Econometrics" or W.H. Greene, "Econometric Analysis").

I am not able to find the proof of above quoted fact in Gujarati's book, nor am able to guess it my self. Can someone please give me proof and if possible some visual intuition of it?

(PS: am quite noob to stats)

Rnj
  • 145
  • 3
  • All it takes to prove such a negative statement is a counterexample. Contemplate the $(x,y)$ dataset $(1,0),$ $(1,2).$ For visual intuition, plot the points and plot the (obvious) fits with and without intercepts. – whuber Jul 15 '21 at 21:42
  • Does "without an intercept" mean "with the intercept constrained to equal 0" (or some other constant value)? – Alexis Jul 15 '21 at 22:31
  • 2
    @Alexis "Without an intercept" means literally that: there is no explicit constant term in the model. That's equivalent to including an intercept and constraining it to zero, of course. A synonymous phrase is "regression through the origin." – whuber Jul 15 '21 at 22:50
  • Thanks @whuber that was my suspicion, but was not sure if it sometimes had some other meaning. – Alexis Jul 15 '21 at 23:12
  • @whuber please help me out or at least point me to some link / source explaining intuition. Am unable to imagine as you said in first comment. – Rnj Jul 16 '21 at 08:59
  • For intuition, draw the plot. – whuber Jul 16 '21 at 11:34

1 Answers1

1

Are you familiar with linear algebra?

Let $Y=X\beta+\epsilon$. $\hat\beta=(X^TX)^{-1}X^TY$ and $\hat Y=X\hat\beta=PY$, where $P=X(X^TX)^{-1}X^T$. The $P$ matrix is symmetric and idempotent, it is the projection matrix of $Y$ onto $\mathcal{C}(X)$, the column space of $X$ (see Wikipedia).

Let $1$ be a column vector of ones and $J=11^T$ a square matrix of ones. You have: $$1^TY=\sum_iY_i,\qquad JY=\begin{bmatrix} \sum_iY_i \\ \vdots \\ \sum_iY_i \end{bmatrix}, \qquad \frac1n JY=\begin{bmatrix}\frac1n\sum_iY_i\\ \vdots \\ \frac1n\sum_iY_i\end{bmatrix}=\begin{bmatrix}\bar Y\\ \vdots \\ \bar Y\end{bmatrix}$$ The $n^{-1}J$ matrix is symmetric and idempotent.

In matrix form, $$\begin{align*}\sum_i(Y_i-\hat Y_i)^2&=(Y-PY)^T(Y-PY)\\ \sum_i(\hat Y_i-\bar Y)^2&=(PY-n^{-1}JY)^T(PY-n^{-1}JY)\\ 2\sum_i(Y_i-\hat Y_i)(\hat Y_i-\bar Y)&=2(Y-PY)^T(PY-n^{-1}JY) \\ &=2Y^T(I-P)^T(P-n^{-1}J)Y\end{align*}$$ where $I$ is the identity matrix.

$(I-P)^T(P-n^{-1}J)=P-n^{-1}J-P+Pn^{-1}J=Pn^{-1}J-n^{-1}J$. But $P$ is the projection matrix onto $\mathcal{C}(X)$, so if there is a column of ones in $X$, then each column of $J$ is in $\mathcal{C}(X)$, $PJ=J$ (each column in $J$ is projected onto itself) and $Pn^{-1}J=n^{-1}PJ=n^{-1}J$.

Therefore, $Pn^{-1}J-n^{-1}J=n^{-1}J-n^{-1}J=0$.

This is why "In a model with an intercept, it can be shown that the third term on the right hand side is zero." As to "this can not be shown for a model without an intercept" (notice that "this can not be shown" is not "this can not happen"), the third term could be zero even in a model without an intercept, but only if the columns in $X$ generate a vector of ones.

Sergio
  • 5,628
  • 2
  • 11
  • 27