4

Consider the least squares problem $Y=X\beta +\epsilon$ while $\epsilon$ is zero mean Gaussian with $E(\epsilon) = 0$ and variance $\sigma^2$. I need to prove that

$\frac{V(\hat{\beta})}{N-(n+m)}$ is an unbiased estimate of $\sigma^2$ with $V(\beta) = ||Y-X\beta||$ .

I wasn't able to find the answer online. I just got confused by a thousand different ways to write things down.

EDIT:
$Y = \begin{pmatrix} y(0)\\ \vdots \\ y(N-1)\end{pmatrix} \quad$ $X = \begin{pmatrix} x^T(0)\\ \vdots \\ x^T(N-1)\end{pmatrix}\quad $ $\beta = \begin{pmatrix} a_1\\ \vdots \\ a_n\\ b_1 \\\vdots \\ b_m \end{pmatrix}$

Glen_b
  • 257,508
  • 32
  • 553
  • 939
bananamanana
  • 43
  • 1
  • 5
  • 1
    What are N, n, and m? – ocram Nov 16 '13 at 16:39
  • Sorry, I updated my question – bananamanana Nov 16 '13 at 16:45
  • 1
    I still don't quite follow your $n$ & $m$, & the way you are representing the matrices is unusual for me. I'm more familiar w/:$$Y=\begin{pmatrix}y_1\\ \vdots\\ y_N\end{pmatrix},\quad X=\begin{pmatrix}1 &x_{11}&\cdots&x_{1p}\\ \vdots&\vdots&\ddots&\vdots\\ 1 &x_{N1}&\cdots&x_{Np}\end{pmatrix},\quad\beta=\begin{pmatrix}\beta_0\\ \vdots\\ \beta_p\end{pmatrix},\quad\varepsilon=\begin{pmatrix}\varepsilon_1\\ \vdots\\ \varepsilon_N\end{pmatrix}$$ – gung - Reinstate Monica Nov 16 '13 at 17:05
  • The notation was given to me (at the university), but it is quite the same as x are vectors and p=m+n.. – bananamanana Nov 16 '13 at 17:07
  • 1
    On the basis of this comment combined with details in your question, I've added the `self-study` tag. Please read its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) and understand what is expected for this sort of question and the limitations on the kinds of answers you should expect. While you can ask about course-related work (or even work you're just doing for your own study purposes), CV isn't a site to just do your study for you. – Glen_b Nov 16 '13 at 17:20

1 Answers1

5

Following your notations, we have $$V(\hat{\beta}) = \|\hat{\epsilon}\|^2 = \text{RSS}$$ i.e., the Residual Sum of Squares.

It is a fact that (cf. here) $$\frac{\text{RSS}}{\sigma²} \sim \chi_{(N-p)}^2$$ with $N$ the total sample size and $p$ the number of parameters in $\beta$ (here, $p = n + m$).

The result follows from the fact that the expectation of a chi-square random variable equals its number of degrees of freedom, i.e., $$ \text{E}\left(\frac{\text{RSS}}{\sigma²}\right) = N - p $$ which can be rewritten as $$ \text{E}\left(\frac{\text{RSS}}{N-p}\right) = \sigma² $$ since $N-p$ and $\sigma²$ are both non-random.

Community
  • 1
ocram
  • 19,898
  • 5
  • 76
  • 77
  • I cant follow why $V(\hat{\beta})$ is $||\hat{\epsilon}||^2$. What exactly is $\hat{\epsilon}$? – bananamanana Nov 16 '13 at 17:13
  • $\|v\| = \sum_{\ell=1}^L v_\ell^2$ for any vector $v=(v_1 \dotsc v_L)$. Thus $V(\hat{\beta}) = \|Y - X \hat{\beta}\|$ is the sum of squared residuals, which I have denoted by $\|\hat{\epsilon}\|$. – ocram Nov 16 '13 at 17:30