2

The following question is from Kutner's Applied Linear Statistical Models - Ch 2 - 2.12

enter image description here

To answer the question a few pieces of information are needed, provided below:

enter image description here

enter image description here

enter image description here

What I gather the question is asking me is that if I take the limit as $n \to \infty$ then what happens to the variance of my new prediction?

Now clearly based on the expressions below, it would appear that in both cases $Var(pred)$ and $Var(\hat{Y_{h}})$ would both be $0$. And as such they can be brought increasingly close to $0$.

But the solution says that this is not the case and I'm wondering why? Is it because they specified the theoretical variances? i.e $\sigma^{2}(pred)$ and $\sigma^{2}(\hat{Y_{h}})$ and I took the estimates by using the sample variances for the respective pieces? But even still the theoretical variances still involve having an $n$ in the denominator.....Am I looking at something wrong?

dc3rd
  • 272
  • 1
  • 3
  • 13

1 Answers1

2

As a preliminary note, I'm not a fan of the notation used in the excerpted material, so I'll avoid using most of it. In any case, as you can see from the formulae, the difference between the variance of the prediced value and the variance of a new value is the variance of the error term in the regression, which is estimated by the $\text{MSE}$ term. So the natural mathematical question here is, what is the limit $\lim_{n \rightarrow \infty} \text{MSE}_n$? (Hint: it is not generally equal to zero.) If you can answer that then you can find the difference in the limits of the two variances.

Stepping back from the mathematics of this, to look at the intuition, it is worth noting that a new value in the regression model will have variability that depends on two things. There is some variability coming from the fact that you don't know the true expected value in the regression (and you estimate this with the predicted value) and then there is additional variance coming from the fact that the new observation will deviate from the true expected value because of the "error term". (This occurs by definition; we define the error term in the model as the deviation of the value from its true expectation.)

If you have a careful look at the behaviour of the MSE, you should be able to find the two variance limits of interest, and it should be possible to explain this difference intuitively by considering the difference between a new point in the regression and the expected value of the response variable for that point.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • 1
    The author has already done you a big favour by telling you that $\text{MSE}_n = \frac{1}{n-2} \sum_{i=1}^n e_i^2$. If you can determine the behaviour of the residual vector $\mathbf{e}$, it might be easier to use this expression. – Ben Mar 19 '21 at 05:10
  • 1
    yes, that is what I expanded. I'm also editing my comment, even though it is wrong, I may run out of time before the edit is complete, so I will post it again just for posterity – dc3rd Mar 19 '21 at 05:12
  • Thanks for the explanation. There are still some pieces I'm not clear about. 1) If I expand the MSE after a bunch of algebra I get an expression of the form: $\frac{\sum Y_{i}^{2}}{n(1-\frac{2}{n})} - \frac{2\sum Y_{i}\hat{Y_{i}}}{n(1-\frac{2}{n})} + \frac{\sum \hat{Y_{i}^{2}}}{n(1-\frac{2}{n})}$. Can I say something for instance about the first and third terms? What I had in mind was $\frac{\sum Y_{i}^{2}}{n(1-\frac{2}{n})} = \bar{Y_{i}^{2}}(\frac{1}{n(1-\frac{2}{n})})$. Then taking that limit I would have $\bar{Y_{i}^{2}}$ and the same idea for the third term (this was my original reply) – dc3rd Mar 19 '21 at 05:16
  • I'm sorry I don't completely get what you mean by determining the behaviour of the residual vector. What I do know is the sum of the residuals will be zero, as well the sum of the squared residuals, $e_{i}^{2}$ is minimized. It is what we minimize to find our parameter estimates.....but I'm not sure what to look for beyond that. – dc3rd Mar 19 '21 at 05:38
  • 1
    Okay, well I think that is what you're going to need to solve this. I recommend you have a bit more of a look into the properties of regression models, with particular emphasis on the distribution of the residual vector and the limiting properties of the MSE. You might also want to look at the Grenander conditions (see similar post [here](https://stats.stackexchange.com/questions/491002/)). – Ben Mar 19 '21 at 07:52
  • I'll go back and revisit it. I'm purposely avoiding doing the linear algebra approach for the moment because it is the introductory sections and I want to understand what is happening fundamentally before extending the complexity. – dc3rd Mar 19 '21 at 20:30
  • 1
    Whilst the actual proof is non-trivial, and requires some linear algebra, the essence of what is happening is this: The residuals are essentially just estimators of the error terms in the model, and the MSE is like the "sample varriance" of these values, so it estimates the true variance of the error terms in the model. Under mild asymptotic assumptions this "sample variance" of the residuals converges to the true variance of the error terms, so you should get $\text{MSE}_n \rightarrow \sigma^2$. This is the part of the variance of the new values that cannot be reduced to zero. – Ben Mar 19 '21 at 20:33
  • 1
    This also reflects the fact that new values in the regression are always going to have some variability (through the variance of the error terms), even if you have so much data that you have perfectly estimated the true regression line. – Ben Mar 19 '21 at 20:34
  • 1
    This gives me a lot more clarity of what is happening. I'm not sure why it never clicked that the MSE is a "sample variance" of sorts. I'm also not sure how they expected me to get this idea in the textbook I'm working from considering no convergence theorems have been introduced, let alone taking limits of vectors.Thank you for taking the time to explain this to me. I will most likely ask you for assistance and guidance in the future if that is ok with you? because I looked at your profile and you are at the spot where I aspire to get to career/academic wise. – dc3rd Mar 19 '21 at 20:44
  • 1
    That is fine --- happy to help. – Ben Mar 19 '21 at 20:46