Why divide by 1-leverage?

Question

I'm reading about resampling methods, and specifically leave-one-out cross-validation.

I understood the method, and how to calculate the estimate of the test MSE (Mean squared error):

In the setup of linear regression we have:

$h_i$ is the leverage of the $i$th observation. I want to know the utility of dividing by $1-h_i$. And why not just use the normal formula of MSE?

Can you please provide a reference from where you read this? It will help contextualising your question better. — usεr11852, Dec 01 '18 at 11:48
Of course. I read this in Introduction to statistical learning by Hastie and Tibshirani. — Youssef Esseddiq, Dec 01 '18 at 11:51
Thank you. What confuses you about what the authors explain as: "*the residuals for high-leverage points are inflated in this formula by exactly the right amount for this equality to hold*". Here by residuals we mean the quantity $y_i - \hat{y}_i$. In general, this form of CV is somewhat uncommon. I think it will help you if you view it in conjunction with generalised cross validation (GCV). — usεr11852, Dec 01 '18 at 11:59
Yes, that quote was unclear for me. For observations with high leverage we are dividing by that quantity which makes the ith term of the sum bigger, while low leverage observation get a slight push. It seems like if it is some kind of way to make the model not underestimate the effect of high leverage points via inflating the MSE. Thank you for your comment I'll try to look at the GCV. — Youssef Esseddiq, Dec 01 '18 at 12:21
The variance of $y_i-\hat{y}_i$ is $\sigma^2(1-h_{ii})$ (a number of posts on site discuss that); consequently, the variance of $\frac{y_i-\hat{y}_i}{1-h_{ii}}$ is $\sigma^2$. It's a way of scaling residuals to have equal variance. — Glen_b, Dec 02 '18 at 09:05

kjetil b halvorsen · Answer 1 · 2019-04-19T22:08:00.490

The $i$th residual is $y_i-\hat{y}_i$, and it has variance $(1-h_{ii})\sigma^2$ (see Leverages and effect of leverage points for this and other algebra used here) so dividing by $\sqrt{1-h_{ii}}$ we get a constant variance: $$ \DeclareMathOperator{\V}{\mathbb{V}} \V \frac{y_i-\hat{y}_i}{\sqrt{1-h_{ii}}} = \frac1{1-h_{ii}} \sigma^2 (1-h_{ii})=\sigma^2 $$ so the variance is indeed constant. So I suspect the squares in the summation should not include also the denominator, or there is missing a square root. This is related to the concept of a studentized residual, see https://en.wikipedia.org/wiki/Studentized_residual.

Why divide by 1-leverage?

1 Answers1