What's the difference between standardization and studentization?

Question

Is it that in standardization variance is known while in studentization it is not known and therefore estimated? Thank you.

Answer 1

A short recap. Given a model $y=X\beta+\varepsilon$, where $X$ is $n\times p$, $\hat\beta=(X'X)^{-1}X'y$ and $\hat y=X\hat\beta=X(X'X)^{-1}X'y=Hy$, where $H=X(X'X)^{-1}X'$ is the "hat matrix". Residuals are $$e=y-\hat y=y-Hy=(I-H)y$$ The population variance $\sigma^2$ is unknown and can be estimated by $MSE$, the mean square error.

Semistudentized residuals are defined as $$e_i^*=\frac{e_i}{\sqrt{MSE}}$$ but, since the variance of residuals depends on both $\sigma^2$ and $X$, their estimated variance is: $$\widehat V(e_i)=MSE(1-h_{ii})$$ where $h_{ii}$ is the $i$th diagonal element of the hat matrix.

Standardized residuals, also called internally studentized residuals, are: $$r_i=\frac{e_i}{\sqrt{MSE(1-h_{ii})}}$$

However the single $e_i$ and $MSE$ are non independent, so $r_i$ can't have a $t$ distribution. The procedure is then to delete the $i$th observation, fit the regression function to the remaining $n-1$ observations, and get new $\hat y$'s which can be denoted by $\hat y_{i(i)}$. The difference: $$d_i=y_i-\hat y_{i(i)}$$ is called deleted residual. An equivalent expression that does not require a recomputation is: $$d_i=\frac{e_i}{1-h_{ii}}$$ Denoting the new $X$ and $MSE$ by $X_{(i)}$ and $MSE_{(i)}$, since they do not depend on the $i$th observation, we get: $$t_i=\frac{d_i}{\sqrt{\frac{MSE_{(i)}}{1-h_{ii}}}} =\frac{e_i}{\sqrt{MSE_{(i)}(1-h_{ii})}}\sim t_{n-p-1}$$ The $t_i$'s are called studentized (deleted) residuals, or externally studentized residuals.

See Kutner et al., Applied Linear Statistical Models, Chapter 10.

Edit: I must say that the answer by rpierce is perfect. I thought that the OP was about standardized and studentized residuals (and dividing by the population standard deviation to get standardized residuals looked odd to me, of course), but I was wrong. I hope that my answer can help someone even if OT.

Answer 2

In social sciences it is typically said that Studentizated scores uses Student's/Gosset's calculation for estimating the population variance/standard deviation from the sample variance/standard deviation ($s$). In contrast, Standardized scores (a noun, a particular type of statistic, the Z score) are said to use the population standard deviation ?($\sigma$).

However, it appears there is some terminological differences across fields (please see the comments on this answer). Therefore, one ought to proceed with caution in making these distinctions. Moreover, studentized scores are rarely called such and one typically sees 'studentized' values in the context of regression. @Sergio provides details about those types of studentized deleted residuals in his answer.

Answer 3

I am very late in answering this question!!. But couldn't find the answer in very simple language so humble attempt to answer this.

Why we do standardization? Imagine you have two models-one predicts craziness from amount of time spent on studying statistics while other predicts log(craziness) with amount of time on statistics.

it would be hard to understand residuals are both are in different units. So we standardize them .(similar theory as Z-score )

Standardized residuals: - When residuals are divided by an estimate of standard deviation . In general if absolute value > 3 then it's cause of concern.

We use this to investigate outliers in model.

Studentized Residual: We use this to study stability of model.

Process is simple. We remove individual test case from model and find out the new predicted value. Difference between new value and original observed value can be standardized by dividing standard error. this value is Studentized Residual

For more infö discovering statics using R -http://www.statisticshell.com/html/dsur.html

Answer 4

Wikipedia has a good overview at https://en.wikipedia.org/wiki/Normalization_(statistics):

Standard score $\frac{X - \mu}{\sigma}$ : Normalizing errors when population parameters are known. Works well for populations that are normally distributed

Student's t-statistic $\frac{X - \overline{X}}{s}$ : Normalizing residuals when population parameters are unknown (estimated).