Interpretation of R's lm() output

Question

The help pages in R assume I know what those numbers mean, but I don't. I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There might (will) be mistakes, as I'll just write what I assume. Mainly I'd like to know what the t-value in the coefficients mean, and why they print the residual standard error.

Call:
lm(formula = iris$Sepal.Width ~ iris$Petal.Width)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.09907 -0.23626 -0.01064  0.23345  1.17532 

This is a 5-point-summary of the residuals (their mean is always 0, right?). The numbers can be used (I'm guessing here) to quickly see if there are any big outliers. Also you can already see it here if the residuals are far from normally distributed (they should be normally distributed).

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)       3.30843    0.06210  53.278  < 2e-16 ***
iris$Petal.Width -0.20936    0.04374  -4.786 4.07e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Estimates $\hat{\beta_i}$, computed by least squares regression. Also, the standard error is $\sigma_{\beta_i}$. I'd like to know how this is calculated. I have no idea where the t-value and the corresponding p-value come from. I know $\hat{\beta}$ should be normal distributed, but how is the t-value calculated?

Residual standard error: 0.407 on 148 degrees of freedom

$\sqrt{ \frac{1}{n-p} \epsilon^T\epsilon }$, I guess. But why do we calculate that, and what does it tell us?

Multiple R-squared: 0.134,  Adjusted R-squared: 0.1282 

$ R^2 = \frac{s_\hat{y}^2}{s_y^2} $, which is $ \frac{\sum_{i=1}^n (\hat{y_i}-\bar{y})^2}{\sum_{i=1}^n (y_i-\bar{y})^2} $. The ratio is close to 1 if the points lie on a straight line, and 0 if they are random. What is the adjusted R-squared?

F-statistic: 22.91 on 1 and 148 DF,  p-value: 4.073e-06 

F and p for the whole model, not only for single $\beta_i$s as previous. The F value is $ \frac{s^2_{\hat{y}}}{\sum\epsilon_i} $. The bigger it grows, the more unlikely it is that the $\beta$'s do not have any effect at all.

Answer 1

Five point summary

yes, the idea is to give a quick summary of the distribution. It should be roughly symmetrical about mean, the median should be close to 0, the 1Q and 3Q values should ideally be roughly similar values.

Coefficients and $\hat{\beta_i}s$

Each coefficient in the model is a Gaussian (Normal) random variable. The $\hat{\beta_i}$ is the estimate of the mean of the distribution of that random variable, and the standard error is the square root of the variance of that distribution. It is a measure of the uncertainty in the estimate of the $\hat{\beta_i}$.

You can look at how these are computed (well the mathematical formulae used) on Wikipedia. Note that any self-respecting stats programme will not use the standard mathematical equations to compute the $\hat{\beta_i}$ because doing them on a computer can lead to a large loss of precision in the computations.

$t$-statistics

The $t$ statistics are the estimates ($\hat{\beta_i}$) divided by their standard errors ($\hat{\sigma_i}$), e.g. $t_i = \frac{\hat{\beta_i}}{\hat{\sigma_i}}$. Assuming you have the same model in object modas your Q:

> mod <- lm(Sepal.Width ~ Petal.Width, data = iris)

then the $t$ values R reports are computed as:

> tstats <- coef(mod) / sqrt(diag(vcov(mod)))
(Intercept) Petal.Width 
  53.277950   -4.786461 

Where coef(mod) are the $\hat{\beta_i}$, and sqrt(diag(vcov(mod))) gives the square roots of the diagonal elements of the covariance matrix of the model parameters, which are the standard errors of the parameters ($\hat{\sigma_i}$).

The p-value is the probability of achieving a $|t|$ as large as or larger than the observed absolute t value if the null hypothesis ($H_0$) was true, where $H_0$ is $\beta_i = 0$. They are computed as (using tstats from above):

> 2 * pt(abs(tstats), df = df.residual(mod), lower.tail = FALSE)
 (Intercept)  Petal.Width 
1.835999e-98 4.073229e-06

So we compute the upper tail probability of achieving the $t$ values we did from a $t$ distribution with degrees of freedom equal to the residual degrees of freedom of the model. This represents the probability of achieving a $t$ value greater than the absolute values of the observed $t$s. It is multiplied by 2, because of course $t$ can be large in the negative direction too.

Residual standard error

The residual standard error is an estimate of the parameter $\sigma$. The assumption in ordinary least squares is that the residuals are individually described by a Gaussian (normal) distribution with mean 0 and standard deviation $\sigma$. The $\sigma$ relates to the constant variance assumption; each residual has the same variance and that variance is equal to $\sigma^2$.

Adjusted $R^2$

Adjusted $R^2$ is computed as:

$$1 - (1 - R^2) \frac{n - 1}{n - p - 1}$$

The adjusted $R^2$ is the same thing as $R^2$, but adjusted for the complexity (i.e. the number of parameters) of the model. Given a model with a single parameter, with a certain $R^2$, if we add another parameter to this model, the $R^2$ of the new model has to increase, even if the added parameter has no statistical power. The adjusted $R^2$ accounts for this by including the number of parameters in the model.

$F$-statistic

The $F$ is the ratio of two variances ($SSR/SSE$), the variance explained by the parameters in the model (sum of squares of regression, SSR) and the residual or unexplained variance (sum of squares of error, SSE). You can see this better if we get the ANOVA table for the model via anova():

> anova(mod)
Analysis of Variance Table

Response: Sepal.Width
             Df  Sum Sq Mean Sq F value    Pr(>F)    
Petal.Width   1  3.7945  3.7945   22.91 4.073e-06 ***
Residuals   148 24.5124  0.1656                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The $F$s are the same in the ANOVA output and the summary(mod) output. The Mean Sq column contains the two variances and $3.7945 / 0.1656 = 22.91$. We can compute the probability of achieving an $F$ that large under the null hypothesis of no effect, from an $F$-distribution with 1 and 148 degrees of freedom. This is what is reported in the final column of the ANOVA table. In the simple case of a single, continuous predictor (as per your example), $F = t_{\mathrm{Petal.Width}}^2$, which is why the p-values are the same. This equivalence only holds in this simple case.

Answer 2

Ronen Israel and Adrienne Ross (AQR) wrote a very nice paper on this subject: Measuring Factor Exposures: Uses and Abuses.

To summarize (see: p. 8),

• Generally, the higher the $R^2$ the better the model explains portfolio returns.
• When the t-statistic is greater than two, we can say with 95% confidence (or a 5% chance we are wrong) that the beta estimate is statistically different than zero. In other words, we can say that a portfolio has significant exposure to a factor.

R's lm() summary calculates the p-value Pr(>|t|). The smaller the p-value is, the more significant the factor is. P-value = 0.05 is a reasonable threshold.