Heteroscedasticity and non-normal errors are only an issue when predicting from a linear model - why?

Question

In Regression and Other Stories, the authors state that heteroscedasticity and non-normal errors are only problematic when predicting from a linear model (1; p. 154-155):

5. Equal variance of errors. Unequal error variance (also called heteroscedasticity, in contrast to equal variances, or homoscedasticity) can be an issue when a regression is used for probabilistic prediction, but it does not affect what is typically the most important aspect of a regression model, which is the information that goes into the predictors and how they are combined. If the variance of the regression errors are unequal, estimation is more efficiently performed by accounting for this in the model, as with weighted least squares discussed in Section 10.8. In most cases, however, this issue is minor.
6. Normality of errors. The distribution of the error term is relevant when predicting individual data points. For the purpose of estimating the regression line (as compared to predicting individual data points), the assumption of normality is typically barely important at all. Thus we do not recommend diagnostics of the normality of regression residuals. For example, many textbooks recommend quantile-quantile (Q-Q) plots, in which the ordered residuals are plotted vs. the corresponding expected values of ordered draws from a normal distribution, with departures of this plot from linearity indicating nonnormality of the error term. There is nothing wrong with making such a plot, and it can be relevant when evaluating the use of the model for predicting individual data points, but we are typically more concerned with the assumptions of validity, representativeness, additivity, linearity, and so on, listed above.

My understanding from the text above is that the authors think violations of these assumptions are not an issue for the estimation of model coefficients. Why are violations of these assumptions only an issue for prediction?

Non-normal errors are discussed in this Cross Validated answer, however I would appreciate an answer that refers more to the underlying mathematics or refers to external sources.

Reference:

1. A. Gelman, J. Hill, A. Vehtari, Regression and Other Stories (Cambridge University Press, 2020) https:/doi.org/10.1017/9781139161879.

Answer 1

I suspect that Gelman and Hill are either overstating the case concerning whether normality is an issue, or that their comments are taken out of context. While linearity (or more precisely, correct functional specification) is usually more important, there are cases of non-normality that indeed should be of great concern. These include:

1. Extremely heavy-tailed distributions. In such a case, you definitely should not use ordinary least squares to estimate the trend, even if it is linear. Quantile regression is a good alternative.

2. Binary or highly discrete dependent variable. In this case, some maximum likelihood method would be preferred, even if the response function happened to be linear (which is very unlikely in this case).

As far as needing normality and homoscedasticity specifically for prediction (assuming OLS), imagine you are regressing an individual stock return percentage ($Y$) on the S&P 500 return ($X$). You would like to predict what would happen to your return when the market declines by 1% (i.e., when $X=-1$). In this case, you know that your $Y$ is a random variable, with some distribution.

Now, unthinking application of OLS would give you the (essentially) the following prediction: $Y$ will lie within $\pm 3\times rmse $ of the regression prediction, with approximate 99.7% probability.

The problem regarding the homoscedasticity assumption of OLS is that $rmse$, being a pooled estimate across all values of $X$, may over- or under- estimate the conditional standard deviation of $Y$ when $X=1$, depending upon the nature of the heteroscedasticity. This results in a prediction interval that may be too wide, or it may be too narrow.

The problem with the normality assumption of OLS is that, say if the distribution of returns is heavy-tailed (and it probably is), there will occasionally be returns that are much farther than three standard deviations from the conditional mean. So the usual predictions are overly optimistic in terms of bounding risk. Further, if the distribution of returns is asymmetric, then the prediction interval should be correspondingly asymmetric.

Another good example is multinomial logistic regression. Say $Y$ is choice of toothpaste brand, and $X$ is age. What is the prediction of $Y$ when $X=50$ years old? It is certainly not some average of Crest, Colgate, Mr. Toms, etc. The prediction is the distribution itself, which is certainly not a normal distribution. Rather, it is a simple discrete probability distribution on all the brands.

Regression may be best understood as a model for the conditional distribution of $Y$, given $X$. This representation resolves a lot of supposed conflicts between OLS and ML; it leads seamlessly to likelihoods and Bayes; it puts ordinary regression, heteroscedastic regression, ANOVA, Poisson regression, multinomial logistic regression, survival analysis, quantile regression, neural net regression, tree regression, etc., all under a common umbrella; and it includes the mean portion of the model that people are usually interested in as a special case.