11

I have the following linear model:

Linear model residuals Observations distribution

To address the residuals heteroscedasticity I have tried to apply a log transformation on the dependent variable as $\log(Y + 1)$ but I still see the same fan out effect on the residuals. The DV values are relatively small so the +1 constant addition before taking the log is probably not appropriate in this case.

> summary(Y)
Min.   :-0.0005647  
1st Qu.: 0.0001066  
Median : 0.0003060  
Mean   : 0.0004617  
3rd Qu.: 0.0006333  
Max.   : 0.0105730  
NA's   :30.0000000

How can I transform the variables to improve the prediction error and variance, particularly for the far right fitted values?

Macro
  • 40,561
  • 8
  • 143
  • 148
Robert Kubrick
  • 4,078
  • 8
  • 38
  • 55

3 Answers3

12

What is your goal? We know that heteroskedasticity does not bias our coefficient estimates; it only makes our standard errors incorrect. Hence, if you only care about the fit of the model, then heteroskedasticity doesn't matter.

You can get a more efficient model (i.e., one with smaller standard errors) if you use weighted least squares. In this case, you need to estimate the variance for each observation and weight each observation by the inverse of that observation-specific variance (in the case of the weights argument to lm). This estimation procedure changes your estimates.

Alternatively, to correct the standard errors for heteroskedasticity without changing your estimates, you can use robust standard errors. For an R application, see the package sandwich.

Using the log transformation can be a good approach to correct for heteroskedasticity, but only if all your values are positive and the new model provides a reasonable interpretation relative to the question that you are asking.

Charlie
  • 13,124
  • 5
  • 38
  • 68
  • My primary goal is to reduce the errors. I will have to look into weighted least squares but I was under the impression that a DV transformation was the right step, given how regularly the residual variance increases for higher fitted values. – Robert Kubrick Feb 29 '12 at 18:41
  • What do you mean "reduce the errors"? The average error is 0. Even looking in your plot, in any window that you choose, the average is 0. – Charlie Feb 29 '12 at 19:28
  • I mean improving the prediction of the model, that is reduce the overall absolute error and error variance, particularly for the higher fitted values. – Robert Kubrick Feb 29 '12 at 19:58
  • 1
    Suppose that you can transform $y$ in such a way that reduces heteroskedasticity. If you want to predict $y$, you'll have to apply the inverse of that transformation, bringing the heteroskedasticity problem back. Transformations are fine if all you care about are coefficients, but aren't going to help you if you're trying to predict $y$. – Charlie Feb 29 '12 at 20:03
  • I am not clear on your last comment, I am only concerned with the prediction precision. How is heteroskedasticity a problem once I have inverse-transformed $y$, in terms of prediction? – Robert Kubrick Feb 29 '12 at 20:15
  • 1
    I'm guessing that you don't want to predict transformed $y$, right? Effectively, your transformation would have to shrink the distance between $y$'s on your original scale. You create prediction intervals on the transformed scale that have a similar width across transformed $y$ values, but when you undo the transformation, the prediction intervals get stretched onto the original $y$ scale. – Charlie Feb 29 '12 at 20:20
  • Good point about the prediction intervals! However $y$ predictions should match the original scale, when inverse-transformed. – Robert Kubrick Feb 29 '12 at 20:24
  • Sure, but if all you care about is the $y$, then you don't need to worry about heteroskedasticity at all; OLS gives unbiased coefficient estimates, giving unbiased estimates for (the expected value of) $y$. So you have to ask yourself just what question you want to answer or what statistic you want at the end. – Charlie Feb 29 '12 at 20:27
  • True, prediction interval is a main concern in my case. However I am not sure heteroskedasticity is not an issue at all, even if one was only interested in predicted $y$. By reducing the errors variance we are improving $R^2$, therefore the absolute error should be smaller for $y$ on the original scale. – Robert Kubrick Feb 29 '12 at 20:47
  • OLS gives unbiased coefficient estimates and thus unbiased predictions of the expected value of $y$ given $x$ if $\text{E}[y \mid x] = x\beta$. No heteroskedasticity assumption needed. The OLS procedure maximizes $R^2$; no linear model can achieve a higher $R^2$. If you are considering non-linear models, then $R^2$ isn't an appropriate metric to compare the models. – Charlie Feb 29 '12 at 21:18
  • What is the observation-specific variance? I can calculate a z-score for each $y$ given the population sd but I'm not clear what the observation variance is. – Robert Kubrick Mar 02 '12 at 20:31
  • You need to estimate the observation-specific variance. You can check econometrics textbooks that discuss weighted least squares or open a new question on that topic. – Charlie Mar 07 '12 at 00:22
4

You would want to try Box-Cox transformation. It is a version of a power transformation:

$$ y \mapsto \left\{ \begin{eqnarray} \frac{y^\lambda-1}{\lambda (\dot y)^{\lambda-1}}, & \lambda \neq 0 \\ \dot y \ln y, & \lambda = 0 \end{eqnarray} \right. $$ where $\dot y$ is the geometric mean of the data. When used as a transformation of the response variable, its nominal role is to make the data closer to the normal distribution, and skewness is the leading reason why the data may look non-normal. My gut feeling with your scatterplot is that it needs to be applied to (some of) the explanatory and the response variables.

Some earlier discussions include What other normalizing transformations are commonly used beyond the common ones like square root, log, etc.? and How should I transform non-negative data including zeros?. You can find R code following How to search for a statistical procedure in R?

Econometricians stopped bothering about heteroskedasticity after seminal work of Halbert White (1980) on setting up inferential procedures robust to heteroskedasticity (which in fact just retold the earlier story by a statistician F. Eicker (1967)). See Wikipedia page that I just rewrote.

Community
  • 1
StasK
  • 29,235
  • 2
  • 80
  • 165
  • Thanks, at this point I'm debating whether to apply a power transform or use robust regression to reduce the errors and improve the prediction intervals. I wonder how the two techniques compare. Also if I use the transformation I would need to back-transform the predicted values. It doesn't look like an obvious formula, does it? – Robert Kubrick Feb 29 '12 at 19:03
  • If by robust regression, you mean robust standard errors as @StasK describes, that doesn't change the residuals/errors at all. The coefficients are exactly the same as OLS, giving exactly the same residuals. The standard errors of the coefficients change and are usually larger than the OLS SEs. Prediction intervals are improved in that you now are using the correct standard errors for your coefficients (though they are likely larger relative to those from OLS). If your goal is to predict $y$, you really should stick with the linear model and use the techniques that I mention in my answer. – Charlie Feb 29 '12 at 19:31
  • @Charlie I mean http://en.wikipedia.org/wiki/Robust_regression. I am new to this, but I understand robust regression changes the estimation technique, therefore the residuals must be different. – Robert Kubrick Feb 29 '12 at 20:04
  • Right, that is a different method and does change your estimates. I think that robust regression is better suited to cases with outliers. Depending upon which version of robust regression you decide to use and your particular data set, you can get wider confidence intervals relative to OLS. – Charlie Feb 29 '12 at 20:25
1

There is a very simple solution to heteroskedasticity issue associated with dependent variables within time series data. I don't know if this is applicable to your dependent variable. Assuming it is, instead of using nominal Y change it to % change in Y from the current period over the prior period. For instance, let's say your nominal Y is GDP of $14 trillion in the most current period. Instead, compute the change in GDP over the most recent period (let's say 2.5%).

A nominal time series always grows and is always heteroskedastic (the variance of the error grows over time because the values grow). A % change series is typically homoskedastic because the dependent variable is pretty much stationary.

Sympa
  • 6,862
  • 3
  • 30
  • 56
  • The $Y$ values I am using are time series % changes from the previous period. – Robert Kubrick Feb 29 '12 at 21:03
  • This is surprising. Usually, % change variables are not heteroskedastic. I am wondering if the residuals are less heteroskedastic than we think. And, that the underlying issue is one of outliers. I see 4 or 5 observations in the 0.15% range that if removed would make the whole graph less heteroskedastic looking. Also, as others have mentioned heteroskedasticity will not corrupt your regression coefficients, but only your confidence intervals and related standard error. However, looking at your graph it seems that CIs may not be too affected. And, could still be useful. – Sympa Feb 29 '12 at 22:05