Dealing with outliers in simple linear regression

Question

I want to perform a simple linear regression in R. However, the plot of fitted and residual values has outliers. Transformations (ie log, square root) did not solve this problem. Removing these outliers created new outliers. So, what can I do? Is it wrong to adjust a Poisson distribution for this case of a simple linear regression and perform a generalized model? Are there other options?

Answer 1

Probably the key question is: what kind of outliers? Outliers in the dependent variable, Y? Or in the predictor(s), X? I'm going to assume that the outliers are in the DV. One inheritance of 20th c assumptions rooted in Gaussian statistics about regression is that the errors be normally distributed and a plethora of techniques have been developed to constrain those extreme values. The problem with this is that outliers, unless they are "bad," errorneous or fraudulent, contain potentially valuable information. This includes the possibility of classes of behaviors and distributions not widely studied (yet) such as power law or heavy-tail distributions and extreme value models. The idea behind power laws can be motivated when thinking of so-called "long-tail" behaviors like Internet sales on online sites like Amazon. Heavy-tails are related to this -- consider stock returns on days with huge losses such as started the Downturn in 2008. Extreme value models (aka generalized extreme value models or GEVs) are a class of models that could use greater exposure and adoption by statisticians as much real-life behavior is extreme valued. They are widely used in insurance and financial risk management but have ready applications, e.g., in creative industries such as film and music where "blockbuster" hits rule the box offices and topline revenues in these businesses. For instance, William Golding, a Hollywood scriptwriter, has been widely quoted as saying that, in Hollywood, "Nobody knows nothing." Golding is referring to the fact that the huge hits that generate huge returns are inherently unpredictable.

So, what does all of this have to do with your problem and question? It's a very long-winded answer to say that my recommendation is that, unless they can be proven to be somehow wrong, that you treat your outliers as real data and valuable information. One reliable workaround to textbook regression is that you find a robust, nonparametric R regression module known as quantile regression. Quantiles can be any percentile as specified by the analyst. The key thing about QR is that, unlike traditional, OLS regression which predicts mean or average behaviors, QR predicts a quantile, which is usually the median value by default, but can be any quantile of the analyst's choosing. QR is much more robust to outliers than traditional, OLS regression. One approach to answering which quantile is to use a grid-like search across a range of values above and below the median.

Let me know if you have any questions.