Robust regression for heavy-tailed random design

Question

As far as I know, there are robust regression methods for outliers in response $Y$ and heavy-tailed error $\epsilon$. The settings for the design matrix (predictor) $X$ is either fixed design or sub-exponential like Normal distribution. What if the design matrix $X$ is random design and the distribution is heavy-tailed? Is there any advanced methods apart from truncation on $X$ or $\log X$?

score 2 · Answer 1 · answered Feb 20 '21 at 15:45

2

One simple method would be to use the quantiles of $X$ rather than $X$ per se as predictor, e.g. adding a column which equals $1$ if $x_k$ is larger than the 90 quantile of $X$. This avoids truncation or arbitrarily changing the distribution of $X$.

answered Feb 20 '21 at 15:45

Arne Jonas Warnke

3,085
1
22
40

Is there any paper I can read more about this? – Hepdrey Feb 20 '21 at 15:59
Perhaps look here for a larger discussion: https://stats.stackexchange.com/questions/68834/what-is-the-benefit-of-breaking-up-a-continuous-predictor-variable – Arne Jonas Warnke Feb 20 '21 at 16:11

score 1 · Accepted Answer · answered Feb 20 '21 at 16:14

One method/idea is resistant regression. The idea is to order the squared residuals, rejecting some percent of the largest ones, and then minimize the sum of squared kept residuals. That is highly nonlinear and nonconvex, and one idea for the optimization is genetic algorithms.

One implementation is lqs in R package MASS, see the companion book by Venables & Ripley. For code example see How to get summary statistics from "resistant regression" - lqs - in R?

score 1 · Answer 3 · answered Mar 22 '21 at 17:32

1

In Python, in scikit-learn there is also RANSACRegessor, and TheilSenRegressor. Both of them are robust to outliers in the design.

answered Mar 22 '21 at 17:32

TMat

716
1
10

Robust regression for heavy-tailed random design

3 Answers3