1

Curious what the functional differences are between using a Huber loss function/ regression and Winsorizing data and then running a classic least squares regression.

Will the resulting outputs be roughly the same? Assuming no gross outliers? Are there situations in which one is preferred over the other?

Michael
  • 2,180
  • 4
  • 23
  • 32
  • 1
    It's unclear how you are thinking of using Winsorizing in a least squares setting. Would you Winsorize the regressors, the response, both? Or would you Winsorize the residuals (effectively capping the quadratic loss) as part of the fitting algorithm? – whuber Jul 21 '17 at 15:40
  • i was thinking mainly of winsorizing the regressors – Michael Jul 21 '17 at 17:35
  • 1
    That would bias the results, perhaps strongly. What would be the point of it? Is it worth that price to mitigate the influence of some outlying regressor values? – whuber Jul 21 '17 at 17:40
  • yes exactly - i thought it was a pretty standard process in statistical modeling – Michael Jul 21 '17 at 17:56
  • 1
    Winsorizing is used in Exploratory Data Analysis to produce robust estimates of mean and variance of univariate datasets. I am unaware of (theoretically justified) applications of it in "statistical modeling" in general. – whuber Jul 21 '17 at 18:37
  • I don't understand what you mean by "winsorizing the regressor", if your model is $Y=\beta^T X+\beta_0$, what is the transformation that you do ? is it a transformation on $\beta$ after you determined it thanks to an optimization process ? i.e. $\beta_j = (-c \vee \beta_j) \wedge c$ in which case it is not the same at all as Huber regression, or is it more complicated than that ? – TMat Jun 26 '20 at 08:58

1 Answers1

1

I don't have enough reputation to comment, but I would like to respond to the question above (in the comments to the original post) by @whuber.

I think this is a good question. Huber (1964) notes, for the intercept-only model, that his proposed "most-robust" M-estimator is very close to an estimate of the Winsorized mean of the dependent variable. Therefore it is natural to wonder about the extension to the case with additional explanatory variables, and about practical differences that may arise if one applies OLS to a Winsorized dependent variable instead of applying an M-estimator to the original (untransformed) dependent variable.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
JS1204
  • 11
  • 1
  • 1
    Winsorizing that I have seen works on the extreme $p$% of observations, e.g. replacing the highest or lowest values by some value closer to the middle of the distribution. An M-estimate usually works in terms of the distance between a value and a reference level. For some simple cases, e.g. symmetrical unimodal distributions, a translation will be fairly simple, but not so far as I can see very helpful otherwise. – Nick Cox May 21 '19 at 23:47