5

There is quite some number of ways how to robustly fit a linear regression model, e.g. using M-estimation based on Tukey's biweight loss or on Huber's loss, see e.g. Wikipedia.

I got two questions about the two mentioned cases:

  1. Is it correct that these are modelling the mean (more precisely the center) of the conditional response distribution as long as the error distribution is symmetric?
  2. What location parameter are they modelling in general? Is it a specially weighted average of the response variable?
Michael M
  • 10,553
  • 5
  • 27
  • 43

1 Answers1

6

There is quite some number of ways how to robustly fit a linear regression model, e.g. using M-estimation based on Tukey's biweight loss or on Huber's loss, see e.g. Wikipedia.

Is it correct that these are modelling the mean of the response as long as the error distribution is symmetric?

Well, estimating, but that may be what you mean by modelling -- but only if the mean exists. But they'll be reasonable estimates of the center of symmetry more generally, which will be the population median, and trimean, and pseudomedian, and any trimmed mean, and, ... and also usually the mean. For example, consider the $t$ distribution with $\nu$ degrees of freedom. If $\nu\leq 1$ there's no mean. But it still has a center of symmetry.

What location parameter are they modelling in general?

M-estimation corresponds to maximum-likelihood estimation if the loss function corresponds to $-\log(f)$ for some density $f$. This is the case for the Huber loss but not the Tukey.

Is it a specially weighted average of the response variable?

Not in general no*. But M-estimators can be obtained** by iterating a weighted average where the weights are updated at each step.

*(at least not unless 'special' is interpreted much more broadly than people generally understand the term 'weighted average' to be able to stretch)

**(in some circumstances at least)

Community
  • 1
Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • 2
    An excellent answer as usual @Glen_b . What's your idea of the best reference for this sort of thing? – Peter Flom Aug 29 '14 at 10:46
  • 2
    I would say that the estimand is whatever the estimator estimates, so the question is backward in one sense. On a simple analogue: you are at liberty to regard the sample median as a robust estimator of the mean, and that may be your motivation for using it, but it is as simple or simpler to regard the sample median as an estimator of the population median. More discussion, including other points of view, at http://stats.stackexchange.com/questions/63386/is-any-quantitative-property-of-the-population-a-parameter – Nick Cox Aug 29 '14 at 11:58
  • 2
    @Peter Best kind of depends on a lot of things - what sort of treatment one seeks, for example. I also prefer to read many sources rather than one - including papers, talks, notes as well as books. But the classic references of Huber (1981) and Hampel et al (1986) are still pretty relevant (see the wikipedia page on Robust statistics for complete references and a number of other good ones). – Glen_b Aug 29 '14 at 14:35
  • 2
    @Peter The book by Staudte and Sheather is a bit more recent and may be suitable for some purposes. [This document](http://www.stats.ox.ac.uk/~ripley/StatMethods/Robust.pdf) by Brian Ripley isn't too bad if you want a short introduction. – Glen_b Aug 29 '14 at 15:02
  • @NickCox: Quite similar thoughts led me to ask the question. Isn't it a bit strange then to model a quantity that is hardly ever used in univariate statistics? – Michael M Aug 29 '14 at 16:15
  • 2
    Indeed, but what is strange is a matter of sociology also. In principle we **could** use more advanced summaries for univariate work; in practice the more advanced methods are applied more to model fitting for responses given predictors, i.e. robust regression is what everyone in the field most wants to devise (or use). Trimming methods are apparently widely used in some sports and competitive activities in which top and bottom judges' ratings are discounted to discourage or downweight partisan biases. So, some subtle methods are widely used outside "science" as usually defined. – Nick Cox Aug 29 '14 at 16:30
  • 3
    I prefer to estimate quantities that are at least somewhat easy to understand: means, mean absolute differences (for variability), quantiles, exceedance probabilities. Cumulative probability ordinal models are robust and can estimate a wide variety of quantities successfully. They do not allow "outliers" on $Y$ to affect regression coefficients, but outliers will have an impact when using these models to estimate the mean. – Frank Harrell Aug 29 '14 at 22:10
  • @Glen_b: The document Robust.pdf you referenced (by Brian Ripley) seems to have disappeared from the web, unfortunately ... – kjetil b halvorsen Mar 19 '17 at 16:30
  • Thanks @kjetil -- I have replaced the link with the place it moved to. – Glen_b Mar 20 '17 at 04:59