What do you call an average that does not include outliers?

Question

What do you call an average that does not include outliers?

For example if you have a set:

{90,89,92,91,5} avg = 73.4

but excluding the outlier (5) we have

{90,89,92,91(,5)} avg = 90.5

How do you describe this average in statistics?

Answer 1

It's called the trimmed mean. Basically what you do is compute the mean of the middle 80% of your data, ignoring the top and bottom 10%. Of course, these numbers can vary, but that's the general idea.

Answer 2

A statistically sensible approach is to use a standard deviation cut-off.

For example, remove any results +/-3 standard deviations.

Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable.

Answer 3

Another standard test for identifying outliers is to use LQ $-$ (1.5$\times$IQR) and UQ $+$ (1.5$\times$ IQR). This is somewhat easier than computing the standard deviation and more general since it doesn't make any assumptions about the underlying data being from a normal distribution.

Answer 4

The "average" you're talking about is actually called the "mean".

It's not exactly answering your question, but a different statistic which is not affected by outliers is the median, that is, the middle number.

{90,89,92,91,5} mean: 73.4
{90,89,92,91,5} median: 90

This might be useful to you, I dunno.

Answer 5

For a very specific name, you'll need to specify the mechanism for outlier rejection. One general term is "robust".

dsimcha mentions one approach: trimming. Another is clipping: all values outside a known-good range are discarded.

Answer 6

There is no official name because of the various mechanisms, such as Q test, used to get rid of outliers.

Removing outliers is called trimming.

No program I have ever used has average() with an integrated trim()

Answer 7

I don't know if it has a name, but you could easily come up with a number of algorithms to reject outliers:

1. Find all numbers between the 10th and 90th percentiles (do this by sorting then rejecting the first $N/10$ and last $N/10$ numbers) and take the mean value of the remaining values.

2. Sort values, reject high and low values as long as by doing so, the mean/standard deviation change more than $X\%$.

3. Sort values, reject high and low values as long as by doing so, the values in question are more than $K$ standard deviations from the mean.

Answer 8

... {90,89,92,91(,5)} avg = 90.5

How do you describe this average in statistics? ...

There's no special designation for that method. Call it any name you want, provided that you always tell the audience how you arrived at your result, and you have the outliers in hand to show them if they request (and believe me: they will request).

Answer 9

The most common way of having a Robust (the usual word meaning resistant to bad data) average is to use the median. This is just the middle value in the sorted list (of half way between the middle two values), so for your example it would be 90.5 = half way between 90 and 91.

If you want to get really into robust statistics (such as robust estimates of standard deviation etc) I would recommend a lost of the code at The AGORAS group but this may be too advanced for your purposes.

Answer 10

If all you have is one variable (as you imply) I think some of the respondents above are being over critical of your approach. Certainly other methods that look at things like leverage are more statistically sound; however that implies you are doing modeling of some sort. If you just have for example scores on a test or age of senior citizens (plausible cases of your example) I think it is practical and reasonable to be suspicious of the outlier you bring up. You could look at the overall mean and the trimmed mean and see how much it changes, but that will be a function of your sample size and the deviation from the mean for your outliers.

With egregious outliers like that, you would certainly want to look into te data generating process to figure out why that's the case. Is it a data entry or administrative fluke? If so and it is likely unrelated to actual true value (that is unobserved) it seems to me perfectly fine to trim. If it is a true value as far as you can tell you may not be able to remove unless you are explicit in your analysis about it.

Answer 11

I love the discussion here - the trimmed mean is a powerful tool to get a central tendency estimate concentrated around the middle of the data.

The one thing I would add is that there is a choice to be made about which "metric" to use in the cases of small and large sample sizes. In some cases we talk about

• means in the context of large samples because of central-limit theorem,
• medians as robust small-sample alternatives
• and trimmed means as robust to outliers.

Obviously the above is a gross generalization, but there are interesting papers that talk about the families and classes of estimators in large and small sample settings and their properties. I work in bioinformatics aand usually you deal with small samples (3-10s) usually in mice models, and what not, and this paper gives a good technical overview of what alternatives exist and what properties these estimators have.

Robust estimation in very small samples

• Reference: Rousseeuw, P. J., & Verboven, S. (2002). Robust estimation in very small samples. Computational Statistics & Data Analysis, 40(4), 741-758.
• Link: https://www.sciencedirect.com/science/article/pii/S0167947302000786

This is off-course one paper, but there are plenty others that discuss these types of estimators. Hope this helps.

Answer 12

disclaimer - this method is ad hoc and without rigorous study. Use at your own risk :)

What I found to be quite good was to reduce the relevancy of a points contribution to the mean by the square of its number of standard deviations from the mean but only if the point is more than one standard deviation from the mean.

Steps:

1. Calculate the mean and standard deviation as usual.
2. Recalculate the mean, but this time, for each value, if it is more than one standard deviation from the mean reduce its contribution to the mean. To do reduce its contribution, divide its value by the square of its number of deviations before adding to the total. Also because it's contributing less, we need to Reduce N, so subtract 1-1/(square of values deviation) from N.
3. Recalculate the standard deviation, but use this new mean rather than the old mean.

example: stddev = 0.5 mean = 10 value = 11

then, deviations = distance from mean / stddev = |10-11|/0.5 = 2

so value changes from 11 to 11/(2)^2 = 11/4

also N changes, it is reduced to N-3/4.

code:

def mean(data):
    """Return the sample arithmetic mean of data."""
    n = len(data)
    if n < 1:
        raise ValueError('mean requires at least one data point')
    return 1.0*sum(data)/n # in Python 2 use sum(data)/float(n)

def _ss(data):
    """Return sum of square deviations of sequence data."""
    c = mean(data)
    ss = sum((x-c)**2 for x in data)
    return ss, c

def stddev(data, ddof=0):
    """Calculates the population standard deviation
    by default; specify ddof=1 to compute the sample
    standard deviation."""
    n = len(data)
    if n < 2:
        raise ValueError('variance requires at least two data points')
    ss, c = _ss(data)
    pvar = ss/(n-ddof)
    return pvar**0.5, c

def rob_adjusted_mean(values, s, m):
    n = 0.0
    tot = 0.0
    for v in values:
        diff = abs(v - m)
        deviations = diff / s
        if deviations > 1:
            #it's an outlier, so reduce its relevancy / weighting by square of its number of deviations
            n += 1.0/deviations**2
            tot += v/deviations**2
        else:
            n += 1
            tot += v
    return tot/n

def rob_adjusted_ss(values, s, m):
    """Return sum of square deviations of sequence data."""
    c = rob_adjusted_mean(values, s, m)
    ss = sum((x-c)**2 for x in values)
    return ss, c

def rob_adjusted_stddev(data, s, m, ddof=0):
    """Calculates the population standard deviation
    by default; specify ddof=1 to compute the sample
    standard deviation."""
    n = len(data)
    if n < 2:
        raise ValueError('variance requires at least two data points')
    ss, c = rob_adjusted_ss(data, s, m)
    pvar = ss/(n-ddof)
    return pvar**0.5, c

s, m = stddev(values,ddof=1)
print s, m
s, m = rob_adjusted_stddev(values, s, m, ddof=1)
print s, m

output before and after adjustment of my 50 measurements:

0.0409789841609 139.04222
0.0425867309757 139.030745443

Answer 13

There are superior methods to the IQR or SD based methods. Due to outliers being present, the distribution likely has issues with normality already (unless ouliers are evenly distributed at both ends of the distribution). This inflates the SD a lot, making the SDs use less than desirable, however the SD method has some desirable aspects over the IQR method, namely 1.5 times the IQR is a relatively subjective cutoff. While subjectivity in these matters is unavoidable it is preferable to reduce it.

A Hampel Identifier on the other hand uses robust methods to estimate outliers. Essentially its the same as the SD method, but you would replace means with medians and SD with Median Absolute Deviations (MAD). MADs are just the median distance from the media. This MAD is multiplied by a scaling constant .675. The formula comes out to (X - Median)/(.675*MAD). The resulting statistic is treated identically to a Z-score. This bypasses the issue of the likely non-normality that if you have outliers may be present.

As for what to call it. Trimmed means are normally reserved for the method of trimming the bottom and top ten percent mentioned by @dsimcha. If it has been completely cleaned you may refer to it as the cleaned mean, or just the mean. Just be sure to be clear what you did to it in your write-up.

Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., & Stahel, W. A. (1986). Robust Statistics. John Wiley & Sons, New York.

Answer 14

My statistics textbook refers to this as a Sample Mean as opposed to a Population Mean. Sample implies there was a restriction applied to the full dataset, though no modification (removal) to the dataset was made.

Answer 15

It can be the median. Not always, but sometimes. I have no idea what it is called in other occasions. Hope this helped. (At least a little.)