2

NB: In this post, I will use the abbreviation $TM_P$ to stand for a symmetric truncated (or trimmed) mean that discards the largest and smallest $P/2$ percent of the data. In fact, for concreteness I will refer mostly to $TM_{50}$, but the same question could be asked for any other "similar value" of $P$.


Question

What are situations for which a sound statistical basis exists to prefer $TM_{50}$ (or $TM_P$ for some other more appropriate $P$) over the median as measure of the data's central tendency?


EDIT: (In response to Nick Cox's ridiculing of my wording.) Here's an example of the sort of justification (speaking very broadly) I had hoped for. Q: Why choose the median over the mean? A: For robustness against outliers. The mean can be made anything one wants by a single sufficiently extreme outlier, whereas the median is naturally immune to such an instability; no outlier censoring required. Granted, very extreme outliers are probably entirely tangential to the process under study, but this reasoning still vividly underscores the robustness of the median. There's nothing subjective about it, it's a mathematical fact, even if different people find the same mathematical fact more or less problematic.


Background

I cannot think of a justification for using $TM_{50}$ that does not apply equally well to the median 1. Furthermore, I figure that, as summary statistics go, the median is the conceptually and analytically simpler of the two (and, therefore, the more thoroughly studied and better characterized one). Thus, my gut reaction is to always prefer the median over $TM_{50}$. The motivation for this thread is to either confirm or refute this gut reaction.


1 Of course, the median can be thought of as, roughly, the limit of $TM_P$ as $P \to 100$. Therefore, I figure that any justification for using $TM_P$ over the median can only get weaker as $P \to 100$. An analogous consideration applies when $P \to 0$, if we also replace the median with the mean.

kjo
  • 1,817
  • 1
  • 16
  • 24
  • 2
    Reduced variance maybe?! – Xi'an Dec 01 '15 at 18:26
  • What would a "sound ...basis" look like in this case, especially if it has to overcome your "gut reaction"? (Insert emoticon to taste.) You can get results, if only by simulation, for specified distributions, but the main merit of trimmed means in my view is in the face of genuine uncertainty or insecurity about the generating process. Besides, no one so far as I know is obliged to choose a particular $P$ (in your notation): it makes sense to experiment: see e.g. http://stats.stackexchange.com/questions/117950/how-can-i-interpret-a-plot-of-trimming-percentage-vs-trimmed-mean for flavour. – Nick Cox Dec 01 '15 at 18:37
  • $TM_{50}$ has a specific name (midmean) and a concrete interpretation as the mean of the values inside the box in a box plot. Neither fact is a justification in principle, naturally, but that may help make them more recognisable to many data analysts. – Nick Cox Dec 01 '15 at 18:39
  • @NickCox: I am suspicious of experimentation when it is not guided by principle. It too easily degenerates into data-snooping and cherry-picking the most publishable angle. I operate in a world where these are constant threats, so I'm perhaps somewhat paranoid about them. "...the main merit of trimmed means in my view is in the face of genuine uncertainty of insecurity about the generation process": as I said, in this case the median is just as good. – kjo Dec 02 '15 at 03:14
  • @NickCox: "I operate in a world..." That came out overly dramatic, didn't it. Let me put it differently: I am constantly under a lot of pressure to sign on to statistical procedures that I consider inadequately justified, and that I suspect are chosen ***only*** because they were found, by *trial-and-error* (IOW, experimentation), to present some results in their most favorable light. I am trying to resist this pressure with something more objectively grounded and workable than a suspicion of malpractice. – kjo Dec 02 '15 at 04:04
  • There is a clear and simple principle behind trying different fractions of trimming. As in your question, you are unclear about how much to trim and on what grounds, so rather than picking 5% or 25% you can try a range. That is just like any other kind of sensitivity analysis where it's good news if conclusions are stable over a range of possibilities and a puzzle otherwise. I don't see that this hinges on personal statistical philosophy or anyone's workplace pressures; but if it does, then clearly different researchers may make different choices in context. – Nick Cox Dec 02 '15 at 09:34
  • On your EDIT: There is no English word "riduling" to my knowledge. If you mean "ridiculing", I had no intent to ridicule you. I was quoting _exactly what you said_ and indicating that I was doing so with a grin. Your words not mine. – Nick Cox Dec 02 '15 at 09:52
  • It's a fallacy that the median is always insensitive to extremes. Consider 49 zeros and 51 ones, or 51 zeros and 49 ones. The median will jump from 1 to 0 in such a case while the mean is the summary that varies smoothly from 0.49 to 0.51. This case is not often important, and people would not usually think of a median in this case. But often there are categorical, unqualified statements that the median is robust or resistant. Usually, but not always. – Nick Cox Dec 02 '15 at 09:56

1 Answers1

2

If you are comparing, say, the sample median, the sample interquartile mean (trimmed mean with 25% of data removed from highest and lowest values) and the sample mean, you have to say what you are trying to do. Otherwise asking which one is better makes no sense at all. If your goal is to estimate the 'center' of a population, you will have to face the problem that these are estimators of DIFFERENT POPULATION PARAMETERS. In this sense, they are not really comparable. For instance, with life expectancies, do you want to estimate the life expectancy that is attained by half of the people? Than you want a median. If you want the average population life expectancy, you'll estimate that with a sample mean. If you want the average life expectancy of the middle 50% of the population, you'll want to estimate it with the interquartile mean. These are not the same value if the population isn't symmetric.

With symmetric populations, all of these are, in some sense, estimators of the same parameter (I say, in some sense, because the sample mean isn't a consistent estimator of population center for some really thick tailed distributions). You'd want to pick the one that minimizes some reasonable loss function (variance?) for the distributions you are likely to work with. Under this criterion, we CAN compare the three.

In general, the mean uses more of the information in the data than does the trimmed mean, which uses more than the median. On the other hand, the median is extremely robust to errors in the data, the trimmed mean somewhat less so and the mean very susceptible to being ruined by outliers.

If your goal is to look for a shift in location for your distribution (for instance, does the life expectancy increase by a fixed amount in one treatment group), you can base comparisons between treatment groups on any measure of center, even if the distribution isn't symmetric. Then you'd select the one that has the lowest variance.

It's pretty easy to compare these via simulations (here I'm using the median, mean and the trimmed mean with 40% trimmed from both extremes). Note that these are all symmetric distributions:

> sims = matrix(ncol=3,nrow=100000,NA)
> colnames(sims) = c("mean","trimmed mean","median")
>  for(i in 1:100000){
+   x = rcauchy(20)
+   sims[i,1] = mean(x)
+   sims[i,2] = mean(x,trim=0.4)
+   sims[i,3] = median(x)
+ }
> #  Variances
> diag(var(sims))
        mean trimmed mean       median 
1.865621e+06 1.360353e-01 1.397239e-01 
> 


>  for(i in 1:100000){
+   x = rnorm(20)
+   sims[i,1] = mean(x)
+   sims[i,2] = mean(x,trim=0.4)
+   sims[i,3] = median(x)
+ }
> #  Variances
> diag(var(sims))
        mean trimmed mean       median 
  0.05007671   0.06883245   0.07347544 
> 


>  for(i in 1:100000){
+   x = rcauchy(20)^3
+   sims[i,1] = mean(x)
+   sims[i,2] = mean(x,trim=0.4)
+   sims[i,3] = median(x)
+ }
> #  Variances
> diag(var(sims))
        mean trimmed mean       median 
2.023236e+27 2.045000e-01 1.308861e-01 
> 

For this simulation:

For the normal distribution, the mean has a higher variance than the others. For the Cauchy, the trimmed mean is best (variance of mean is actually infinite!), while for the cubed Cauchy, the median beats both of them.

AlaskaRon
  • 2,219
  • 8
  • 12