0

There are two kinds of estimates of variance from an iid sample $X1, \dots, X_n$

  • $1/n * \sum_i (X_i - \bar{X})^2$, which is MLE

  • $1/(n-1) * \sum_i (X_i - \bar{X})^2$, which is unbiased.

The unbiased estimate has a bigger variance than the MLE, for a given $n$.

So I wonder when to prefer which of the two? Thanks!

Ferdi
  • 4,882
  • 7
  • 42
  • 62
Tim
  • 1
  • 29
  • 102
  • 189
  • 1
    Note that the first one is the MLE if the variables have a normal distribution, not necessarily if they have a different distribution. Also, if they *do* have a normal distribution, dividing by $n + 1$ (sic!) will give the lowest mean square error. So you actually have three good options. :) But go with the unbiased one; simply because it’s the one most often used. – Karl Ove Hufthammer Feb 01 '14 at 23:37
  • THanks, but why "if they do have a normal distribution, dividing by n+1 (sic!) will give the lowest mean square error"? – Tim Feb 01 '14 at 23:44
  • 2
    "When should I prefer this or that estimator" is like asking "In this recipe, when should I prefer a higher proportion of choc chips and when should I prefer more frosting?". Ultimately, it depends on your loss function / equivalently the utility of the various properties you're choosing between. How much do you like unbiasedness? How much mean square error? How much do you like the nice invariance properties that come with MLEs? etc etc – Glen_b Feb 01 '14 at 23:58
  • Tim, if you want to ask a followup question about why the MMSE at the normal has an n+1 divisor, that's a whole new question. Please don't go back to the habit of asking a series of questions in comments, when each is really a question that stands on its own. – Glen_b Feb 02 '14 at 00:02

1 Answers1

1

I'm not sure there is ever a reason to use the MLE estimate over the unbiased estimate, even though there will be little difference between the two for large sample sizes. For small sample sizes, you should definitely use the unbiased estimator.

Just so you know, the formulas you have given are specific to a normal distribution. They aren't universal. There may be more of a reason to prefer the MLE variance estimate for different situations.

Previous question with more details: Unbiased estimator for variance or Maximum Likelihood Estimator?

Community
  • 1
ahwillia
  • 2,406
  • 1
  • 14
  • 26
  • 2
    Alex, when you say "For small sample sizes, you should definitely use the unbiased estimator." -- if I care about the invariance properties of MLEs (what's ML for the variance corresponds to ML for the precision and the standard deviation - that is, $\widehat{g(\theta)}=g(\hat \theta)$, which doesn't hold for unbiasedness nor MMSE), or I prefer smaller MSEs to larger, why should I still 'definitely use the unbiased estimator'? If I am calculating variance in order to obtain the standard deviation, say, or the precision, the $n-1$ divisor is not unbiased for those. – Glen_b Feb 02 '14 at 00:05
  • 1
    @Glen_b -- I guess I'm just used to situations where unbiasedness is less important than those considerations. I'm willing to admit that I'm wrong, however. Perhaps a concrete example would help me understand. Also, why not divide by $n-0.5$ as a compromise? As Karl Ove Hufthammer mentioned, the unbiased estimate is most commonly used. It seems to me that you would have to have a pretty good reason to not use it. Calculating an unbiased estimate of the standard deviation seems like a different problem. – ahwillia Feb 02 '14 at 06:17
  • 1
    I presume you mean "where unbiasedness is more important" (not less); usually what makes unbiasedness important would tend to be your priorities that make it important. Note that if I compromise between minimum MSE and unbiased, the compromise value would be $n$ - so it depends on which things you compromise between. Or if I have a normal distribution, and I want an approximately unbiased *standard deviation*, I might choose $(n-1.5)$. I don't think the unbiased estimation of standard deviation is really a different problem: much of the time $\sum (x_i-\bar x)^2/(n-1)$ is done, $√$ comes next. – Glen_b Feb 02 '14 at 07:27
  • How often is an unbiased variance actually *necessary*, rather than just a function of preferences? – Glen_b Feb 02 '14 at 07:29
  • I agree that it is a function of preferences. Point taken. However, unless I was communicating to an educated audience that understands these subtleties (e.g. to other statisticians), I would use the unbiased estimate. I spend most of my time interacting with biologists and other scientists, who would probably be misled if I used anything other than $n-1$. Thus, while it is a matter of preference, it is also a matter of convention. I think it is important not to underestimate the importance of this. It is interesting that we don't use the unbiased estimate for standard deviation by convention. – ahwillia Feb 02 '14 at 14:50