1

The documentation for the numpy np.std() function states:

The average squared deviation is typically calculated as x.sum() / N, where N = len(x). If, however, ddof is specified, the divisor N - ddof is used instead. In standard statistical practice, ddof=1 provides an unbiased estimator of the variance of the infinite population. ddof=0 provides a maximum likelihood estimate of the variance for normally distributed variables. The standard deviation computed in this function is the square root of the estimated variance, so even with ddof=1, it will not be an unbiased estimate of the standard deviation per se.

(emphasis added)

What does the final statement in bold mean? Where does the estimate break down and become biased?

Josiah Yoder
  • 155
  • 8
  • In addition to the questions linked in the "close" message, i will add that, for a normal distribution, an unbiased estimator of standard deviation has been found. However, unlike for variance, there is not an unbiased estimator of standard deviation that applies to all distributions. (I forget if one simply has not been found or if it has been shown that no such estimator can exist, though I lean towards the latter.) That is, there is no `ddof` value where you can count on the result to be an unbiased estimate of the standard deviation. – Dave Jun 08 '21 at 21:38
  • @Dave That sounds like a good follow-up question: "Has it been proven that there is no unbiased estimator (no fixed ddof) that applies to all distributions?" Now to avoid making a SECOND duplicate question.... – Josiah Yoder Jun 09 '21 at 17:54
  • @Dave indeed, the question has already been asked: [Unbiased estimator of the std](https://stats.stackexchange.com/questions/404480/unbiased-estimator-of-the-standard-deviation-of-the-sample-standard-deviation). No answer yet. Well, that's on std of sample std, so it might be a little different. – Josiah Yoder Jun 09 '21 at 18:00

1 Answers1

4

While the estimate of the variance is unbiased, the estimate of the standard deviation is not.

An unbiased estimate $V_{est}$ is one such that the expected value of the estimate is the true value $V$ being estimated: $E[V_{est}] = V$.

But applying a nonlinear operator like the square or square root destroys this property. In general, $E[\sqrt{V_{est}}] \neq \sqrt{V}$ so $E[\sigma_{est}] \neq \sigma$. That is, the estimated std deviation $\sigma_{est}$ will be biased even though $V_{est}$ is not.

The distribution of V estimates is not symmetric -- since V must be positive, it will have a longer tail above the true variance and a shorter, fatter tail below the true variance. When we take the square root of V, the new distribution's mode can be found by taking the square root of the location of the mode, but we can't say the same thing about the mean.

Other answers on earlier questions take this discussion further. For example, this answer points out that Jensen's inequality can prove in which direction the estimated standard deviation will be biased and this answer shows a plot of the bias for different sample sizes.

But my personal favorite is this answer which proves using simple expectation algebra that the estimated standard deviation will (in general) be less than the true standard deviation of the data. This answer relies on the identity $\mathrm{Var}[S_n] = \mathrm{E}[S_n^2] - \mathrm{E}^2[S_n]$ which would have to also be explained to the uninitiated, but there is an elegance to it that ties it into basic statistical theory nicely.

Josiah Yoder
  • 155
  • 8