3

Conceptually we learn that the sample variance formula, we have to divide by n-1 such that it gives the correct estimation for the true variance.

However if we have this example,

let a sample $X_i = 1$ with probability $p$, and $0$ with probability $1-p$.

so if we define a statistic $\hat{p} = \frac{1}{N} \sum X_i$,

we find that $E(\hat{p}) = \frac{1}{N} N p = p$; and

working from first principle we can find that variance of $\hat{p}$

$= \textrm{var}(\frac{1}{N} * \sum X_i)$

$= \frac{1}{N^2} Np(1-p)$

$= \frac{p(1-p)}{N}$

So my question is that is this variance obtained the same as the sample variance formula (since there is no divide by n-1 term)? If not which should be the correct answer?

Thank you.

Edit: Is it because the N-1 formula is talking about the sample variance, and my example we are actually looking at the variance of the sample mean (i.e. sample mean variance)?

ocram
  • 19,898
  • 5
  • 76
  • 77
John Tan
  • 41
  • 3
  • 1
    The point is that while $E[(\hat{p}-p)^2]=\frac{p(1-p)}{N}$ you also have to consider the relationship between $\frac{p(1-p)}{N}$ and $E[\frac{\hat{p}(1-\hat{p})}{N}]$ – Henry Mar 25 '12 at 11:11

3 Answers3

2

In your example, you are indeed confusing the variance of the mean with the variance of the population. To understand where the $N-1$ factor is coming from, you would have to follow the geometry of a multivariate normal distribution and Cochran's theorem that describes the decomposition of the sums of squares.

StasK
  • 29,235
  • 2
  • 80
  • 165
  • (+1) this is the only one of the 3 answers (so far) that seems to be attacking the real problem. This isn't about $n$ vs. $n-1$, it's about sample variance (a function of the data set) vs. the value of $E[(X-\mu)^2]$ - the variance of a theoretical distribution - why should one expect there to be agreement? – Macro May 02 '12 at 23:47
0

The n vs. n-1 denominator in the sample variance is a common source of confusion for students. To see where the difference comes from, try calculating the method of moments estimator of the variance, and then the maximum likelihood estimator of the variance.

One explanation for the use of n-1 is that we start of with n data values, and therefore n degrees of freedom, but before we get the sample variance, we need to estimate the mean, (we plug x-bar into the sample variance formula) so we have used 1 degree of freedom, and thus, are left with n-1.

Gschneider
  • 504
  • 2
  • 17
  • $\hat p(1−\hat p)/N$ IS the maximum likelihood estimator of variance of a Bernoulli observation. You must have meant normal likelihood. The argument about degrees of freedom is handwaving: why should anybody care about degrees of freedom, and what is it, anyway? – StasK May 02 '12 at 21:41
  • 2
    @StasK The DF argument may be "handwaving," but it's *standard* handwaving! For an excellent question about DF, see http://stats.stackexchange.com/questions/16921/how-to-understand-degrees-of-freedom. – whuber May 02 '12 at 22:02
  • 1
    @whuber, that's a good question and a good discussion, indeed. However, the degrees of freedom question is an oversimplification once we deviate from the i.i.d. $X_i\sim N(\mu,\sigma^2)$ situation. What are the degrees of freedom of the Satterthwaite $t$-test? How does one count independent observations in multilevel models? My take is that the concept of degrees of freedom is immensely helpful as a supplement, but it can never serve as the backbone of an argument. – StasK May 03 '12 at 18:17
0

The statement that the true answer for the sample variance is to divide by N-1 is a misstatement. There is no "true" answer. Dividing by N-1 gives you an unbiased estimate. As stated earlier the maximum likelihood estimate is obtained by dividing by N. It has a slight bias on the low side. I think it is unfortunate that in elementary statistics courses unbiasedness is emphasized. But an unbiased estimate is not always best. Best is most accurate and that is measured by mean square error which is the square of the bias plus the variance. For an unbiased estimate the bias squared is 0 but if the variance is large ther can be an estimate with small bias and small enough variance to have smaller mean square error than the unbiased one. Also as mentioned before there could be confusion between the population variance and the variance of the mean. The variance of the mean is 1/N times the population variance and the variance of the sample mean is 1/N times the sample variance. The term sample variance is commonly used to mean the unbiased estimator of the population variance and hence has the factor 1/(N-1).

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143