Why do "most" (basically all) statistics text books use $\hat{\sigma}^2=\hat{p} (1-\hat{p})$ as an estimator for the variance of a Bernoulli process which we know is biased. Should the default not be the bias corrected sample version $s^2 = \hat{p} (1-\hat{p}) \cdot n/(n-1)$, which is the default for non-binary data.
Asked
Active
Viewed 133 times
0
-
3Bias is not the ultimate property to seek. For instance, $s^2$ has a larger variance than $\hat\sigma^2$. (And all Bayes estimators are biased.) – Xi'an Jan 13 '21 at 12:56
-
While true, your answer applies to all estimators and does not address my specific question why for binary data alone we seem to prefer the biased variance ("1/n") when for all other cases, the default sample variance is chosen to be the unbiased version ("1/(n-1)"). – Markus Loecher Jan 15 '21 at 10:06
-
I disagree with the closing of my question. None of the previous discussions addresses my very specific/narrow question as rephrased above. I am aware of the general scientific discourse on unbiased estimators which skirts the difference in textbook defaults for Bernoulli versus "non Bernoulli" sample variance. – Markus Loecher Jan 15 '21 at 10:11