2

Why do "most" (basically all) statistics text books use $\hat{\sigma}^2=\hat{p} (1-\hat{p})$ as an estimator for the variance of a Bernoulli process which we know is biased. Should the first choice not be the bias corrected sample version $s^2 = \hat{p} (1-\hat{p}) \cdot n/(n-1)$, which is the default for non-binary data.

I am aware of the multiple related threads that seem similar to my question. However, I am not seeking a discussion on the merits of unbiased estimators or the origin/relevance of the $n/(n-1)$ correction.

My specific question is about teaching and communicating, namely why for binary data alone we seem to prefer the biased variance ($1/n$) when for all other cases, the default sample variance is chosen to be the unbiased version ($1/(n-1)$)

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Markus Loecher
  • 788
  • 3
  • 11
  • 1
    could you give at least one examples of the textbooks? – seanv507 Jan 17 '21 at 12:38
  • In *most* other cases, the test statistics based on the uncorrected estimator would follow distributions that are more complicated or onerous to tabulate, whereas that's not an issue in the Bernoulli case. Consideration of the tests for OLS, for instance, shows how one set of tables of t statistics will suffice *provided* the error variance is corrected by dividing by $n-p$ rather than $n$ itself. – whuber Jan 17 '21 at 18:46
  • 1
    @whuber Thank you for this VERY helpful and wise inferential perspective which would explain the limited upsides. I still wonder what the downside of using the unbiased version would be. Is it simply that p*(1-p) is so concise and elegant compared to p*(1-p)*n/(n-1)? – Markus Loecher Jan 19 '21 at 10:18
  • I think it might depend on the application, Markus. I would keep an eye out for statistical procedures that appear to involve factors that are functions of $(n-1)/n,$ because that would suggest they could be simplified by using the unbiased estimator. But unless the target of estimation is $p(1-p)$ itself, it is hard to see how adjusting for the bias would be relevant to most statistical procedures. In particular, you cannot simultaneously estimate $p$ and $p(1-p)$ in an unbiased way. – whuber Jan 19 '21 at 14:04
  • Thanks again, especially for the reminder that Bernoulli/Binomial are one-parameter distributions, which I agree is relevant for my question. – Markus Loecher Jan 21 '21 at 22:37

0 Answers0