2

I am wondering whether there is a general recipe for finding unbiased and consistent estimators of some non-random quantity.

For concreteness, I will discuss only discrete probability distributions with a finite support, but I would appreciate answers also for the continuous case.

Suppose that there is a discrete distribution over $M$ values:

$p_1=P(x_1)\\p_2=P(x_2)\\...\\p_M=P(x_M)$

Denote $x=(x_1,x_2,...,x_M)$ and $p=(p_1,p_2,...,p_M)$. The $x_i$ values are all different. The $p_i$ values are all non-negative and add up to $1$.

Any quantity that we may want to estimate can be written as $g(x,p)$ for some function $g$.

I am assuming that we know the function $g$ exactly, and that we know the value of $M$, but we do not know the values of $x$ and $p$.

So, given the function $g$ and given $M$:

  • Is there a general method that tells us whether an unbiased estimator for $g(x,p)$ exists?
  • Is there a general method that can find an unbiased estimator for $g(x,p)$, whenever it exists?
  • Is there a general method that tells us whether a consistent estimator for $g(x,p)$ exists?
  • Is there a general method that can find a consistent estimator for $g(x,p)$, whenever it exists?

(It may be the case that, given the function $g$ and given $M$, an unbiased or consistent estimator can be found only provided that $x$ and $p$ belong to some subset of values. In this case, I also wonder about whether there are methods to find this set).

For the sake of completeness, let me provide some definitions:

  • An estimator is a function $f(y_1,y_2,...)$ that takes as input a finite sequence of numbers $y_1,y_2,...\in \mathbb{R}$, and returns a value from $\mathbb{R}$. ($f$ does not depend on $p$ or $x$ defined above)

  • The estimator is said to be an unbiased estimator of $g(x,p)$ (defined above) (for a specific $N$), if for all $x$ and $p$, the expected value of $f(Y_1,Y_2,...,Y_N)$ equals $g(x,p)$, where $Y_1,Y_2,...,Y_N$ are independent random variables whose distribution is defined by $x$ and $p$ (see above).

  • (informal) The estimator is said to be a consistent estimator of $g(x,p)$ (defined above), if for all $x$ and $p$, in the limit of large $N$, the expected value of $f(Y_1,Y_2,...,Y_N)$ has limit $g(x,p)$, where $Y_1,Y_2,...,Y_N$ are independent random variables whose distribution is defined by $x$ and $p$ (see above).

Lior
  • 517
  • 3
  • 14
  • 1
    On this forum, you needn't give definintions of estimators, unbiasedness, or consistency; these are all well-known concepts among the practitioners here. (And you certainly shouldn't give definitions that are wrong, or are at cross-purposes to the notation in your question.) – Ben Dec 27 '20 at 01:36
  • Of course they are well known. But I wanted to reduce ambiguity. For example, I wanted to make it clear that I'm talking about the IID case (sure, this is the standard case, but by no means is it the only case). Also, there are some subtleties involved here. For example, there could be a known unbiased estimator for some parameter given that we know the parametric family of the distribution. But my definitions made it clear that I am do not want to assume a parametric family ("for all x and p..."). Did I give a wrong definition anywhere? – Lior Dec 27 '20 at 01:54
  • If you want to add assumptions to your question, it is best to add them explicitly rather than trying to build them into "definitions" of statistical concepts. For example, your definition of the estimator uses a multiset input and therefore ignores the order of the input random variables, which is not the proper definition. If you want to specify that your estimator is invariant to permutations of the inputs, just say that explicitly, rather than giving a false definition of what an estimator is. – Ben Dec 27 '20 at 01:59
  • Ok, thanks for the comment. I'll take it into consideration next time. – Lior Dec 27 '20 at 02:00
  • This question was closed because it "needs to be more focused.". I would appreciate if anyone can explain in what way is the question not focused enough? I think that the question is well-defined, and that the scope of the question is very specific. (True, I ask both about unbiased estimators and consistent estimators, but these are very related and I think it makes sense to ask about both of these together). I think that the question is intelligible so that a concrete, well-defined answer could be given in principle (assuming of course that the answer is known to someone). – Lior Dec 27 '20 at 04:18
  • I voted to close this post because it is comprised of four extremely general questions, which is not a good fit for this site. – whuber Dec 27 '20 at 14:46
  • 1
    You should do a search on this site for answers dealing with unbiased and consistent estimators, and ask more specific questions on this site if none of them answer those questions. – Peter O. Dec 27 '20 at 19:20
  • @Peter O. but the exact intent of the question is to be broad. I'm wondering whether there is a *general* algorithm for unbiased estimators, not an algorithm for a particular case. In fact, if someone would give me an answer for, e.g, a particular parametric family of distributions, I would not be satisfied, because that's not what I wanted to ask. And it's 4 questions, but they are extremely related and not independent, as you can see. Is there any argument not valid in the this comment? – Lior Dec 28 '20 at 11:25
  • @Peter O. I would appreciate if you could give me a hint for a different way to phrase my question, given that my interest is in the existence/absence of a *general* algorithm. – Lior Dec 28 '20 at 11:30
  • An analogous question on another site might be "is there a general method to cook vegetables?" Such broad questions do not work on SE sites. – whuber Dec 28 '20 at 13:48
  • @whuber Please consider two additional comments: 1)Let's assume for a moment that there is a general method for finding an unbiased estimator, and that some user of this site knows about it. Then, all it takes for this user is to explain this algorithm in the answer, or to give a reference. This would be a precise answer to my precise question, despite my question requiring a general method. Similarly for the case when there is a known proof that no such algorithm exists. Your example about cooking vegetables is not analogous, because my question might really have a precise answer! – Lior Dec 28 '20 at 21:42
  • @whuber 2) I can easily find questions on this network which are high-voted, answered, and ask about general methods. Maybe this is just "survival bias", but here are some examples: https://stats.stackexchange.com/q/89154/112286 https://cs.stackexchange.com/q/2789/10385 https://math.stackexchange.com/q/3829335/273547 – Lior Dec 28 '20 at 21:43
  • Let $D$ be the distribution determined by $x$ and $p$. The answer to your first question is given in Halmos (1946): An unbiased estimator for $g$ exists if and only if $g$ is homogeneous over $D$ of degree $k \leq N$. – Peter O. Dec 28 '20 at 21:57
  • What other sites on SE do is up to them. The question here about SEs is more specific than any of yours. Conceivably, if you limited your post to one of your four questions it might have a definite answer, as @Peter indicates. – whuber Dec 28 '20 at 22:21
  • @whuber Ok. Thanks for your comments. After reflecting, I still feel that this question should be eligible for an answer here as-it-is, but I acknowledge that I might be under a Dunning–Kruger bias. – Lior Dec 28 '20 at 22:33

1 Answers1

5

Let $D$ be the distribution determined by $x$ and $p$, and let $\phi$ be the mean of $(g(x, p)_0, g(x, p)_1, ..., g(x, p)_N)$, where $g(x, p)_i$ are i.i.d. random variables distributed as $g(D)$.

The answer to your first question is given in Halmos (1946): An unbiased estimator for $\phi$ exists if and only if $\phi$ is a homogeneous function over $D$ of degree at most $N$. In particular, if $M$ is 2 and $x$ is {0, 1} (so that the distribution is Bernoulli), then the only functions for which an unbiased estimator exists are polynomials of degree at most $N$ (Wästlund 1999, Goyal and Sigman 2012; see also this answer).

However, if $N$ is unlimited and $M$ is 2, then $g$ has an unbiased estimator in some subset of (0, 1) if and only if $g$ is continuous and is identically 0, identically 1, or polynomially bounded away from 0 and 1 in that subset (Keane and O'Brien 1994, Łatuszyński et al. 2009/2011).

I am not aware of any results that show whether a consistent estimator exists for a function if and only if an unbiased estimator exists for that function.

For more general distributions, Jacob and Thiery (2015) investigates the following question: Given a sequence of unbiased estimators, can we turn that sequence into an unbiased estimator of $g$ that is nonnegative almost surely? Without knowledge on what values $x$ can take on, the answer is no. However, if upper and lower bounds on $x$ are known, for example, this is possible for certain functions $g$; see the paper for details. See also Glynn (2016).

REFERENCES:

  • Wästlund, J., "Functions arising by coin flipping", 1999.
  • Goyal, V. and Sigman, K., 2012. On simulating a class of Bernstein polynomials. ACM Transactions on Modeling and Computer Simulation (TOMACS), 22(2), pp.1-5.
  • Keane, M. S., and O'Brien, G. L., "A Bernoulli factory", ACM Transactions on Modeling and Computer Simulation 4(2), 1994.
  • Jacob, P.E., Thiery, A.H., "On nonnegative unbiased estimators", Ann. Statist., Volume 43, Number 2 (2015), 769-784.
  • Glynn, P.W., "Exact simulation vs exact estimation", Proceedings of the 2016 Winter Simulation Conference, 2016.
  • Halmos, P.R., "The theory of unbiased estimation", Annals of Mathematical Statistics 17(1), 1946.
Peter O.
  • 863
  • 1
  • 4
  • 18