9

Would it be wrong to say that a Riemann sum approximation of an integral

\begin{align} \int_a^b f(t) \mathrm{d}t \approx \sum_{k=1}^{n_\text{samples}} f(t^{\ast}_k)\Delta t, \end{align}

where $\Delta t = \left(b - a\right)/n_\text{samples}$, and where $t^{\ast}_k$ is the left or right end point or the midpoint of the sub-intervals is an unbiased estimate of the true integral?

The argument for Monte Carlo integral approximation with $N$ uniform samples in the interval $[a,b]$ seems to be that as the number of samples goes to infinity, then the approximation will be the exact integral with probability 1 (see e.g https://cs.dartmouth.edu/wjarosz/publications/dissertation/appendixA.pdf, page 153)

This same limit for the Riemann sum is (the definition of) the Riemann integral, hence I would argue that a Riemann sum is also unbiased.

According to a blog post (https://blog.evjang.com/2016/09/riemann-bias.html) which I found on Google, then the Riemann sum is biased because of the deterministic steps.

But since the argument for the Monte Carlo integration being unbiased uses that $N$ goes to infinity, I can't see why the same argument can't be used for the Riemann sum approximation.

If it indeed is wrong that the Riemann sum is unbiased, I would be happy if anyone could explain the differences in the arguments.

themainhatch
  • 101
  • 4

2 Answers2

14

It seems that you are swapping two different concepts here. The concepts are unbiased and consistent, which are properties of an estimator. A sequence of estimators $(T_n)_{n=1}^\infty$ is said to be unbiased for a quantity $\theta$ if, for all $n\,\in\mathbb{N}$,

$$ E[T_n] = \theta \quad.$$

It is said to be consistent if it converges in probability to $\theta$.

These are different concepts: the first says that, for every finite sample size, the average of your estimator is $\theta$. The other states that, as the sample sizes grow, the estimator getting arbitrarily close to $\theta$ with increasing probability.

Let $I = \int_a^bf(x)dx$ be your quantity of interest (assume it exists). What the most basic Monte Carlo method does is to observe that

$$I = \int_a^bf(x)dx = (b-a)\int_a^bf(x)\frac{1}{b-a}dx = (b-a)E[f(X)] \quad.$$

In the last line, we wrote the integral as being the expectation of $f(X)$, where $X$ has a uniform distribution in $(a,b)$. Hence, if we sample i.i.d. random variables $(X_i)_{i=1}^n$ with $X_1 \sim U((a,b))$, then the estimator

$$T_n = \frac{(b-a)}{n}\sum_{i=1}^nf(X_i) \quad,$$

is easily shown to be unbiased for $I$.

When you think of Riemann sums, it is usual to take a deterministic partition. If it is deterministic, then the expected value for any fixed sample size is the value of the summation it self, which in general is not the value of the integral.

Lucas Prates
  • 1,183
  • 4
  • 15
  • Thanks for the detailed explanation. – themainhatch Jul 24 '21 at 13:13
  • 2
    _"for every finite sample size"_ — choose a sample of size 1. The average of this sample must be $\theta$. This then implies that every element must equal $\theta$, thus an unbiased estimator is an exact computer of $\theta$. Is this really what you supposed to define? – Ruslan Jul 24 '21 at 21:50
  • 4
    @Ruslan The "sample size" $n$ here is not the set over which the average (that must equal $\theta$) is taken. The estimator is computed from the *entire sample*, and what must equal $\theta$ is its average (expectation) over *all possible samples* of size $n$. For example, an unbiased estimator of the mean from a sample of size $1$ is just the sample value. This does not always equal the true mean $\theta$, but its expectation value over samples of size $1$ equals $\theta$. – nanoman Jul 25 '21 at 00:47
  • As @nanoman has well said, the expectation is not the average over a fixed sample. The expectation is taken over the **process that generates** samples of size $1$. What is fixed is the sample size $n$, not the sample values. – Lucas Prates Jul 25 '21 at 15:34
6

Any constant is a biased estimator of any different constant

Since you are using a deterministic procedure here, your Riemann sum depends only on $n$, so it is a sequence of constants. Applying the concept of statistical bias to constants is simple --- any constant is a biased estimator of any different constant and an unbiased estimator of itself. So, for example, $3$ is a biased estimator of $2$, but it is an unbiased estimator of $3$.

Your Riemann sum is generally going to be a biased estimator for the corresponding integral because you are selecting the points $t_k^*$ deterministically within the interval and so your estimator is a constant. There are exceptions, for functions where the Riemann sum happens to be exactly equal to the integral (e.g., piecewise linear functions). When the Riemann sum gives a different value to the integral it is biased (in the same way that $3$ is a biased estimator of $2$). When the Riemann sum gives the same value as the integral it is unbiased (in the same way that $3$ is an unbiased estimator of $3$). Irrespective of whether the Riemann sum is a biased estimator or not, it will still be a consistent estimator, since it converges to the true integral as $n \rightarrow \infty$; indeed, this is the essence of the Riemann integral.

Now, if you were to select the $t$ point uniformly at random within the interval, the resulting Riemann sum would be an unbiased estimator of the integral. This would be a variation of estimation by importance sampling, where you are varying things by generating your points conditionally within segments of a partition.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • 2
    What definition or sense of "biased" are you invoking here?? – whuber Jul 24 '21 at 14:18
  • The usual one --- that the expected value of the sum (which is deterministic anyway so has expectation equal to itself) is not exactly equal to the integral. – Ben Jul 24 '21 at 22:48
  • 1
    Sometimes it will be equal, sometimes not, depending on the integrand, the intervals, and the kind of Riemann sum. But since there's no probability model in evidence, it doesn't look like your "biased" has the same meaning it does in estimation theory. – whuber Jul 25 '21 at 14:28
  • It is biased in the same way that $3$ is a biased estimator of $2$, using the standard statistical definition. In the usual case where they are is not equal, let $\hat{R}(n)$ denote the Reimann sum (with $R$ being the integral). Since the Riemann sum is deterministic you have $\text{Bias}(\hat{R},R) = \mathbb{E}(\hat{R}(n)) - R = \hat{R}(n) - R \neq 0$, so it is biased in the usual meaning of the term. While it is true that the Riemann sum might exactly equal the integral, this is why I say that it is "generally" biased ---i.e., except in the coincidental case. – Ben Jul 25 '21 at 23:11
  • Your inequality does not necessarily follow. For instance, a well-chosen Riemann sum is always unbiased for any piecewise linear function. This, coupled with the fact you are shoehorning a deterministic procedure into a statistical setting, makes it a little difficult to understand what point you're trying to make. – whuber Jul 26 '21 at 17:22
  • 1
    I'm not really trying to make any particular point --- just to answer the question that was asked. My answer would be largely the same if the OP had asked whether $3$ is a biased estimator of $2$. Re the exceptions, I already conceded there are exceptions, both in the question and again in my above comment, so why continue to approach the matter as if I don't know this? When you have a mathematical proposition $P(x)$ that is true for some $x$ and false for (many more) others, and someone asks "Is it wrong to say "$P(x)$ is true in general", you tell them, yes, generally that is wrong. – Ben Jul 26 '21 at 21:53
  • 1
    Also, I reject your characterisation of "shoehorning" --- all constants are special cases of random variables (when appropriately defined as mappings from the sample space to that constant), so all statistical procedures that apply to random variables should apply to constants (even when this renders them trivial). This is an important check on working for any statistical procedure --- i.e., does it make sense for constants. In this case, any constant is a biased estimator of another different constant, so yes, it does. – Ben Jul 26 '21 at 21:57
  • 1
    Rather than leave all this in the comments, I have decided to flesh out my answer to include this material in the body. – Ben Jul 26 '21 at 22:09
  • Wouldn't it just be a whole lot clearer to explain that the Riemann sum, although an *approximation,* is not an *estimate* in the statistical sense of the word? Even though you can frame it as such, that appears not to be conceptually helpful. It seems only to risk propagating some of the confusion evident in the question itself. – whuber Jul 26 '21 at 22:46
  • As you concede in your second sentence, it *is* an estimator in the statistical sense when constructed to be. Consequently, all the normal questions about estimators (bias, consistency, etc.) apply. It is possible that the OP misconceives the Riemann sum as a non-deterministic estimator, but my answer disabuses him of that notion and notes that it is a constant. As I said, I see no reason that it is unhelpful or misleading to examine the statistical properties of estimators that are constants; that provides a useful check on the validity of broader statistical concepts. – Ben Jul 26 '21 at 23:46
  • The [linked blog post](https://blog.evjang.com/2016/09/riemann-bias.html) in the question is precisely about comparing the Riemann sum to stratified sampling, contrasting the bias properties of the deterministic method to the non-deterministic method. In view of this context, IMHO it is *more* misleading to confuse the issue by denying that the Riemann sum is an estimator. – Ben Jul 26 '21 at 23:52