When the Central Limit Theorem and the Law of Large Numbers disagree

Question

This is essentially a replication of a question I found over at math.se, which didn't get the answers I hoped for.

Let $\{ X_i \}_{i \in \mathbb{N}}$ be a sequence of independent, identically distributed random variables, with $\mathbb{E}[X_i] = 1$ and $\mathbb{V}[X_i] = 1$.

Consider the evaluation of

$$ \lim_{n \to \infty} \mathbb{P}\left(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \leq \sqrt{n}\right) $$

This expression has to be manipulated since, as is, both sides of the inequality event tend to infinity.

A) TRY SUBTRACTION

Before considering the limiting statement, subtract $\sqrt{n}$ from both sides:

$$\lim_{n \to \infty} \mathbb{P}\left(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i -\sqrt{n} \leq \sqrt{n}-\sqrt{n} \right) = \lim_{n \to \infty} \mathbb{P}\left(\frac{1}{\sqrt{n}} \sum_{i=1}^n (X_i - 1) \leq 0\right) \\ = \Phi(0) = \frac{1}{2}$$

the last equality by the CLT, where $\Phi()$ is the standard normal distribution function.

B) TRY MULTIPLICATION

Multiply both sides by $1/\sqrt{n}$ $$\lim_{n \to \infty} \mathbb{P}\left(\frac {1}{\sqrt{n}}\cdot \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \leq \frac {1}{\sqrt{n}}\cdot\sqrt{n} \right) = \lim_{n \to \infty} \mathbb{P}\left(\frac{1}{n} \sum_{i=1}^n X_i \leq 1\right) $$

$$= \lim_{n \to \infty} \mathbb{P}\left(\bar X_n \leq 1\right) = \lim_{n \to \infty}F_{\bar X_n}(1) = 1$$

where $F_{\bar X_n}()$ is the distribution function of the sample mean $\bar X_n$, which by the LLN converges in probability (and so also in distribution) to the constant $1$, hence the last equality.

So we get conflicting results. Which is the right one? And why the other is wrong?

Answer 1

The error here is likely in the following fact: convergence in distribution implicitly assumes that $F_n(x)$ converges to $F(x)$ at points of continuity of $F(x)$. As the limit distribution is of a constant random variable, it has a jump discontinuity at $x=1$, hence it is incorrect to conclude that the CDF converges to $F(x)=1$.

Answer 2

For iid random variables $X_i$ with $E[X_i]= \operatorname{var}(X_i)=1$ define \begin{align}Z_n &= \frac{1}{\sqrt{n}}\sum_{i=1}^n X_i,\\ Y_n &= \frac{1}{{n}}\sum_{i=1}^n X_i. \end{align} Now, the CLT says that for every fixed real number $z$, $\lim_{n\to\infty} F_{Z_n}(z) = \Phi(z-1)$. The OP applies the CLT to evaluate $$\lim_{n\to\infty}P\left(Z_n \leq \frac{1}{\sqrt{n}}\right) = \Phi(0) = \frac 12.$$

As the other answers as well as several of the comments on the OP's question have pointed out, it is the OP's evaluation of $\lim_{n\to\infty} P(Y_n \leq 1)$ that is suspect. Consider the special case when the iid $X_i$ are discrete random variables taking on values $0$ and $2$ with equal probability $\frac 12$. Now, $\sum_{i=1}^n X_i$ can take on all even integer values in $[0,2n]$ and so when $n$ is odd, $\sum_{i=1}^n X_i$ cannot take on value $n$ and hence $Y_n = \frac 1n \sum_{i=1}^n X_i$ cannot take on value $1$. Furthermore, since the distribution of $Y_n$ is symmetric about $1$, we have that $P(Y_n \leq 1) = F_{Y_n}(1)$ has value $\frac 12$ whenever $n$ is odd. Thus, the sequence of numbers $$P(Y_1 \leq 1), P(Y_2 \leq 1), \ldots, P(Y_n \leq 1), \ldots$$ contains the subsequence $$P(Y_1 \leq 1), P(Y_3 \leq 1), \ldots, P(Y_{2k-1} \leq 1), \ldots$$ in which all the terms have value $\frac 12$. On the other hand, the subsequence $$P(Y_2 \leq 1), P(Y_ 4\leq 1), \ldots, P(Y_{2k} \leq 1), \ldots$$ is converging to $1$. Hence, $\lim_{n\to\infty} P(Y_n \leq 1)$ does not exist and claims of convergence of $P(Y_n\leq 1)$ to 1 must be viewed with a great deal of suspicion.

Answer 3

Your first result is the correct one. Your error occurs in the second part, in the following erroneous statement:

$$\lim_{n \rightarrow \infty} F_{\bar{X}_n}(1) = 1.$$

This statement is false (the right-hand-side should be $\tfrac{1}{2}$) and it does not follow from the law of large numbers as asserted. The weak law of large numbers (which you invoke) says that:

$$\lim_{n \rightarrow \infty} \mathbb{P} \Big( |\bar{X}_n - 1| \leqslant \varepsilon \Big) = 1 \quad \quad \text{for all } \varepsilon > 0.$$

For all $\varepsilon > 0$ the condition $|\bar{X}_n - 1| \leqslant \varepsilon$ spans some values where $\bar{X}_n \leqslant 1$ and some values where $\bar{X}_n > 1$. Hence, it does not follow from the LLN that $\lim_{n \rightarrow \infty} \mathbb{P} ( \bar{X}_n \leqslant 1 ) = 1$.

Answer 4

Convergence in probability implies convergence in distribution. But... what distribution? If the limiting distribution has a jump discontinuity then the limits become ambiguous (because multiple values are possible at the discontinuity).

where $F_{\bar X_n}()$ is the distribution function of the sample mean $\bar X_n$, which by the LLN converges in probability (and so also in distribution) to the constant $1$,

This is not right, and it is also easy to show that it can not be right (different from the disagreement between CLT and LLN). The limiting distribution (which can be seen as the limit for a sequence of normal distributed variables) should be:

$$F_{\bar{X}_\infty}(x) = \begin{cases} 0 & \text{for } x<1 \\ 0.5& \text{for } x=1\\ 1 & \text{for } x>1 \end{cases}$$

for this function you have that, for any $\epsilon>0$ and every $x$, the difference $|F_{\bar{X}_n}(x)-F_{\bar{X}_\infty}(x)|<\epsilon$ for sufficiently large $n$. This would fail if $F_{\bar{X}_\infty}(1)=1$ instead of $F_{\bar{X}_\infty}(1)=0.5$

---

Limit of a normal distribution

It may be helpful to explicitly write out the sum used to invoke the law of large numbers.

$$\bar{X}_n=\frac{1}{n}\sum_{i=1}^n X_i \sim N(1,\frac{1}{n}) $$

^{The limit $n\to \infty$ for $\hat{X}_n$ is actually equivalent to the Dirac Delta function when it is represented as the limit of the normal distribution with the variance going to zero.}

Using that expression it is more easy to see what is going on under the hood, rather than using the ready-made laws of the CLT an LLN which obscure the reasoning behind the laws.

---

Convergence in probability

The law of large numbers gives you 'convergence in probability'

$$\lim_{n \to \infty} P(|\bar{X}_n-1|>\epsilon) =0 $$

with $\epsilon > 0$

^{An equivalent statement could be made for the central limit theorem with $\lim_{n \to \infty} P(|\frac{1}{\sqrt{n}}\sum \left( X_i-1 \right)|>\frac{\epsilon}{n}) =0 $}

It is wrong to state that this implies $$\lim_{n \to \infty} P(|\bar{X}_n-1|>0) =0 $$

^{It is less nice that this question is cross-posted so early (confusing, yet interesting to see the different discussions/approaches math vs stats, so not that too bad). The answer by Michael Hardy on the math stackexchange deals with it very effectively in terms of the strong law of large numbers (the same principle as the accepted answer from drhab in the cross posted question and Dilip here). We are almost sure that a sequence $\bar{X}_1, \bar{X}_2, \bar{X}_3, ... \bar{X}_n$ converges to 1, but this does not mean that $\lim_{n \to \infty} P(\bar{X}_n = 1)$ will be equal to 1 (or it may not even exist as Dilip shows). The dice example in the comments by Tomasz shows this very nicely from a different angle (instead of the limit not existing, the limit goes to zero). The mean of a sequence of dice rolls will converge to the mean of the dice but the probability to be equal to this goes to zero.}

---

Heaviside step function and Dirac delta function

The CDF of $\bar{X}_n$ is the following:

$$F_{\bar{X}_n}(x) = \frac{1}{2} \left(1 + \text{erf} \frac{x-1}{\sqrt{2/n}} \right)$$

with, if you like, $\lim_{n \to \infty} F_{\bar{X}_n}(1) = 0.5$ (related to the Heaviside step function, the integral of the Dirac delta function when viewed as the limit of normal distribution).

---

I believe that this view intuitively resolves your question regarding 'show that it is wrong' or at least it shows that the question about understanding the cause of this disagreement of CLT and LLN is equivalent to the question of understanding the integral of the Dirac delta function or a sequence of normal distributions with variance decreasing to zero.

Answer 5

I believe it should be clear by now that "the CLT approach" gives the right answer.

Let's pinpoint exactly where the "LLN approach" goes wrong.

Starting with the finite statements, it is clear then that we can equivalently either subtract $\sqrt{n}$ from both sides, or multliply both sides by $1/\sqrt{n}$. We get

$$\mathbb{P}\left(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \leq \sqrt{n}\right)=\mathbb{P}\left(\frac{1}{\sqrt{n}} \sum_{i=1}^n(X_i-1) \leq 0\right) = \mathbb{P}\left(\frac{1}{n} \sum_{i=1}^nX_i \leq 1\right)$$

So if the limit exists, it will be identical. Setting $Z_n = \frac{1}{\sqrt{n}} \sum_{i=1}^n(X_i-1)$, we have, using distribution functions

$$\mathbb{P}\left(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \leq \sqrt{n}\right)= F_{Z_n}(0) = F_{\bar X_n}(1)$$

...and it is true that $\lim_{n\to \infty}F_{Z_n}(0)= \Phi(0) = 1/2$.

The thinking in the "LLN approach" goes as follows: "We know from the LLN that $\bar X_n$ converges in probabililty to a constant. And we also know that "convergence in probability implies convergence in distribution". So, $\bar X_n$ converges in distribution to a constant". Up to here we are correct.
Then we state: "therefore, limiting probabilities for $\bar X_n$ are given by the distribution function of the constant at $1$ random variable",

$$F_1(x) = \cases {1 \;\;\;\;x\geq 1 \\ 0 \;\;\;\;x<1} \implies F_1(1) = 1$$

... so $\lim_{n\to \infty} F_{\bar X_n}(1) = F_1(1) = 1$...

...and we just made our mistake. Why? Because, as @AlexR. answer noted, "convergence in distribution" covers only the points of continuity of the limiting distribution function. And $1$ is a point of discontinuity for $F_1$. This means that $\lim_{n\to \infty} F_{\bar X_n}(1)$ may be equal to $F_1(1)$ but it may be not, without negating the "convergence in distribution to a constant" implication of the LLN.

And since from the CLT approach we know what the value of the limit must be ($1/2$). I do not know of a way to prove directly that $\lim_{n\to \infty} F_{\bar X_n}(1) = 1/2$.

Did we learn anything new?

I did. The LLN asserts that

$$\lim_{n \rightarrow \infty} \mathbb{P} \Big( |\bar{X}_n - 1| \leqslant \varepsilon \Big) = 1 \quad \quad \text{for all } \varepsilon > 0$$

$$\implies \lim_{n \rightarrow \infty} \Big[ \mathbb{P} \Big( 1-\varepsilon <\bar{X}_n \leq 1\Big) + \mathbb{P} \Big( 1 <\bar{X}_n \leq 1+\varepsilon\Big)\Big] = 1$$

$$\implies \lim_{n \rightarrow \infty} \Big[ \mathbb{P} \Big(\bar{X}_n \leq 1\Big) + \mathbb{P} \Big( 1 <\bar{X}_n \leq 1+\varepsilon\Big)\Big] = 1$$

The LLN does not say how is the probability allocated in the $(1-\varepsilon, 1+\varepsilon)$ interval. What I learned is that, in this class of convergence results, the probability is at the limit allocated equally on the two sides of the centerpoint of the collapsing interval.

The general statement here is, assume

$$X_n\to_p \theta,\;\;\; h(n)(X_n-\theta) \to_d D(0,V)$$

where $D$ is some rv with distribution function $F_D$. Then

$$\lim_{n\to \infty} \mathbb P[X_n \leq \theta] = \lim_{n\to \infty}\mathbb P[h(n)(X_n-\theta) \leq 0] = F_D(0)$$

...which may not be equal to $F_{\theta}(0)$ (the distribution function of the constant rv).

Also, this is a strong example that, when the distribution function of the limiting random variable has discontinuities, then "convergence in distribution to a random variable" may describe a situation where "the limiting distribution" may disagree with the "distribution of the limiting random variable" at the discontinuity points. Strictly speaking, the limiting distribution for the continuity points is that of the constant random variable. For the discontinuity points we may be able to calculate the limiting probability, as "separate" entities.