11

Almost all books teach Law of Large Numbers first then the Central Limit Theorem one next. But what are the relationship and differences between two theorem?

My attempt:

Here is my understanding (Informally),

  • Law of Large Numbers says if we have very large i.i.d sample, the mean will converge to a number.

  • Central Limit Theorem is similar idea, but require "less data" (I agree with Tim in the comment that both refer to $\infty$, but I cannot find a better word to describe..). Comparing to Law of Large Numbers, because it require "less data", it has a relaxation in conclusion: not converge to a number, it converge to a normal distribution.

Thanks for Yuri and Antoni's links, I think my question is different from the two questions linked.

For question

Central limit theorem versus law of large numbers

It focusing on more math, where I want more "intuitive explainable" on application domain or the scope on LLN and CLT, but not only the math derivation.

For question

Central limit theorem and the law of large numbers

It is closer to what I want to ask, but still focusing on "how then can CLT also converge to the expected value at the same time?"

Haitao Du
  • 32,885
  • 17
  • 118
  • 213
  • 2
    Both theorems refer to $n \to \infty$ – Tim May 10 '17 at 14:06
  • 1
    There are many on the site (not really surprising), some of them with great answers and even this pearl of wisdom by @cardinal ["The Central Limit Theorem is about the journey and the Strong Law of Large Numbers is about the destination."](https://stats.stackexchange.com/q/49123/67822) – Antoni Parellada May 10 '17 at 14:53
  • @AntoniParellada I love the term "journey" and "destination". I am trying to describe similar thing in "my attempt". Do not know if we can get a good intuitive answer here. – Haitao Du May 10 '17 at 15:42
  • @whuber's answer seems pretty intuitively-oriented to me. – gung - Reinstate Monica May 10 '17 at 16:12
  • I do not agree with your intuition that central limit theorem "requires less data" (in your sense). In my understanding, CLT requires extra assumptions on top of those needed for LLN. So you can have LLN without CLT but not the other way around. However, I am talking about very simple cases here, such as convergence of OLS estimator of a normal linear model or the like. The answers linked in your question go deeper and consider more complex cases. – Richard Hardy May 10 '17 at 16:41

1 Answers1

6

The law of large numbers stems from two things:

  • The variance of the estimator of the mean goes like ~ 1/N
  • Markov's inequality

You can do it with a few definitions of Markov's inequality:

\begin{eqnarray} P(\vert X \vert \ge a) \le & \frac{\mathrm{E}\left(X\right)}{a} \end{eqnarray}

and statistical properties of the estimatory of the mean: \begin{eqnarray} \bar{X} &=& \sum_{n=1}^N \frac{x}{N} \\ \mathrm{E}\left(\bar{X} \right) &=& \mu \\ \mathrm{Var}\left( \bar{X}^2 \right) &=& \frac{\sigma^2}{N} \end{eqnarray}

Doing a quick trick we find:

\begin{eqnarray} P( (\bar{X}-\mu)^2 \ge \epsilon^2 ) &\le& \frac{\mathrm{E}\left((\bar{X}-\mu)^2 \right)}{\epsilon^2}\\ & \le & \frac{\sigma^2}{N \epsilon} \end{eqnarray}

So as $N \to \infty$, the right hand side goes to zero, and so find the $\bar{X}$ becomes arbitrarily $\epsilon$-close to the real mean, $\mu$.

The central limit theorem hinges on:

  • As you sum up a bunch for random variables $x_1+x_2+x_3$, no matter their distributions, you are essentially "mixing" the probability densities $P(x_1), P(x_2), P(x_3)$ -- technically you're convolving them in $x$ space and multiplying their characteristic fucntions $\phi(k)$ in fourier space -- making the resulting "aggregate" pdf $P(x_1+x_2+x_3)$ look more and more Gaussian.

The proof of why this happens is a bit tricky. But stems from the fact that pdf's are upper bounded by 1 and positive definite, so when you multiply / convolve them over and over again, they are efficiently described in both fourier $k$ and $x$ space by a log taylor expansion up to second order -- meaning you only need to now the first and second cumulants, the mean $\mu$ and variance $\sigma^2$.

rspeare
  • 86
  • 1