21

What are some theorems which might explain (i.e., generatively) why real-world data might be expected to be normally distributed?

There are two that I know of:

  1. The Central Limit Theorem (of course), which tells us that the sum of several independent random variables with mean and variance (even when they are not identically distributed) tends towards being normally distributed

  2. Let X and Y be independent continuous RV's with differentiable densities such that their joint density only depends on $x^2$ + $y^2$. Then X and Y are normal.

(cross-post from mathexchange)

Edit: To clarify, I am not making any claims about how much real world data is normally distributed. I am just asking about theorems that can give insight into what sort of processes might lead to normally distributed data.

Community
  • 1
anonymous
  • 755
  • 1
  • 4
  • 13
  • 7
    You might find interesting related material in our thread at http://stats.stackexchange.com/questions/4364. To avoid potential confusion among some readers, I would like to add (and I hope this was your intention) that your question should not be read as suggesting that all or even most actual datasets can be adequately approximated by a normal distribution. Rather, in certain cases when certain conditions hold, it could be useful to employ a normal distribution as a frame of reference for understanding or interpreting the data: so what might those conditions be? – whuber Dec 02 '14 at 23:19
  • Thank you for the link! And that is exactly right, thank you for the clarification. I will edit it to the original post. – anonymous Dec 02 '14 at 23:25
  • @user43228, "***There are, of course, tons of other distributions that arise in real world problems that don’t look normal at all.***" http://www.askamathematician.com/2010/02/q-whats-so-special-about-the-gaussian-distribution-a-k-a-a-normal-distribution-or-bell-curve – Pacerier Jun 03 '15 at 12:43

4 Answers4

19

Many limiting distributions of discrete RVs (poisson, binomial, etc) are approximately normal. Think of plinko. In almost all instances when approximate normality holds, normality kicks in only for large samples.

Most real-world data are NOT normally distributed. A paper by Micceri (1989) called "The unicorn, the normal curve, and other improbable creatures" examined 440 large-scale achievement and psychometric measures. He found a lot of variability in distributions w.r.t. their moments and not much evidence for (even approximate) normality.

In a 1977 paper by Steven Stigler called "Do Robust Estimators Work with Real Data" he used 24 data sets collected from famous 18th century attempts to measure the distance from the earth to the sun and 19th century attempts to measure the speed of light. He reported sample skewness and kurtosis in Table 3. The data are heavy-tailed.

In statistics, we assume normality oftentimes because it makes maximum likelihood (or some other method) convenient. What the two papers cited above show, however, is that the assumption is often tenuous. This is why robustness studies are useful.

Tim
  • 108,699
  • 20
  • 212
  • 390
bsbk
  • 1,127
  • 11
  • 19
  • 4
    Most of this post is great, but the introductory paragraph bothers me because it could so easily be misinterpreted. It *seems* to say--rather explicitly--that in general, a "large sample" will look normally distributed. In light of your subsequent remarks I don't believe you really meant to say that. – whuber Dec 02 '14 at 23:25
  • I should have been more clear - I'm not suggesting that most real world data is normally distributed. But that is a great point to raise. And I'm assuming what you mean is that binomial distribution with large n is normal, and that poisson distribution with large mean is normal. What other distributions tend towards normality? – anonymous Dec 02 '14 at 23:27
  • Thanks, I edited the first paragraph. See Wald and Wolfowitz (1944) for a theorem on linear forms under permutation, for example. I.e., they showed the two sample t statistic under permutation is asymptotically normal. – bsbk Dec 02 '14 at 23:31
  • A sampling distribution is not a "real world dataset"! Perhaps the difficulty I am having with apparent inconsistencies in your post stems from this confusion between distribution and data. Perhaps it stems from a lack of clarity about what "limiting" process you actually have in mind. – whuber Dec 02 '14 at 23:33
  • 3
    The original question was about explaining "generatively" how normal real-world data might come about. It is conceivable that real data might be generated from a binomial or poisson process, both of which can be approximated by the normal distribution. The op asked for other examples and the one that came to mind was the permutation distribution, which is asymptotically normal (in the absence of ties). I can't think of a way off-hand that real data would be generated from that distribution so maybe that one is a stretch. – bsbk Dec 02 '14 at 23:43
11

There is also an information theoretic justification for use of the normal distribution. Given mean and variance, the normal distribution has maximum entropy among all real-valued probability distributions. There are plenty of sources discussing this property. A brief one can be found here. A more general discussion of the motivation for using Gaussian distribution involving most of the arguments mentioned so far can be found in this article from Signal Processing magazine.

Igor
  • 403
  • 2
  • 10
  • 6
    This is backwards, as I understand it. It's about how making the assumption of normality is in a strictly defined sense a weak assumption. I don't see what that implies about real-world data. You might as well argue that curves are typically straight because that's the simplest assumption you can make about curvature. Epistemology does not limit ontology! If the reference you cite goes beyond that, please spell out the arguments. – Nick Cox Dec 09 '14 at 19:58
3

In physics it is CLT which is usually cited as a reason for having normally distributed errors in many measurements.

The two most common errors distributions in experimental physics are normal and Poisson. The latter is usually encountered in count measurements, such as radioactive decay.

Another interesting feature of these two distributions is that a sum of random variables from Gaussian and Poisson belongs to Gaussian and Poisson.

There are several books on statistics in experimental sciences such as this one:Gerhard Bohm, Günter Zech, Introduction to Statistics and Data Analysis for Physicists, ISBN 978-3-935702-41-6

Aksakal
  • 55,939
  • 5
  • 90
  • 176
0

The CLT is extremely useful when making inferences about things like the population mean because we get there by computing some sort of linear combination of a bunch of individual measurements. However, when we try to make inferences about individual observations, especially future ones (eg, prediction intervals), deviations from normality are much more important if we are interested in the tails of the distribution. For example, if we have 50 observations, we're making a very big extrapolation (and leap of faith) when we say something about the probability of a future observation being at least 3 standard deviations from the mean.

Emil Friedman
  • 711
  • 4
  • 6