2

I recently noticed that if you generate 10000 normally distributed numbers and then find the probability associated with each number (pnorm), each probability from 0 to 1 occurs with approximately the same frequency. Here's how I did it in R:

var2 <- numeric(10000)
normnos <- rnorm(10000)
for (i in 1:10000) {
  var2[i] <- pnorm(normnos[i])
}
hist(var2)

enter image description here

How is this possible? If all probabilities have equal likelihood of occurring, then wouldn't the resulting distribution be uniform instead of normal? I'm really confused and would appreciate an explanation.

Ryan Ma
  • 33
  • 3
  • 1
    The hits at https://stats.stackexchange.com/search?tab=votes&q=%22probability%20integral%22%20transform answer your question. – whuber Jan 03 '21 at 16:09

1 Answers1

5
  1. pnorm doesn't compute the probability of the sampled number - it rather computes $P(X \leq x)$ - which is the cumulative distribution function. To compute the probability of sampled number, you will have to use the PDF - normal distribution in this case, that is, $p(x_i - \delta < X < x_i + \delta) = N(x_i | \mu = 0, \sigma = 1)$ ($\delta$ very small).
  2. The histogram you plotted is the distribution of cdf values, which is always uniform regardless of the distribution. This is known as "Universality of the Uniform"
  3. Mathematically, suppose $X$ is a random variable with pdf $p_X(x)$ and cdf $F_X(x) = P(X \leq x)$. Let $T$ be the random variable $T = F_X(X)$ - the samples of which you plotted in the histogram. $T$ is random because $X$ (normal variable in your case) is random. Then, $$F_T(t) = P(T \leq t) = P(F_X(X) \leq t) = P(X \leq F_X^{-1}(t)) = F_X(F_X^{-1}(t)) = t$$
  4. $F_T(t) = t$ -- this is the cdf of a uniform distribution. So, the pdf of T is uniform - which is what you plotted. Note that the inverse of $F_{X}(x)$ exists only if $F_X$ is continuous and strictly increasing.

Hope this helps! :)

stbv
  • 136
  • 3
  • I see, it makes sense now. Thanks so much! :) – Ryan Ma Jan 03 '21 at 08:04
  • 1
    Correction: The pdf at $x$, $\varphi(x)$, is _not_ the probability of the event $X=x$, which is of probability zero. – Xi'an Jan 03 '21 at 11:25
  • 1
    Correction #2: the cdf transform of a sample is only uniform when the cdf is continuous and strictly increasing. – Xi'an Jan 03 '21 at 11:27
  • @Xi'an thanks for the corrections. the inverse of cdf can be taken only if it is continuous and strictly increasing. And for a continuous pdf, the probability of any single point is zero. Edited the answer. – stbv Jan 03 '21 at 11:47
  • Not exactly: when using a generalised inverse, $F_X^{-}($ the result that $F_X^{-}(U)$ is distributed as $F_X$ is a universal result, as opposed to $F_X(X)$ being uniform only for an invertible $F_X$. – Xi'an Jan 03 '21 at 13:04