24

I read here that given a sample $ X_1,X_2,...,X_n $ from a continuous distribution with cdf $ F_X $, the sample corresponding to $ U_i = F_X(X_i) $ follows a standard uniform distribution.

I have verified this using qualitative simulations in Python, and I was easily able to verify the relationship.

import matplotlib.pyplot as plt
import scipy.stats

xs = scipy.stats.norm.rvs(5, 2, 10000)

fig, axes = plt.subplots(1, 2, figsize=(9, 3))
axes[0].hist(xs, bins=50)
axes[0].set_title("Samples")
axes[1].hist(
    scipy.stats.norm.cdf(xs, 5, 2),
    bins=50
)
axes[1].set_title("CDF(samples)")

Resulting in the following plot:

Plot showing the sample of a normal distribution and the cdf of the sample.

I am unable to grasp why this happens. I assume it has to do with the definition of the CDF and it's relationship to the PDF, but I am missing something...

I would appreciate it if someone could point me to some reading on the subject or help me get some intuition on the subject.

EDIT: The CDF looks like this:

CDF of the sampled distribution

Maxime Tremblay
  • 243
  • 1
  • 3
  • 5
  • 3
    Compute the cdf of $F_X(X)$. – Zhanxiong Jul 15 '15 at 16:50
  • 3
    You would find a proof of this property (for continuous rv's) in any book about simulation as this is the basis of the inverse cdf simulation method. – Xi'an Jul 15 '15 at 17:06
  • 3
    Also try google-ing *probability integral transform* – Zachary Blumenfeld Jul 15 '15 at 17:13
  • 3
    @Xi'an It is good to point out the conclusion holds only for continuous random variables. Sometimes this result is mistakenly used for discrete random variables. On the other hand, also note many proofs involves the step $P(F(X) \leq x) = P(X \leq F^{-1}(x))$ in which assumes the strict monotonicity of $F$, which is also a too strong assumption. The following link provides a rigorous summary on this topic:https://people.math.ethz.ch/~embrecht/ftp/generalized_inverse.pdf – Zhanxiong Jul 15 '15 at 17:20
  • @Zhanxiong the only condition necessary for $F$ is that it is càdlàg. – AdamO Jul 15 '15 at 17:26
  • @AdamO Hmm, I don't think so. For a somewhat extreme example, $X \equiv 1$, then its cdf $F$ is càdlàg, however, $F(X) \equiv 1$, which is not uniform $(0, 1)$. – Zhanxiong Jul 15 '15 at 17:33
  • @Zhanxiong sorry you are right, the somewhat stronger condition of OP's statement is that, random variables following any càdlàg distribution $F$ can be "simulated" from inverse transformed uniform 0,1 RVs. That would be the case with your atomic DF. – AdamO Jul 15 '15 at 17:39
  • @AdamO Yes, you are right. Your statement can actually be viewed as the opposite direction of the statement in this post. – Zhanxiong Jul 15 '15 at 17:49

3 Answers3

28

Assume $F_X$ is continuous and increasing. Define $Z = F_X(X)$ and note that $Z$ takes values in $[0, 1]$. Then $$F_Z(x) = P(F_X(X) \leq x) = P(X \leq F_X^{-1}(x)) = F_X(F_X^{-1}(x)) = x.$$

On the other hand, if $U$ is a uniform random variable that takes values in $[0, 1]$, $$F_U(x) = \int_R f_U(u)\,du =\int_0^x \,du =x.$$

Thus $F_Z(x) = F_U(x)$ for every $x\in[0, 1]$. Since $Z$ and $U$ has the same distribution function $Z$ must also be uniform on $[0, 1]$.

Hunaphu
  • 1,334
  • 8
  • 12
11

Intuitively, perhaps it makes sense to think of $F(x)$ as a percentile function, e.g. $F(x)$ of a randomly generated sample from the DF $F$ is expected to fall below $x$. Alternately $F^{-1}$ (think inverse images, not a proper inverse function per se) is a "quantile" function. That is, $x = F^{-1}(p)$ is the point $x$ behind which falls $p$ proportion of the sample. The functional composition is measurably commutative $F \circ F^{-1} =_\lambda F^{-1} \circ F$.

The uniform distribution is the only distribution having a quantile function equal to a percentile function: they are the identity function. So the image space is the same as the probability space. $F$ maps continuous random variables into a (0, 1) space with equal measure. Since for any two percentiles, $a < b$, we have $P(F^{-1}(a) < x < F^{-1}(b)) = P(a < F(X) < b) = b-a$

AdamO
  • 52,330
  • 5
  • 104
  • 209
  • I struggled for hours, but finally it clicked why the derived random variable $Y = F(X)$ is uniformly distributed. Your answer really helped, thanks a lot. It seems very much like in algebra where 1 was the multiplicative identity. – Aditya P Sep 21 '18 at 12:50
6

Here's some intuition. Let's use a discrete example.

Say after an exam the students' scores are $X = [10, 50, 60, 90]$. But you want the scores to be more even or uniform. $h(X) = [25, 50, 75, 100]$ looks better.

One way to achieve this is to find the percentiles of each student's score. Score $10$ is $25\%$, score $50$ is $50\%$, and so on. Note that the percentile is just the CDF. So the CDF of a sample is "uniform".

When $X$ is a random variable, the percentile of $X$ is "uniform" (e.g. the number $X$'s in $0-25$ percentile should be the same as the number of $X$'s in $25-50$ percentile). Therefore the CDF of $X$ is uniformly distributed.

ezheng
  • 61
  • 1
  • 2