18

I know that for continuous variable $P[X=x]=0$.

But i can't visualize that if $P[X=x]=0$, there is infinite number of possible $x$'s. And also why do their probabilities get infinitely small ?

Tim
  • 108,699
  • 20
  • 212
  • 390
time
  • 1,167
  • 5
  • 15
  • 31
  • possible duplicate of [Conditional probability of continuous variable](http://stats.stackexchange.com/questions/136523/conditional-probability-of-continuous-variable) – Xi'an Mar 21 '15 at 15:37
  • 2
    There are already two votes to close this question as duplicate. I don't agree. This is a pretty basic topic, one of those that will probably re-appear in the future, so it would be good if it had a direct and high quality answer, so we could refer to it in the future. The link provided by @Xi'an may be threated as duplicate but is also quite specific and hard to find via search. The link also does not provide an exhaustive answer, while this threat seems to converge to such. I think it should be left open as a future reference. – Tim Mar 21 '15 at 16:51
  • It might help to consider the inverse of this situation. Let $X$ be *any* random variable and let $\epsilon$ be any positive real number. There can be only a finite number of $\omega$ for which $\Pr(X=\omega)\ge\epsilon$, for otherwise--by adding up all these probabilities over disjoint events--you would conclude that the total probability is at least $\epsilon+\epsilon+\cdots$, which eventually exceeds $1$. (This is the *Archimedean property* of real numbers.) This reasoning uses only three *axioms*: probabilities of disjoint events add, total probability is $1$, and the Archimedean axiom. – whuber Mar 21 '15 at 17:07
  • @whuber why not posting it as an answer? IMHO it is a very good answer. – Tim Mar 21 '15 at 17:30
  • 1
    @Tim Thank you, but I posted this thought as a comment, rather than an answer, because it's incomplete: I haven't figured out an elementary way to explain what happens in the limit as $\epsilon\to 0$. It seems to require some knowledge of cardinalities of infinite sets. – whuber Mar 21 '15 at 17:32
  • This question appears again and again on X validated, with the same arguments repeated. No point in more duplications. – Xi'an Mar 21 '15 at 17:57
  • 3
    @Xi'an I agree, but the thread you proposed is not a sufficiently close duplicate. This is a difficult thing to search for. Are you perhaps aware of other threads that duplicate this question? – whuber Mar 21 '15 at 18:23

4 Answers4

16

Probabilities are models for the relative frequencies of observations. If an event $A$ is observed to have occurred $N_A$ times on $N$ trials, then its relative frequency is $$\text{relative frequency of }(A) = \frac{N_A}{N}$$ and it is generally believed that the numerical value of the above ratio is a close approximation to $P(A)$ when $N$ is "large" where what is meant by "large" is best left to the imagination (and credulity) of the reader.

Now, it has been observed that if our model of $X$ is that of a continuous random variable, then the samples of $X$ $\{x_1, x_2, \ldots, x_N\}$ are $N$ distinct numbers. Thus, the relative frequency of a specific number $x$ (or, more pedantically, the event $\{X = x\}$) is either $\frac 1N$ if one of the $x_i$ has value $x$, or $\frac 0N$ if all the $x_i$ are different from $x$. If a more skeptical reader collects an additional $N$ samples, the relative frequency of the event $\{X=x\}$ is either $\frac{1}{2N}$ or continues to enjoy the value $\frac 0N$. Thus, one is tempted to guess that $P\{X = x\}$ should be assigned the value $0$ since that is a good approximation to the observed relative frequency.

Note: the above explanation is (usually) satisfactory to engineers and others interested in the application of probability and statistics (i.e. those who believe that the axioms of probability were chosen so as to make the theory a good model of reality), but totally unsatisfactory to many others. It is also possible to approach your question from a purely mathematical or statistical perspective and prove that $P\{X = x\}$ must have value $0$ whenever $X$ is a continuous random variable via logical deductions from the axioms of probability, and without any reference to relative frequency or physical observations etc.

Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200
  • 1
    +1 "Note: the above explanation is... satisfactory to... those who believe that the axioms of probability were chosen so as to make the theory a good model of reality), but totally unsatisfactory...", in the internet's preferred phrasing, lol. – gung - Reinstate Monica Mar 21 '15 at 16:30
  • 2
    I don't understand what do you mean by *it has been observed that if $X$ is continuous, then ...*. How can we observe that ? – Stéphane Laurent Mar 21 '15 at 17:11
  • 3
    @StéphaneLaurent That sentence is a little complicated, so it bears re-reading. Stripped of some parenthetical remarks, it says "it has been observed that ... the samples ... are $N$ distinct numbers." In other words, *when one assumes that $X$ has a continuous distribution*, then (almost surely) there will be no duplicates in any finite iid sample of $X$. That can be mathematically proven: it's not a mere observation. – whuber Mar 21 '15 at 21:12
  • @whuber Ok but then one cannot justify $\Pr(X=x)=0$ by this fact because one needs $\Pr(X=x)=0$ in order to prove it. – Stéphane Laurent Mar 21 '15 at 21:18
  • 2
    @StéphaneLaurent I think Dilip's remarks are being made in a different spirit than that. This answer is not an effort to provide a mathematically rigorous demonstration, but to provide some intuition and motivation for a fact that puzzles the OP. I am intrigued by this approach because it has such potential to bridge the gap between the discrete probability theory traditionally taught to beginners and the richer general theory of probability based on measure theory. – whuber Mar 21 '15 at 21:25
  • 2
    @whuber I understand the spirit, but at first glance I was not convinced that the no-ties property is more intuitive than the zero-probability property. For $N=2$ this is really the same thing: "$x_2 \text{ is never } x_1$" $\iff$ $\Pr(X_2=x_1)=0$. – Stéphane Laurent Mar 21 '15 at 21:53
15

Let $(\Omega,\mathscr{F},P)$ be the underlying probability space. We say that a measurable function $X:\Omega\to\mathbb{R}$ is an absolutely continuous random variable if the probability measure $\mu_X$ over $(\mathbb{R},\mathscr{B})$ defined by $\mu_X(B)=P\{X\in B\}$, known as the distribution of $X$, is dominated by Lebesgue measure $\lambda$, in the sense that for every Borel set $B$, if $\lambda(B)=0$, then $\mu_X(B)=0$. In this case, the Radon-Nikodym theorem tells us that there is a measurable $f_X:\mathbb{R}\to\mathbb{R}$, defined up to almost everywhere equivalence, such that $\mu_X(B)=\int_B f(x)\,d\lambda(x)$. Let $B=\{x_1,x_2,\dots\}$ be a countable subset of $\mathbb{R}$. Since $\lambda$ is countably additive, $\lambda(B)=\lambda\left(\cup_{i\geq 1}\{x_i\}\right)=\sum_{i\geq 1}\lambda(\{x_i\})$. But $$ \lambda(\{x_i\}) = \lambda\left(\cap_{k\geq 1}[x_i,x_i+1/k)\right) \leq \lambda\left([x_i,x_i+1/n)\right) = \frac{1}{n} \, ,\qquad (*) $$ for every $n\geq 1$. Due to the Archimedean property of the real numbers, since $\lambda(\{x_i\})\geq 0$, the inequality $(*)$ holds for every $n\geq 1$ if and only if $\lambda(\{x_i\})=0$, entailing that $\lambda(B)=0$. From the assumed absolute continuity of $X$ it follows that $\mu_X(B)=P\{X\in B\}=0$.

Zen
  • 21,786
  • 3
  • 72
  • 114
  • Continuous random variable doesn't need to be absolutely continuous (it could have no density.) – Zhanxiong Jul 24 '15 at 13:49
  • 2
    Baloney. "Continuous random variable" is an informal name for "a random variable which is absolutely continuous with respect to Lebesgue measure". Hence, Radon-Nikodym guarantees that a density exists. A random variable with a singular distribution (e.g. Cantor) is a different thing. You're misleading potential students with your bogus comment. – Zen Jul 24 '15 at 14:04
  • When you criticized someone, please show the citation you referred to. Which probability text book said that *"Continuous random variable" is an informal name for "a random variable which is absolutely continuous with respect to Lebesgue measure"*? In addition, this problem can be solved without requiring $X$ has a density, see my proof below. – Zhanxiong Jul 24 '15 at 14:13
  • Wikipedia disagrees with you, @Solitary: "[A continuous probability distribution](https://en.wikipedia.org/wiki/Probability_distribution#Continuous_probability_distribution) is a probability distribution that has a probability density function. Mathematicians also call such a distribution absolutely continuous [...]". – amoeba Jan 18 '16 at 11:41
5

$X$ is a continuous random variable means its distribution function $F$ is continuous. This is the only condition we have but from which we can derive that $P(X = x) = 0$.

In fact, by continuity of $F$, we have $F(x) = F(x-)$ for every $x \in \mathbb{R}^1$, therefore: $$P(X = x) = P(X \leq x) - P(X < x) = F(x) - F(x-) = 0.$$

Zhanxiong
  • 5,052
  • 21
  • 24
  • 1
    If the distribution of a r.v. $X$ is Cantor, then its distribution function is continuous, but $X$ is a singular random variable; it's not a continuous random variable. – Zen Jul 24 '15 at 14:22
  • My friend, this actually can be a counterexample to your own answer, not mine. Since the existence of such ***Singular continuous*** r.v., it is necessary to distinguish ***absolute continuous*** r.v. and ***singular continuous*** r.v., although their distribution functions are all continuous. To equalize ***continuous*** r.v. and ***absolute continuous*** r.v. is ambiguous. – Zhanxiong Jul 24 '15 at 14:54
  • 1
    It isn't, but you won't hear, my friend. – Zen Jul 24 '15 at 21:03
  • 1
    By the way, you're actually "proving" that if $P(X=x)=0$ for every $x$, then $P(X=x)=0$ for every $x$. – Zen Jul 24 '15 at 21:06
0

This question is very simple. PDF is the density, then to get the probability you need to multiply it by the width of the region. So, when you get a smaller region around the point of interest $x$ the height of the density doesn't change, hence the probability is smaller and smaller while you squeeze the area around your point until it becomes exactly zero.

The difficult question is the reverse one, like this: https://stats.stackexchange.com/a/273407/36041

Aksakal
  • 55,939
  • 5
  • 90
  • 176