19

In Bayes' formula:

$$P(x|a) = \frac{P(a|x) P(x)}{P(a)}$$

can the posterior probability $P(x|a)$ exceed 1?

I think it is possible if for example, assuming that $0 < P(a) < 1$, and $P(a) < P(x) < 1$, and $P(a)/P(x) < P(a|x) < 1$. But I'm not sure about this, because what would it mean for a probability to be greater than one?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Thomas Moore
  • 1,375
  • 10
  • 17
  • 6
    One should be precise in defining notation. It is unclear what $P(\cdot)$ represents. If $P(\cdot)$ is (a) a probability distribution (in which case $a$ and $x$ are sets) or (b) a mass function on a discrete space, then the answers you already have are essentially correct. If $P(\cdot)$ is understood to be a density function, then it is not true that $P(x \mid a) \le 1$. The reason for nitpicking is that all three types of functions satisfy Bayes rule. The notation $P(\cdot)$ usually is for a distribution, but using lower case characters for arguments suggests a density. – guy May 01 '18 at 02:28
  • 2
    $P(x\mid a) = \frac{P(x,a) }{P(a)} \le \frac{P(a) }{P(a)} =1$ so the posterior probability cannot exceed $1$. (The posterior density is a different matter - plenty of continuous distributions have densities exceeding $1$ for some values) – Henry May 01 '18 at 10:17
  • If the calculated posterior exceeds one, you made a mistake somewhere. – Emil M Friedman May 01 '18 at 20:30
  • 2
    @EmilMFriedman, your answer is ambiguous (and, for that reason, potentially harmful), because it doesn't indicate whether it refers to a "calculated posterior" *probability* or *density.* – whuber May 01 '18 at 22:21
  • The unity barrier in probability can and has been broken. See my post AT https://stats.stackexchange.com/questions/4220/can-a-probability-distribution-value-exceeding-1-be-ok/160979#160979 . – Mark L. Stone May 02 '18 at 02:13
  • @guy this is what I was also thinking is confusing the OP. To expand on your point about discrete vs continuous for a $P(x | a)$, this probability would be calculated for $x \in (a,b)$ where $a \neq b$. A single value of the density could be greater than 1. – Lucas Roberts May 22 '21 at 23:55

3 Answers3

29

No, it is not possible for the posterior probability to exceed one. That would be a breach of the norming axiom of probability theory. In your question you specify that $\mathbb{P}(a)/\mathbb{P}(x) < \mathbb{P}(a | x)$ as part of your example. However, using the rules of conditional probability, you must have:

$$\mathbb{P}(a | x) = \frac{\mathbb{P}(a,x)}{\mathbb{P}(x)} \leqslant \frac{\mathbb{P}(a)}{\mathbb{P}(x)}.$$

This means that you cannot have the inequality conditions you have specified. (Incidentally, this is a good question: it is good that you are probing the probability laws looking for problems. It shows that you are exploring these matters with a greater degree of rigour than most students.)

An additional point: It is worth making one additional point about this situation, which is about the logical priority of different characteristics of probability. Remember that probability theory starts with a set of axioms that characterise what a probability measure actually is. From these axioms we can derive "rules of probability" which are theorems derived from the axioms. These rules of probability must be consistent with the axioms to be valid. If you ever found that a rule of probability leads to a contradiction with one of the axioms (e.g., the probability of the sample space is greater than one), this would not falsify the axiom - it would falsify the probability rule. Hence, even if it were the case the Bayes' rule could lead to a posterior probability greater than one (it doesn't), this wouldn't mean that you can have a posterior probability greater than one; it would simply mean that Bayes' rule is not a valid rule of probability.

Ben
  • 91,027
  • 3
  • 150
  • 376
10

The Bayes formula $\displaystyle P(B \mid A) = \frac{P(A\mid B)P(B)}{P(A)}$ cannot give values for $P(B\mid A)$ exceeding $1$. An intuitive way to see this is to express $P(A)$ via the law of total probability as $$P(A) = P(A\mid B)P(B) + P(A\mid B^c)P(B^c)$$ giving that $$P(B \mid A) = \frac{P(A\mid B)P(B)}{P(A)} = \frac{P(A\mid B)P(B)}{P(A\mid B)P(B) + P(A\mid B^c)P(B^c)}$$ which shows that the numerator is just one of the terms in the sum in the denominator, and so the fraction cannot exceed $1$ in value.

Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200
  • +1 this is the easiest proof to me. – user541686 May 02 '18 at 09:07
  • @Mehrdad Thanks. The other answers essentially prove that a conditional probability $P(B\mid A)$ cannot exceed $1$ via the result that $P(A\mid B)P(B) = P\A\cap B)$ cannot exceed $P(A)$ because $A\cap B \subset A$ and so it must be that $P\A\cap B) \leq P(A)$, and have little relationship _per se_ to Bayes' formula (as it is used in statistics to derive posterior probabilities from prior probabilities). – Dilip Sarwate May 02 '18 at 19:16
9

The assumed conditions do not hold- it can never be true that $P(a)/P(x) < P(a|x)$ by the definition of conditional probability:

$P(a|x) = P(a\cap x) / P(x) \leq P(a) / P(x)$

khol
  • 777
  • 7
  • 13