Better understanding independence of events

Question

I undersrand that a probability space is composed of a sample space $\Omega$ containing all possible outcomes of an experiment, an event space $\mathcal F$ containing all events of interest (e.g. the power set of $\Omega$) and a probability measure $\mathbb P$ that assigns a probability $p$ to each event in $\mathcal F$.

• We say two events $A$ and $B$ in $\mathcal F$ are disjoint iff $A\cap B = \emptyset$

• On the other hand, two events are said to be independent iff $\mathbb P(A\cap B) = \mathbb P(A) \mathbb P(B)$. Meaning that additional knowledge of one event does not change the probability of occurrence of the other event. But I can't seem to fully wrap my mind around this. What does independent mean here? How can two events have a nonzero intersection and yet have nothing to do with one another? Some people say it means the state space can be factorized into a cartesian product, which I don't really get.

• For example: given $\Omega =\{1,2,3,4\}$ how can one intuitively realise that the events $A=\{1,2\}$, $B=\{1,3\}$ and $C=\{1,4\}$ are only pairwise independent and not independent?

Answer 1

It can be helpful to think of the definition of conditional probability when we try to understand the definition of independence. Recall that $$ P(A \mid B) = \frac{P(A \cap B)}{P(B)} $$ for $B$ with $P(B) > 0$. If $A$ and $B$ are independent, which means $B$ tells us nothing about $A$, then the right hand side should reduce to $P(A)$. If we plug in $P(A \cap B) = P(A) P(B)$ then $P(B)$ simply cancels and the right side does indeed become $P(A)$. But the nice thing about defining independence in this way is that we can talk about independence of events even when conditional probabilities are undefined (which happens whenever the conditioning event has probability zero).

Another way to think about is to view probabilities as fractions of a sample space. If $B$ happens then we're left with $P(B)$ of the sample space remaining. Now if $A$ also happens then $B$ just becomes the new sample space and we reduce it by a factor of $P(A)$ just as we would if we were reducing the original sample space. So in a certain sense $A$ is "blind" to the occurrence of the event $B$, which is more or less what independence means.

Disjointness on the either hand is a very different concept. It says that $A$ and $B$ cannot simultaneously occur, so they are very much dependent (this is strictly speaking not true, can you find a counterexample?).

Regarding your last question, if we look at the sample space $\Omega = \{1, 2, 3, 4 \}$ and the events $A = \{1, 2 \}$, $B = \{1, 3 \}$ and $C = \{1, 4 \}$ we can start by supposing the event $A \cap B = \{ 1 \}$ has occurred. Then $C$ has no choice but to occur as well, so it isn't as though when $C$ occurs here we begin with a sample space and then reduce it to a proper subset whose relative size is $P(C)$, as would be the case if $C$ was independent of $A \cap B$. The new reduced sample space is now a subset of $C$, and $C$ always occurs. The fact that these events happen to be pairwise independent is mostly a contrived property of the problem, and when we talk about independence we almost always mean mutual independence, where no event has any knowledge of occurrence of the others.

Answer 2

Meaning that additional knowledge of one event does not change the probability of occurrence of the other event. But I can't seem to fully wrap my mind around this. What does independent mean here? How can two events have a nonzero intersection and yet have nothing to do with one another?

"two events have nothing to do with one another" means that both could occur simultaneously, or one could occur but not the other, or both might not occur. If two events have zero intersection, then they very much do have something to do with each other. If we know (or we are told or we assume) that one event has occurred, then we know that the other event cannot possibly have occurred. (Remember that an event (a collection of outcomes) is said to have occurred on a trial if the outcome of the trial is a member of that event). If $A\cap B = \emptyset$, then whenever the outcome $x \in A$, we know for sure that $x \notin B$. In short, the occurrence of $A$ tells us for sure that $B$ did not occur, but instead $B^c$ has occurred. Thus, $P(B\mid A) = 0$, $P(B^c\mid A) = 1$. Unless $P(B)$ happens to have value $0$, it is clear that knowledge of the occurrence of $A$ has indeed "changed the probability of the occurrence of $B$."

Thus, at the very least, I hope that you can disabuse yourself of the notion that "nothing to do with one another" means disjoint. But, just having $P(A\cap B)$, $P(A\cap B^c)$, $P(A^c \cap B)$, and $P(A^c \cap B^c)$ all be nonzero is not sufficient to assert independence. As pointed out by @dsaxton, what is needed to assert independence is $$P(B\mid A) = P(B),$$ that is, "knowledge of one event does not change the probability of occurrence of the other event": the conditional probability that $B$ occurs conditioned on the occurrence of $A$ is the same as the unconditional probability.

For a more extensive discussion of these matters as well as an answer to the query posed in the penultimate paragraph of dsaxton's answer, I refer you to this previous answer of mine from four years ago.