Is the Intersection Area Between 2 PDFs a Probability?

Question

Say that we have two processes $a$ and $b$ that generate numbers. Then say that the random variables $A$ and $B$ take values in the set of numbers that those processes generate, respectively. So $f_A$ and $f_B$ are their probability density functions, respectively.

Now, suppose that we plot those PDFs $f_A$ and $f_B$, and find that the intersection area between them is $0.6$. In other words (as Aksakal and Tim put it): $$\int_x \text{min}\big(f_A(x), f_B(x)\big)\,\,\text{d}x = 0.6$$

Is $0.6$ a probability? Is it the probability that the processes $a$ and $b$ might generate numbers that are statistically indistinguishable?

Or, do I need to normalize $0.6$ somehow in order to make it into a probability?

To further clarify:

• The question is not:

  "What is the probability that the processes are the same?". In fact, we can assume that the processes are necessarily different.

• The question is rather:

  "Suppose that a randomly sampled number, from either one of the processes, was handed to you, without telling you the process that generated the number. Then, you were asked to predict the process that generated that number. Is this intersection area, $0.6$, the probability that the number that is given to you is statistically uninformative to you with respect to the mother process that generated it?"

Answer 1

No, it is not, it is twice a probability!

Let $P$ and $Q$ be probability measures with corresponding densities $f$, $g$. Then the variation distance between $P$ and $Q$ is $$ \| P-Q \| = \max_A |P(A) -Q(A) | $$ so it measures the absolute difference in probability for an event that maximizes that difference.

Then one can show that (*) $$ \| P-Q \| =\frac12 \int |f(x)-g(x)| \; dx = \frac12 \max_{\| h \| \le 1 } | P(h)-Q(h)| $$ where $h$ is a real-valued function satisfying $|h(x) \le 1$ and $P(h), Q(h)$ denotes the expectation of $h$ (under that measure). Paul Switzer suggested the following interpretation: You are given a single observation, from $P$ or from $Q$, each with probability $\frac12$. You have to guess from which distribution it came. Your probability of being right is then $\frac12 + \frac12 \|P-Q\|$.

Despite requests, the OP did'nt clarify his interpretation in the question. @Aksakal offered another interpretation, $\int \min(f(x),g(x))\; dx$. But the variational norm can be used to give an answer in that case too. First, note that $$ f(x) + g(x) = \min(f(x),g(x)) + \max(f(x),g(x)) $$ and $$ \max(f(x),g(x)) = \min(f(x), g(x)) + \mid f(x)-g(x) \mid $$ Now, integrating this two equations and doing some algebra gives $$ \int \min(f(x),g(x)) \; dx = 1 - \frac12\int\mid f(x)-g(x)\mid \; dx= 1-\|P-Q\| $$ so, indeed, the variational norm gives an interpretation also in this case.

(*) First, the factor $\frac12$ is easy to understand, since $\int |f(x)-g(x)| \; dx$ has an infimum of zero (when $f=g$) and a supremum of 2 (when $f$ and $g$ have disjoint supports. Now a proof: There is a proof in a comment at this post: https://math.stackexchange.com/questions/69166/understanding-the-relationship-of-the-l1-norm-to-the-total-variation-distance

Answer 2

If I understood your question correctly, then this is what you're looking for: $$\int_x\min [f_A(x),f_B(x)] dx$$

Is it the probability that the processes a and b might generate numbers that are statistically indistinguishable?

No. Only if this quantity is equal to 1, you'll get two processes statistically indistinguishable. They'll be like two realizations of the same process.

UPDATE Say you have 2 rows of 10 bins each. You fill the first row with 100 red balls, and the second row with 100 blue balls. Next, you look at the first two bins, and pick either red or blue balls, whichever is the fewest. Then you do the same with a next pair of bins, and so on until all 10 pairs are exhausted. How many balls out of total 200 you collected? The quantity that you're looking at answers this question.

Overall, this type of quantity seems like something you might encounter in game theory or stochastic programming and dynamic decision process

Answer 3

If by "intersection" you mean $f(x) = \min\{f_A(x), f_B(x)\}$, then your function could be paraphrased, "among the possible generators, what is the lowest probability that this event will occur?" In general, this will not be a probability distribution.

This is easier to see in the discrete case. For example, say we have two unfair coins, $A$ and $B$, with chances for landing heads or tails given follows: $$f_A(x) = \begin{cases} 0.9 & x = heads \\ 0.1 & x = tails \\ 0 & otherwise \end{cases} \\ f_B(x)= \begin{cases} 0.1 & x = heads \\ 0.9 & x = tails \\ 0 & otherwise \end{cases}$$ Then your intersection function will give $$\begin{align} f(heads) & = \min\{f_A(heads), f_B(heads) \} \\ & = \min\{0.9, 0.1\} \\ & = 0.1 \end{align} $$ Similarly $f(tails) = 0.1$. Then, since $\sum_{x \in \{heads, tails\}} f(x) = 0.2 < 1$, we clearly don't have a probability distribution. Neither is it the probability that both $A$ and $B$ are "indistinguishable" $$\begin{align} P("indistinguishable") & = P(A = B) \\ &= \sum_{x \in \{heads, tails\}}P[f_A(x) \cap f_B(x)] \\ &=[f_A(heads) \cap f_B(heads)] +[f_A(tails) \cap f_B(tails)] \\ &= [0.9 \times 0.1] + [0.1 \times 0.9] = 0.18\end{align}$$

Intersecting PDFs is different from the intersecting the events $a$ and $b$. Then we talk about joint, marginal, and conditional probability distributions. An intersection of two events in the joint probability distribution for the events. The way to think about this, rather than intersecting two PDFs, you have to increase the dimensionality of your probability space to 3 dimensions: $x$ is values of $A$, $y$ is values of $B$ and $z$ is the joint probability that $P(A = x \cap B = y)$. If you project the PDF onto the $x-z$ plane, you should get back your marginal distribution $f_A$, and if you project it onto the $y-z$ plane, you get your $f_B$ marginal back.

When you talk about "statistically indistinguishable," it makes me also think of hypothesis testing. Frequently, hypothesis testing textbooks will show two distributions side by side, and one tail is highlighted. These graphics can be confusing, but it isn't taking the intersection of the distributions. One distribution is the null hypothesis ($H_0$) and the other is the alternative hypothesis ($H_A$). The pictures then usually highlight the tail of $H_0$, which are the unlikely events. When we observe these events, we say with a certain degree of statistical confidence that $H_0$ is not the generator for those events, and instead we favor $H_A$ as a more likely generator.

Answer 4

As others already noted, this is not a probability. However you can make probability out of it by scaling it appropriately.

Take

$$ f_{A \cap B}(x) = \min\{\, f_A(x),\, f_B(x) \,\} \\ f_{A \cup B}(x) = \max\{\, f_A(x),\, f_B(x) \,\} $$

then

$$ \frac{\int_x \,f_{A \cap B}(x) \;dx}{\int_x \,f_{A \cup B}(x) \;dx} $$

is probability, as it is the $A \cap B$ fraction of total area of $A \cup B$. However it is hard to find a meaningful interpretation of such probability (or possible variations of this problem).

Check the Mutual Information as probability thread for a similar solution of different problem.