What is the number of parameters needed for a joint probability distribution?

Question

Let's suppose we have $4$ discrete random variables, say $X_1, X_2, X_3, X_4$, with $3,2,2$ and $3$ states, respectively.

Then the joint probability distribution would require $3 \cdot2 \cdot2 \cdot 3-1 $ parameters (we don't know any independence relations). Considering the Chain Rule, and considering the fact that you need one parameter, $p$, for the marginal distribution of each node with two states, and $2$ for the ones with $3$ states, we have

$$P(X_4,X_3,X_2,X_1)=P(X_4 | X_3,X_2,X_1)P(X_3 | X_2,X_1)P(X_2 | X_1)P(X_1)$$

so we need $3 \cdot 2 \cdot 2 \cdot 2$ parameters for the first conditional probability distribution (as there are $2 \cdot 2 \cdot 3$ combinations of the first three variables and we need the $2$ parameters of $X_4$ for each one), $3 \cdot 2$ for the second one, $3$ for the third one and $2$ for the last one.

So... do we need $3 \cdot 2 \cdot 2 \cdot 2 +3 \cdot 2 + 3 +2 $ parameters?

It is actually true that $3 \cdot 2 \cdot 2 \cdot 2 +3 \cdot 2 + 3 +2 =3 \cdot 2 \cdot 2 \cdot 3$ ?

Answer 1

It takes $3\times 2 \times 2 \times 3 = 36$ numbers to write down a probability distribution on all possible values of these variables. They are redundant, because they must sum to $1$. Therefore the number of (functionally independent) parameters is $35$.

If you need more convincing (that was a rather hand-waving argument), read on.

---

By definition, a sequence of such random variables is a measurable function

$$\mathbf{X}=(X_1,X_2,X_3,X_4):\Omega\to\mathbb{R}^4$$

defined on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$. By limiting the range of $X_1$ to a set of three elements ("states"), etc., you guarantee the range of $\mathbf{X}$ itself is limited to $3\times 2\times 2 \times 3=36$ possible values. Any probability distribution for $\mathbf{X}$ can be written as a set of $36$ probabilities, one for each one of those values. The axioms of probability impose $36+1$ constraints on those probabilities: they must be nonnegative ($36$ inequality constraints) and sum to unity (one equality constraint).

Conversely, any set of $36$ numbers satisfying all $37$ constraints gives a possible probability measure on $\Omega$. It should be obvious how this works, but to be explicit, let's introduce some notation:

• Let the possible values of $X_i$ be $a_i^{(1)}, a_i^{(2)}, \ldots, a_i^{(k_i)}$ where $X_i$ has $k_i$ possible values.

• Let the nonnegative numbers, summing to $1$, associated with $\mathbf{a}=(a_1^{(i_1)}, a_2^{(i_2)}, a_3^{(i_3)}, a_4^{(i_4)})$ be written $p_{i_1i_2i_3i_4}$.

• For any vector of possible values $\mathbf{a}$ for $\mathbf{X}$, we know (because random variables are measureable) that $$\mathbf{X}^{-1}(\mathbf{a}) = \{\omega\in\Omega\mid \mathbf{X}(\omega)=\mathbf{a}\}$$ is a measurable set (in $\mathcal{F}$). Define $$\mathbb{P}\left(\mathbf{X}^{-1}(\mathbf{a})\right) = p_{i_1i_2i_3i_4}.$$

It is trivial to check that $\mathbb{P}$ is an $\mathcal{F}$-measurable probability measure on $\Omega$.

The set of all such $p_{i_1i_2i_3i_4}$, with $36$ subscripts, nonnegative values, and summing to unity, form the unit simplex in $\mathbb{R}^{36}$.

We have thereby a established a natural one-to-one correspondence between the points of this simplex and the set of all possible probability distributions of all such $\mathbf{X}$ (regardless of what $\Omega$ or $\mathcal{F}$ might happen to be). The unit simplex in this case is a $36-1=35$-dimensional submanifold-with-corners: any continuous (or differentiable, or algebraic) coordinate system for this set requires $35$ numbers.

---

This construction is closely related to a basic tool used by Efron, Tibshirani, and others for studying the Bootstrap as well as to the influence function used to study M-estimators. It is called the "sampling representation."

To see the connection, suppose you have a batch of $36$ data points $y_1, y_2, \ldots, y_{36}$. A bootstrap sample consists of $36$ independent realizations from the random variable $\mathbf X$ that has a $p_1=1/36$ chance of equaling $y_1$, a $p_2=1/36$ chance of equaling $y_2$, and so on: it is the empirical distribution.

To understand the properties of the Bootstrap and other resampling statistics, Efron et al consider modifying this to some other distribution where the $p_i$ are no longer necessarily equal to one another. For instance, by changing $p_k$ to $1/36 + \epsilon$ and changing all the other $p_j$ ($j\ne k$) by $-\epsilon/35$ you obtain (for sufficiently small $\epsilon$) a distribution that represents overweighting the data value $X_k$ (when $\epsilon$ is positive) or underweighting it (when $\epsilon$ is negative) or even deleting it altogether (when $\epsilon=-1/36$), which leads to the "Jackknife".

As such, this representation of all the weighted resampling possibilities by means of a vector $\mathbf{p} = (p_1,p_2, \ldots, p_{36})$ allows us to visualize and reason about different resampling schemes as points on the unit simplex. The influence function of the value $X_k$ for any (differentiable) functional statistic $t$, for instance, is simply proportional to the partial derivative of $t(X)$ with respect to $p_k$.

Reference

Efron and Tibshirani (1993), An Introduction to The Bootstrap (Chapters 20 and 21).

Answer 2

The number of parameters needed to represent a random variable is only defined with reference to a model, that is, a family of cumulative distribution functions equipped with a set of parameters that can be used to index them. For example, a normally distributed random variable with mean 3 and standard deviation 1 could be represented with a 0-parameter model (where the only legal distribution is $N(3, 1)$), a 2-parameter model (e.g., $N(μ, σ)$ where $μ$ and $σ$ are parameters), or a 4-parameter model (e.g., $N(μ_1, σ_1) + N(μ_2, σ_2)$).

Answer 3

I would get concrete here. Suppose one has this table, credit

| t    | w    | p(t,w) |
|------|------|--------|
| hot  | sun  | 0.4    |
| hot  | rain | 0.1    |
| cold | sun  | 0.2    |
| cold | rain | 0.3    |

To calculate all $p(t,w)$, do we need four params? Yes, or we can get away with three params and let the last param be 1 minus the sum of the other three params.

Now, what if $t$ and $w$ are independent? To generate the whole table, we only need $2-1=1$ param for $t$, let's say $p(t=hot)$ and $2-1=1$ param for $w$, say $p(w=sun)$.