What is the normal approximation of the multinomial distribution?

Question

If there are multiple possible approximations, I'm looking for the most basic one.

Answer 1

You can approximate it with the multivariate normal distribution in the same way that binomial distribution is approximated by univariate normal distribution. Check Elements of Distribution Theory and Multinomial Distribution pages 15-16-17.

Let $P=(p_1,...,p_k)$ be the vector of your probabilities. Then the mean vector of the multivariate normal distribution is $ np=(np_1,np_2,...,np_k)$. The covariance matrix is a $k \times k$ symmetric matrix. The diagonal elements are actually the variance of $X_i$'s; i.e.$ np_i(1-p_i)$, $i=1,2...,k$. The off-diagonal element in the ith row and jth column is $\text{Cov}(X_i,X_j)=-np_ip_j$, where $i$ is not equal to $j$.

Answer 2

The density given in this answer is degenerate, and so I used the following to calculate the density that results from the normal approximation:

There's a theorem that says given a random variable $X = [X_1, \ldots, X_m]^T \sim \text{Multinom}(n, p)$, for an $m$-dimensional vector $p$ with $\sum_i p_i = 1$ and $\sum_i X_i = n$, that;

$$ X \xrightarrow{d} \sqrt{n} \, \text{diag}(u) \, Q \begin{bmatrix} Z_1 \\ \vdots \\ Z_{m-1} \\ 0 \end{bmatrix} + \begin{bmatrix} n p_1 \\ \vdots \\ n p_m \end{bmatrix}, $$

for large $n$, given;

• a vector $u$ with $u_i = \sqrt{p_i}$;
• random variables $Z_i \sim N(0,1)$ for $i = 1, \ldots, m-1$, and;
• an orthogonal matrix $Q$ with final column $u$.

That is to say, with some rearrangement, we can work out an $m-1$ dimensional multivariate normal distribution for the first $m-1$ components of $X$ (which are the only interesting components because $X_m$ is the sum of the others).

A suitable value of the matrix $Q$ is $I - 2 v v^T$ with $v_i = (\delta_{im} - u_i) / \sqrt{2(1 - u_m)}$ - i.e. a particular Householder transformation.

If we restrict the left-hand side to the first $m-1$ rows, and restrict $Q$ to its first $m-1$ rows and $m-1$ columns (denote these $\hat{X}$ and $\hat{Q}$ respectively) then:

$$ \hat{X} \xrightarrow{d} \sqrt{n} \text{diag}(\hat{u}) \hat{Q} \begin{bmatrix} Z_1 \\ \vdots \\ Z_{m-1} \end{bmatrix} + \begin{bmatrix} n p_1 \\ \vdots \\ n p_{m-1} \end{bmatrix} \sim \mathcal{N} \left( \mu, n \Sigma \right), $$

for large $n$, where;

• $\hat{u}$ denotes the first $m-1$ terms of $u$;
• the mean is $\mu = [ n p_1, \ldots, n p_{m-1}]^T$, and;
• the covariance matrix $n \Sigma = n A A^T$ with $A = \text{diag}( \hat{u} ) \hat{Q}$.

The right hand side of that final equation is the non-degenerate density used in calculation.

As expected, when you plug everything in, you get the following covariance matrix:

$$ (n\Sigma)_{ij} = n \sqrt{p_i p_j} (\delta_{ij} - \sqrt{p_i p_j}) $$

for $i,j = 1, \ldots, m-1$, which is exactly the covariance matrix in the original answer restricted to its first $m-1$ rows and $m-1$ columns.

This blog entry was my starting point.