Estimating maximum entropy distribution given first n moments

Question

Is there a good way to estimate the pdf, pdf up to a constant multiple, cdf, or quantile function of a distribution given the first n moments? A closed form for one of those functions in terms of the moments would be ideal, but I suspect that one does not exist in general.

The practical motivation here is that I'm trying to write a library for estimating various properties of a time series after the fact. Collections of moments have nice algebraic properties (because expectation is linear) which makes parallel processing easier.

---

A normal distribution is the maximum entropy distribution among distributions constrained to have mean $\mu$ and variance $\sigma^2$ .

According to this question and the Wikipedia article on variance, the mean (102) and variance (103) can be defined in terms of raw moments ($\mu_n$ where $n$ is the power).

$$ \mu_n \stackrel{\small{\text{def}}}{=} \mathop{\mathbb{E}} \left[ x^n \right] \tag{101} $$ $$ \mu \stackrel{\small{\text{def}}}{=} \mu_1 \tag{102} $$ $$ \sigma^2 \stackrel{\small{\text{def}}}{=} \mu_2 - \mu_1^2 \tag{103} $$

The normal distribution is naturally parameterized by $\mu$ and $\sigma^2$, but can also be thought of as being parameterized by $\mu$ and $\mu_2$ .

The answer by Glen_b references a theorem by Ludwig Boltzmann which suggests to me that the pdf in a situation where the first $n$ moments are known will be proportional to (104).

$$ \exp \circ \left( \text{some $n$ degree polynomial with no constant term} \right) \tag{104} $$

Assuming I haven't horribly misinterpreted the answer, is there a computationally efficient way to estimate these coefficients?

Moreover, is there a way to do an online estimate of the coefficients? Ideally I'd like to be able to update the coefficients of the polynomial in (104) at the same time that I update the moments when "merging in" new observations.

Answer 1

Maximum entropy principle helps you find the distribution with the highest entropy (most uncertainty, least biased) among all distributions that match first n sample moments. As you mentioned, the maximum entropy distribution for $n=2$ is a Gaussian distribution. For general case, such distribution is derived using first order optimality conditions of the corresponding optimization problem and is formulated as follows \begin{align} f(x) = \int \exp(-\sum_{j=0}^n \lambda_jx^j) \, dx\\ \text{s.t.} \int x^i \exp(-\sum_{j=0}^k \lambda_jx^j)\, dx = m_i, \forall i\in \{0,\dots,n\} [*] \end{align} where [*] are the moment constraints and $m_i$ are the sample moments with $m_0=1$.

So Lagrange Multipliers $\lambda \in \mathcal{R}^{k+1}$ are unknown parameters of the distribution. One way to find them is to switch to dual formulation and maximize the scaled log-likelihood function based on Gibbs distribution

\begin{align} -\big( \ln \int \exp(-\sum_{j=1}^n \lambda_jx^j) \, dx + \sum_{i=1}^n m_i\lambda_i \big) \rightarrow \text{max}_{\lambda} \end{align}

The problem is concave so you can use gradient-based methods to solve this maximization problem.

You can find more details in the paper of Mead and Papanicolaou "Maximum entropy in the problem of moments".