Is the expectation of the sufficient statistics $S(X)$ transverse the whole space in an exponential family?

Question

An exponential family is defined using two ingredients: - a base density $q_0(x)$ - a number of sufficient statistics $S_i(x)$

The family is all probability densities which can be written as: $$ q(x| (\lambda)_i ) \propto q_0(x) \exp \left( \sum_i \lambda_i S_i(x) \right) $$

It is well known that the relationship between parameters $ (\lambda_i) $ and expected value of the sufficient statistics: $$ E_q( S_i(x) | (\lambda_i) ) = \frac{\int S_i (x) q_0(x) \exp \left( \sum_i \lambda_i S_i(x) \right) dx}{ \int q_0(x) \exp \left( \sum_i \lambda_i S_i(x) \right) dx} $$ is a bijection.

My question is whether this bijection furthermore reaches "all possible values" for $E_q( S_i(x) | (\lambda_i) )$. In my original question, I gave a very poor definition of this ensemble of "all possible values" which made it so that the answer to my question was, somewhat trivially, no.

To define, "all possible values", we have to consider the image of the vector-valued function:

$$ x \rightarrow \mathbf{S}(x) $$

A value in $\mathbb{R}^d$ can be reached by the expected value of $\mathbf{S}$ under a probability density $p(x)$ if and only if it is inside the convex hull of the image of $\mathbf{S}$.

The question is then: when does the expected value of $\mathbf{S}$ inside the exponential family also have this property of spanning the whole convex hull of the image of $\mathbf{S}$ ?

Here are two examples:

The gaussian family in n-dimensions: the sufficient statistics are all first and second moments. It is indeed the case that all first and second moments can be reached by a Gaussian.

The exponential family: $$ q(x|\lambda) = exp( - |x| + \lambda x^2 ) $$

does not reach all values for the second moment: values over the upper bound at $\lambda=0$ are not reached.

This second example makes me think that problems will occur in the tails, if they occur at all, but maybe that intuition is wrong.

Answer 1

Short answer to OP

Not necessarily, it depends on whether the extended convex cone spanned by varying the $\lambda_i$ outside the range of your parameter space can cover the whole mean parameter space. One situation is that your exponential model is over-parameterized, where you cannot "fill" the whole space by varying $\lambda$'s; the other situation is that your exponential family is curved, which also fails such an attempt. See also my answer HERE

I do not really want to turn this answer into an exposition of mean parameter space(Maybe [Brown], but far from complete), but it seems unavoidable now according to successive discussions. I do not really know if there is any treatises devoted to mean parameter space, but mean parameter space gives a natural representation of the family of probability measures and thus convenient in convex analysis. In most statistical applications, differentiability is way too strong for reality.

$\bullet$ Differentiability w.r.t. parameters is usually only assumed to guarante some kind of consistency when other assumptions are either trivial or too complicated to formalize. E.g. Bootstrap methods.

$\bullet$ Continuity w.r.t. parameters is quite a mild assumption, and we sometimes want to assume it even if the collected data seems to be discrete. Continuity will allow sort of local inference but some optimization technique lost their power. E.g. Rapidest gradient method.

$\bullet$ Convexity w.r.t. parameters is the weakest assumption among three, but even so we do not always want to assume that. E.g. Concave loss function.

The mean parameter space is introduced to give the "convex family" a useful representation. Later in the study of positivity [Karlin], the duality between mean parameter space and the family of p.m.s is found to be very useful. There are other motivations like Frenchel duality from convex analysis that explains why we study the mean parameter space. But you can see that some sort of subgradients is usually introduced to model the convex space after Hilbert space.

Now we explain why mean parameter space is important. Define the dual cone for a convex cone $C\subset\mathbb{R}^{n+1}$ $C^+:=\{v\in\mathbb{R}^{n+1}:<v,u>\geq 0,\forall u\in C\}$ For a special case, Karlin-Shapley representation theorem[Karlin&Shapley] told us that the extreme rays/points of the dual cone of the moment space associated with the family lies exactly at the boundary of the parameter space.

For what kind of values can the sufficient family attain, if the sufficient statistics $S(X)$ is also complete and have the same dimension as the parameter space, which mean the structure of the exponential family is linear, then I believe as long as the parameter space does not have a degenerate boundary, the expectation $ES(X)$ of sufficient statistics can transverse all values. From perspective of geometry, only when the dual cone degenerates, you can transverse the whole space, reaching all "possible values" $ES(X)$ possibly have.

When it is not complete or over-parameterized or curved, I am not sure.

In your second example, the dual cone of moment space is actually a strict cone while in the first example a degenerated cone (the whole $\mathbb{R}\times\mathbb{R}$, just imagine that the diameter of a cone tends to $\infty$). But I am still not clear how your reach the result you claimed.

Reference

[Brown]Brown, Lawrence D. "Fundamentals of statistical exponential families with applications in statistical decision theory." Lecture Notes-monograph series 9 (1986): i-279.

[Karlin]Karlin, Samuel. Total positivity. Vol. 1. Stanford University Press, 1968.

[Karlin&Shapley]Karlin, Samuel, and Lloyd S. Shapley. Geometry of moment spaces. No. 12. American Mathematical Soc., 1953.