2

I am trying to grapple with the following problem.

I have an application that develops empirical distributions. In essence, I end up with a histogram of equally spaced $x$ values, with both a $max$ and a $min$, and the probability for each bucket. What I need to do is 'compress' it for storage (there may be 1,000 to 10,000 buckets) so that the distribution can be recreated with reasonable accuracy.

The obvious choices are:

  1. find a Taylor series approximation

  2. find a Fourier transform approximation

  3. find a moment-generating approximation

Obviously with #1 one only needs convergences between the $max$ and $min$. With #2, one would model the distribution as periodic, with one period occurring between $min(x)$ and $max(x)$. In both cases it is obvious how to recover the distribution based on the very definition of the Taylor series and Fourier transform.

My question is this: if one went the route of #3, what is the 'inversion' formula that recovers a generalized functional representation of a distribution based on the first n integral moments being known?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
eSurfsnake
  • 1,004
  • 5
  • 13
  • 2
    I'm not sure why you have to compress it; with equally-spaced $x$ values, all you need to store is the overall $\min$, $\delta$, and count of number of values ($n$) - two floats and an int. Then you need to store $n$ integers, maybe even short integers, so your total storage space is about $4n$ bytes - only 40 KB with $n=10000$, give or take a little. This doesn't seem particularly onerous. – jbowman Dec 24 '17 at 02:12
  • 1
    Consider other options. For instance, you could fit a spline to the *actual* frequencies (not the relative frequencies) using a Poisson GLM and then normalize it. That would likely require just a handful of coefficients and would give an easily computed formula for the log density. – whuber Dec 24 '17 at 17:19
  • See also https://stats.stackexchange.com/questions/141652/constructing-a-continuous-distribution-to-match-m-moments – kjetil b halvorsen Jan 08 '20 at 13:55

0 Answers0