14

I have the sample population of a certain signal's registered amplitude maxima. Population is about 15 million samples. I produced a histogram of the population, but cannot guess the distribution with such a histogram.

EDIT1: File with raw sample values is here: raw data

Can anyone help estimate the distribution with the following histogram: enter image description here

mbaitoff
  • 757
  • 1
  • 8
  • 16
  • 1
    not that it matters dramatically but when using histograms it usually helps to have the relative frequency instead of absolute frequency on the y-axis. – posdef Mar 23 '11 at 10:27
  • that is, to provide 120000/15000000=0.008 instead 120000 on vertical axis? – mbaitoff Mar 23 '11 at 10:53
  • @mbaitoff: Your comments to the answer of schenectady indicate, that you are less interested in getting the name of the distribution but in finding out WHY the values are distributed this way. Is this correct ? – mlwida Mar 23 '11 at 15:34
  • @mbaitoff - one problem with the way you have presented this question is that we have to guess at what the values are. If you could supply some actual data points that would be helpful – Ralph Winters Mar 23 '11 at 17:12
  • @steffen: Actually, both questions are of much importance. I expect that to derive the physical background from guessed distribution is easier than to derive the distribution from the physical background, since the measured process is specific (an amplitude measurements of inducted seismic waves). – mbaitoff Mar 23 '11 at 18:49
  • 1
    @mbaitoff, I'm not sure it would quite fit your application, but in related application areas, magnitudes of waves that undergo (many) random reflections between source and receiver are modeled by a Rayleigh distribution or one of its generalizations, e.g., Rice or Nakagami-$m$ distributions. – cardinal Mar 23 '11 at 20:01
  • @Ralph Winters: I have edited the post to provide a link to file with raw data values. – mbaitoff Mar 24 '11 at 04:55
  • 2
    The real interest in these data lies in the dozen or more spikes: the amount of data is large enough that those are *real*, in the sense that they are evidence of actual local modes. There seems to be a rich set of data here with a wealth of information that would be overlooked were a simple parametric formula used to summarize their distribution. – whuber Mar 25 '11 at 02:40
  • @whuber♦: You're right, those spikes are not random outliers, they seem to represent groups of outliers with some seriously deviant properties. Since the background of the raw data acquisition is indeed group-based, this makes sense. However, raw data represents the maximums of the registered wavelets, and anomalous spikes in wavelets can distort the distribution. I'm now calculating some more robust statistics of the wavelets. – mbaitoff Mar 25 '11 at 04:25

3 Answers3

23

Use fitdistrplus:

Here's the CRAN link to fitdistrplus.

Here's the old vignette link for fitdistrplus.

If the vignette link doesn't work, do a search for "Use of the library fitdistrplus to specify a distribution from data".

The vignette does a good job of explaining how to use the package. You can look at how various distributions fit in a short period of time. It also produces a Cullen/Frey Diagram.

#Example from the vignette
library(fitdistrplus)
x1 <- c(6.4, 13.3, 4.1, 1.3, 14.1, 10.6, 9.9, 9.6, 15.3, 22.1, 13.4, 13.2, 8.4, 6.3, 8.9, 5.2, 10.9, 14.4)
plotdist(x1)
descdist(x1)

f1g <- fitdist(x1, "gamma")
plot(f1g)
summary(f1g)

enter image description here

enter image description here

bill_080
  • 3,458
  • 1
  • 20
  • 21
  • (+1): Didnt know that package before. – mlwida Mar 23 '11 at 21:53
  • 1
    (+1( did not know that was called a Cullen/Frey diagram. I had to come up with that myself at one point. – Glen_b Apr 22 '13 at 06:09
  • the second image is with `plotdist` comamnd? How I can get the Cullen/Frey Diagram? – juanpablo Jul 02 '13 at 18:51
  • 1
    @juanpablo - Try `descdist()`. I updated the above post to include some code and a link to the old vignette. I could't get the above vignette link to work. So, Google the following: "Use of the library fitdistrplus to specify a distribution from data". It is a .pdf file. – bill_080 Jul 03 '13 at 00:10
  • thanks @bill_080, in the top-left image of `plot(f1g)`, what is the interpretation of that chart ? the bars is a representation of the histogram ? why the density curve over the histogram ? – juanpablo Jul 03 '13 at 14:09
  • 3
    @juanpablo - The statement `f1g – bill_080 Jul 03 '13 at 15:13
  • @bill_080 many thanks, but the histogram and density have different "scales", how can they be compared ? – juanpablo Jul 03 '13 at 15:47
  • @juanpablo - You'll have to dig into the package to verify the exact histogram scaling calculation (probably peak histogram value scaled to match the peak density value), but it's the similarity in shape that is important. That applies to all four of those images. It each case, the empirical plot is being compared to the "theoretical" (or "fit") plot. – bill_080 Jul 03 '13 at 17:30
7

Population is about 15 million samples.

Then you will very likely be able to reject any particular distribution of a simple, closed form.

Even that tiny bump at the left of the graph is likely to be enough to cause us to say 'clearly not such and such'.

On the other hand, it's probably pretty well approximated by a number of common distributions; obvious candidates are things like lognormal and gamma, but there are a host of others. It you look at the log of the x-variable, you can probably decide whether the lognormal is okay on sight (after taking logs, the histogram should look symmetric).

If the log is left skew, consider whether Gamma is okay, if it's right skew, consider whether inverse Gamma or (even more skew) inverse Gaussian is okay. But this exercise is more one of finding a distribution that's close enough to live with; none of these suggestions actually have all the features that appear to be present there.

If you have any theory at all to support a choice, toss out all this discussion and use that.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
1

I am not sure why you would want to classify a sample to a specific distribution with such a large sample size; parsimony, comparing it to another sample, looking for physical interpretation of the paramters?

Most statistical packages(R, SAS, Minitab) allow one to plot data on a graph that yields a straight line if the data come from a particular distribution. I have seen graphs that yield a straight line if the data is normal(log normal-after a log transformation), Weibull, and chi-squared come to mine immediately. This technique will allow you to see outliers and give you the possiblity to assign reasons for why data points are outliers. In R, the normal probability plot is called qqnorm.

schenectady
  • 1,079
  • 7
  • 11
  • Good idea suggesting the qqplot. However, I think that your explanation of the technique is a little vague/hard to understand. Can you provide some exemplary R-code ? This would increase the value of the answer drastically. – mlwida Mar 23 '11 at 12:07
  • I expect that somebody encountered the picture like mine and investigated the underlying distribution, because the values have physical basis. – mbaitoff Mar 23 '11 at 12:14
  • I am investigating the physical background of the sample distribution - how it is distributed and why. – mbaitoff Mar 23 '11 at 12:18