0

I need to construct a statistical test which outputs the p value for the hypothesis, H0: Data are Pareto distributed Vs H1: Data are not Pareto distributed.

I found a test in stack exchange, but it's very slow and, not working for binomial data (for validation, I generated Pareto, Normal, Exponential, Binomial etc and fed the function. It particularly gave pvalue = 0 for non pareto data, and a pvalue > 0.05 for pareto data, but an error message for binomial so it cannot be used any further) How to know if my data fits Pareto distribution?

And in another reference, I could find another test, where my R code doesn't give the desired answers. https://www.researchgate.net/publication/225199125_Goodness-of-Fit_Tests_for_Pareto_Distribution

Please do share a link for the test for Pareto data.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Dovini Jayasinghe
  • 269
  • 1
  • 13
  • Note that if (see [wikipedia](https://en.wikipedia.org/wiki/Pareto_distribution#Relation_to_other_distributions) $X$ is pareto distributed then $\log (X/x_m)$ is exponentially distributed, so look into goodness of fit for exponential distribution. – kjetil b halvorsen Feb 20 '19 at 09:07
  • Yes, then we have to check for testing of whether log(X/xm) is exponentially distributed. Is there any way of doing that? I suppose this link also says the same. – Dovini Jayasinghe Feb 20 '19 at 09:28
  • What is the context? That is, what kind of alternatives are relevant? What kind of phenomenon are you modeling? – kjetil b halvorsen Feb 20 '19 at 09:50
  • I'm trying to write a new package for R. There I need to add a function such that the user can input his set of data to it and obtain a pvalue. Based on that pvalue, the user can get his data set checked whether pareto distributed or not. For instance, I can generate random data from R and input the function and see. There I expect, random pareto data to give the p value greater than 0.05 and all the other non pareto random data (data generated from other distributions) to give a pvalue less than 0.05 – Dovini Jayasinghe Feb 20 '19 at 11:00
  • Nice with R-package, but that package must have some specific goal, specialized for some specific kind of data, giving a context. Which? In the question you say you implemented some test, but it did not work with data generated from a binomial distribution. But you would not expect that! This distributions do not even have the same support, and in a context where Pareto is a reasonable model, binomial would not be! So testing methods in such a haphazard way do not make sense. So again, tell us what kind of data you want to analyze. – kjetil b halvorsen Feb 20 '19 at 22:23
  • Any kind of data it may be. A user is free to give even a vector like c(1,2,3,4,5,6) for that function, who needs to check whether it's pareto or not. Because in this package, after testing for pareto, if the test passes, the end user can find the estimates for the parameter of pareto using different methods of estimation. – Dovini Jayasinghe Feb 21 '19 at 04:30

1 Answers1

1

There are no classical tests which can confirm that a set of data follow a particular distribution. Even proposed tests of normality, or other distributions, all have the common pitfall that they are arbitrarily powerful in large samples. This means that you would expect to reject the null (to say the data do not follow the given distribution) even when it's practically true in every sense. Even stating as a null hypothesis that the data do follow a Pareto distribution, a pristine Pareto sample could lead to a Chi-sq test statistic of 1 and a two sided p-value of p=0.5, yet it would fail to convince that the null hypothesis is in fact true because that is not the correct interpretation of a "null result" as it is called. Furthermore, if you were to state as an alternative hypothesis that the data are Pareto, then what is your null? Could the data follow a Pareto except that the 99-th percentile of the distribution is truncated? What would the power of any test be to detect such a difference?

If you take a pragmatic approach, the QQ plot and draw the imposed maximum likelihood estimate over the smoothed empirical density, you can attempt to show graphically when there are egregious departures from the Pareto distribution.

AdamO
  • 52,330
  • 5
  • 104
  • 209
  • I ran code for the qq plot and the corresponding p values to see whether there's any significant difference from pareto. But this didn't give the desired pvalue. Have I done anything wrong here? – Dovini Jayasinghe Feb 22 '19 at 09:24
  • @DoviniJayasinghe 1) CV isn't for debugging code. 2) you don't get the "p-value you 'want'", you get the p-value that there is. 3) I never recommended testing in the first place. I already mentioned goodness-of-fit tests are arbitrarily powerful. Please reread my answer. – AdamO Feb 22 '19 at 15:47