How can one compute Lilliefors' test for arbitrary distributions?

Question

My question is actually a follow-up to Glen_b's answer to the question "Simulation of KS-test with estimated parameters."

I am mostly interested on how to compute Lilliefors' test (or, more exactly, the corrected version of Kolmogorov-Smirnov test when the parameters from the target distribution has been actually been estimated from the data - be it with Lilliefors' test or something else) for distributions other than the Normal. It seems that most of the time the Lilliefors test is discussed it is used to check whether a sample comes from a normal distribution, but this is not really a limitation of this test.

As such, my question actually is twofold:

1. Are there any limitations on which distributions Lilliefors' test can work with? i.e. can it be extended to work with a Gamma, Chi-square, or maybe even the empirical distribution function?
2. How can we extend it to work with those distributions?

I have a rough idea on how 2 can be accomplished, but I still couldn't fully understand some parts. For example, in an answer to the aforementioned question, Glen_b gave the following description on how to apply the test through simulation:

Repeat many times:

1. Simulate a sample of the desired sample size from the assumed distribution.

2. Estimate the parameters of the distribution.

3. Treating the estimated parameters as the population values, transform to uniformity via the probability integral transform. (You can compute a KS statistic without transforming at this step; however, it makes the computation a bit simpler.)

4. Compute a KS test statistic.

Collect the simulated statistics, and work out the proportion of times the simulated statistic is at least as extreme (more consistent with $H_1$) as the observed sample value.

Some of my doubts:

• In step 1, which parameters should we use for the assumed distribution when we are sampling? Is it before or after fitting with the data we have?

• What exactly it means to "work out the proportion of times the simulated statistic is at least as extreme as the observed sample value"?

• With this method, the end result will be a new p-value that we can compare against our chosen significance level? Or the significance level had to be somehow taken into consideration for the last part (working out the proportion)?

Answer 1

[At present this only deals with the initial question regarding limitations. I may come back to address some of the other questions.]

The Kolmogorov-Smirnov test (i.e. one with a fully specified continuous distribution) is itself distribution-free -- the distribution of the test statistic doesn't depend on what that specified distribution is.

In the case of the Lilliefors test we know the distributional form but we don't know one or more parameters, so the distribution isn't fully specified (we estimate those unknown parameters) and as a result the test isn't distribution free - we need to treat each distribution separately.

The central issue with the standard approach to the Lilliefors' test is that you want the distribution of the test statistic to stay the same across different sets of parameter values.

Given some specified distribution, what we want then is that the test works the same no matter what the true parameter values are.

Consider the cases Lilliefors looked at - a normal and an expoenential. Let's take the exponential first. When we estimate the scale parameter ($\mu$ say), and then divide the observed values by that scale ($V_i=X_i/\hat{\mu}$) to get a standardized set of values, the distribution of those standardized values doesn't depend on the true scale parameter, $\mu$ (it's in both the numerator and denominator and so cancels out).

Similarly, if we estimate both parameters in the normal -- the distribution of the standardized values $Z_i=\frac{X_i-\hat{\mu}}{\hat{\sigma}}$ don't depend on $\mu$ and $\sigma$.

(You may find it useful at this point to read about pivotal quantities and ancillary statistics)

As a result, in cases like these, the distribution of the test statistic doesn't change as we change the parameter values; it only depends on the particular distribution, which parameters are estimated (e.g. it changes again if we only estimate one of the parameters in the normal), and the sample size.

This is not always so. For example if we were looking at a beta distribution, it's not the case that simply putting in the estimated parameter values and using the probability integral transform leaves the distribution of the test statistic unchanged as you change parameter values. I look at a gamma example below.

In some circumstances it may not make a huge difference (you might still have an approximate test), and in some cases it might not work well at small sample size but may be reasonable at large sample sizes. Such things are a matter for investigation -- but unless you have the property discussed above you can't necessarily just assume that things will just work without some reason to believe they will.

This is the reason for my caution in the original thread you refer to.

Example of the issue at hand:

You mentioned the gamma in your question, so here's an small example of the issue with that, looking at a small and a large value of the shape parameter. Note that it's just the shape parameter that's the problem here, since the scale parameter estimate can just be used to scale the data in the same way as the exponential:

As you see the right tail of the two distributions is different. However, for non-small values of the parameter the cube root transformation leaves gammas having almost the same shape (but differing in location and scale, as a function of the parameters). This suggests that you could safely have a "large-shape-parameter" approximate test - it suggests that the distribution should be almost the same for say $\alpha=10$ as $\alpha=100$, for example.

[Further, it looks like the quantiles of the KS statistic at small values of the shape parameter is nearly linear in the quantiles for larger values, so it's possible there may be something approximate that could be done with smaller estimated shape parameters to get a test of about the right size.]