Can I simulate a two-sample K-S test when I can't perform a one-sample test because a distribution parameter is unknown?

Question

Let's say I have a signal which I want to test for normality. I know that the mean from the theoretical distribution is zero, but it has an undetermined (unknown) variance. If I knew mean and variance a priori, I would run a one-sample Kolmogorov-Smirnov test. Since I don't know what the variance should be, I am thinking of generating some random data from a normal distribution with zero mean and same variance as the sample (signal) and perform a two-sample K-S test, one sample being the true (original) sample, and the other being the randomly generated one. Is this procedure correct/valid?

Some contextualization:

♫ I know one should not perform a one-sample K-S test with theoretical distribution parameters predicted from the data, that's why I'm thinking of doing this "simulation" for the two-sample test and I'm not confident of its validity;

♫ The signal is an acoustical room impulse response (its amplitude is proportional to the sound pressure). The normality is expected for the later part of the signal. It is also expected for the real and imaginary parts of its Fourier Transform. The signal may also be a simulated impulse response, created through the decaying exponential weighting of a gaussian noise;

♫ I'm not interested in other tests (like the Lilliefors test) because research with the procedure I am explaining here has already been published (applied to all topics cited above). So I'd like to know if the procedure is valid or not (and its effects) so I can better understand and evaluate the results. I'm not one of the authors, so I don't have access to the exact utilized procedures or algorithms;

Answer 1

Your test will be rather conservative if you suppose the K–S statistic has the same distribution as in the usual two-sample K–S test; it'll be very liberal if you suppose the statistic has the same distribution as in the one-sample Lilliefors test. Here are the results of a simulation^† under the null using the sample size of 120 you mentioned in comments:

If you increase the size of the simulated sample enough (there's no reason it has to be equal to that of the observed sample), the distribution of the test statistic will approximate the one in the Lilliefors test. That's merely a roundabout way of calculating the Gaussian distribution function, which is already done very well by any statistical software. In any case, you'll still need to simulate the distribution under the null to calculate p-values (unless you want to look for tables or large-sample formulae for critical values, for the case when the mean is known & the variance unknown).

You could of course obtain p-values/critical values from the null distribution simulated above. But common sense says that evidence about the shape of the distribution from which a sample is drawn is to be found only in the observed sample; & not in extraneous noise introduced by coin tosses, or dice rolls, or pseudo-random number generators. Statistical theory goes a little further, saying it's to be found only in the the sample configuration—the (unordered) set of standardized residuals, or an equivalent. The consequences of ignoring these considerations are (1) an loss of power relative to to Lilliefors test, which is quite unnecessary, & (2) that different researchers, given the same data, & making the same assumptions, may likely come to very different conclusions about lack of fit because they draw different simulated samples, which is quite absurd.

---

† In R:

set.seed(1815)
n.obs <- 120
n.sim <- 120
B <- 50e3

stat.1s <- numeric(B)
stat.2s <- numeric(B)
stat.prop <- numeric(B)

for (i in 1:B) {
  x <- rnorm(n.obs, mean = 0, sd = 1)
  s <- sqrt(sum(x^2)/n.obs)
  stat.1s[i] <- ks.test(
    x, "pnorm", mean = 0, sd = s, exact = TRUE
  )$statistic
  y <- rnorm(n.sim, mean = 0, sd = 1)
  stat.2s[i] <- ks.test(
    x, y, exact = TRUE
  )$statistic
  y <- rnorm(n.sim, mean = 0, sd = s)
  stat.prop[i] <- ks.test(
    x, y, exact = TRUE
  )$statistic
}

df.1s <- ecdf(stat.1s)
df.2s <- ecdf(stat.2s)
df.prop <- ecdf(stat.prop)

plot(df.2s, col="forestgreen", verticals=TRUE, do.points= FALSE,
     xlim=c(0,0.2),
     xlab = "test statistic", ylab = "distribution function under null", main = NA)
plot(df.1s, col="dodgerblue", verticals=TRUE, do.points= FALSE, add = TRUE)
plot(df.prop, col="darkorange", verticals=TRUE, do.points= FALSE, add = TRUE)
legend(
  x = 0.11, y = 0.2, bty="n", #title = "max. distance of EDF from ...",
  lty=1, col=c("dodgerblue", col="forestgreen", "darkorange"),
  legend=c("Lilliefors test", "2-sample K-S test", "proposed test")
)