Simulations-based alternative to NHST?

Question

In this PyCon Canada tutorial, the author details a simulation process, in lieu of A/B testing, where two respective distributions are sampled from compared for X iterations. You can collect whatever metadata you like, such as:

• "Is $\theta_1 > \theta_2$?"
• Or "By how much is $\theta_1 > \theta_2$?"

Note, this tutorial assumed fully Bayesian posterior inference on $\theta_1$ and $\theta_2$.

My question is could this logic be applied from a non-Bayesian perspective. For example, assume that the click through rates (CTR) of two ad campaigns are approximately normally distributed, fit their normal PDFs via MLE, then sample a,b from their respective distributions, compare, and capture the difference for x iterations?

Would this be a viable alternative to NHST?

Answer 1

You've invented something very much akin to permutation testing.

This is often done by

1. Computing the statistic of interest between the true groups.
2. Randomly shuffling the data into "surrogate" groups.
3. Recomputing the statistic on those groups.
4. Repeat steps 2-3 a large number of times (or until you've exhausted all possible permutations of the data)
5. Compare the true different from Step 1 against the distribution of surrogate differences created in Steps 2-4.

The shuffled data literally implements the null hypothesis--you know there's no difference between the surrogate groups because you've randomly assigned points to them. The differences between different surrogate groups, collected across many repetitions, therefore provide a distribution of your chosen statistic under the mean. A "significant" difference, in the NHST sense, would be way out in the tails of that distribution.

Bootstraping is a related concept, more aimed at assess uncertainty than inference. I found Randomization, Bootstrap and Monte Carlo Methods in Biology by Manley and Alberto to be really helpful (you do not need much biology background to make sense of it).