How to determine effect size in addition to significance in statistical inference tests that do not explicitly estimate effect size?

Question

Students in linear regression courses are taught that more data is good. They're taught that checking assumptions are good. They're taught that the Shapiro-Wilk test is good. Then they're taught that having a lot of data and testing a normality assumption with the Shapiro-Wilk test is bad, because it is too good at detecting deviations from normality when you have a large sample. When you look at the mathematics of the test, this makes perfect sense. But if you take a step back from the mathematics, you may begin to realize how utterly farcical the situation that we've created here is. It's like the punchline of a joke about mathematicians and assumptions.

In a similar vein to the famous aphorism "all models are wrong, but some are useful," I think we can also admit "most assumptions in statistics are wrong, but some are close enough." With that in mind, is there a way to quantify the practical relevance of a test result in a way that checks a "close-enough" assumption? I'd like to avoid the farce of "we can't trust our tests, because they're too accurate."

EDIT 2: This question is being misunderstood as a question about model assumptions or normality assumptions, but it is not. Here's an attempt to clarify the question. The power of statistical tests is the ability of the test to detect deviation from the null hypothesis. The power of tests increases with sample size. If the sample size is huge, even a trivial deviation from the null could lead to "statistically significant" results. If the effect size is small, we may not care that the test is statistically significant. This is not a problem for tests that are performed within a model, e.g. a t-test on a coefficient in OLS, because we also have an estimate of the effect size.

In many other statistical tests, we do not have an estimate of the effect size. Moreover, we already know that the null hypothesis in many tests is false (e.g. we know that real data is never perfectly normal, we know that two populations are never perfectly identical), and what we really care about is the effect size. If the sample size is small or medium, we can imitate knowing the effect size, because if the power of the test is low or moderate, it won't detect small departures from the null. In such a situation, a test finding statistical significance can be interpreted as a rough proxy for a significantly sized effect, because that's the only way the effect could have been detected with a low power test. That scheme breaks down if the sample is large, because the power is so high that any tiny departure from the null is detected and considered statistically significant. In that latter case, statistical significance does not contribute anything to our understanding of the problem we're trying to address.

The only purpose of my question is to determine if there are currently existing ways to estimate the effect size of such tests so that they can still be used with large samples.

Answer 1

You're describing measures of effect size rather than statistical significance. Significance is often used to describe likelihood of there being exactly zero effect. With enough data, you will often find that effects are not truly zero, but are too small to actually care about. For example, one could conduct an enormous clinical trial to show that Drug A is statistically significantly better than Drug B, indicating that the drugs do not have identical efficacy. But if you look at differences in survival, you might find that Drug A improves survival times by 5 minutes - the significance values allows you to conclude that the survival difference is almost certainly not 0 minutes, but examination of survival differences shows no real practical benefit. There are many ways to quantify effect size depending on corresponding statistical test, which could include differences in survival or differences in mean/median, fold-changes between values, etc. Effect sizes allow you to quantify the size of the differences observed, rather than just checking whether it is feasibly zero or not. What you'd define as a "meaningful" effect size is highly domain-dependent, however.

As we've discussed in the comments, there's no single measure of effect size for normality tests, since a distribution can deviate from normal in an infinite number of ways. Effect sizes typically capture differences in a single parameter (mean, survival times, etc), but there is no single parameter that captures normality. One can quantify particular aspects of a distribution like skewness or kurtosis and compare against those of a true normal distribution, but I'm not aware of a single effect measure that can quantify how non-normal a distribution is.

Answer 2

Some things to know about model assumptions:

1. Models are thought constructs and are by construction different from the real world. No model assumption is ever fulfilled, so I'm with Box on this one. Models are tools for thinking.
2. This also implies that no model is ever in any well defined sense "approximately fulfilled". As real data never are truly i.i.d., for example, finding small distances such as Kolmogorov-Smirnov between an empirical and an assumed distribution does not imply that the model assumption is approximately fulfilled.
3. That a certain method has a model assumption means that assuming the model it was shown to have some favourable properties. This is fine, as models are our favourite tool to make sure a method makes sense. It does not imply at all that in order for the method to work the model must be true or even approximately true.
4. Unfortunately there are situations in which a method will give misleading results, so we can't just rely on a method that is optimal under a certain model, or even a method that has been derived without parametric assumptions (the latter will still assume i.i.d. or similar, which does not hold in reality). What is of real importance is not making sure that the model assumptions (approximately) hold, but rather checking whether there is any indication that the result of the method will be misleading.
5. Model-based theory helps to some extent with this, as it gives us an idealised idea of when the method will get it right. It can also give us ideas about when a method can get it wrong, but this requires analysing what happens if the method is applied to data from certain wrong models. Robustness theory does some of this.
6. Whether a method will be misleading or not is an issue that is essentially different from what "distance"/misspecification test value we get under a formal test of a certain assumption. Quite generally, the danger of a misleading result when using a normality-based method on real data is not proportional to the strength of indication of these tests against normality. The problem with Shapiro-Wilks is not that it is "too good for large samples", but rather that diagnosing whether data mislead a model-based method is a task essentially different from testing a certain distributional shape. For example, most normality-based methods are just fine with distributions with light tails, even though such distributions may look very non-normal, whereas a t-distribution with low degrees of freedom is much harder to detect but will mess up most normality-based inference. Unfortunately it is hard to give general guidelines, as what is problematic depends on the method to be applied, but also on how results are interpreted. In my view it is a general problem in statistics that model assumptions are taken too literally, and there isn't much work helping with the real problem, which is diagnosing what may go wrong, rather than rejection (or not) of an assumed model.