0

I prefer the Bayesian approach to hypothesis testing myself, and when I look at how traditional null hypothesis testing is taught, I see a problem that is never talked about.

For example: While testing for a difference between two independent groups, it is very common to find people testing each group for normality, then testing for equality of variances and then doing either a parametric or a non-parametric test. But then the overall probability of a Type-I error then is no longer 0.05 - it is closer to 1 - 0.95^3 or 1 - 0.95^4 (0.14 or 0.19) depending on how many assumption tests have been done.

But you never see this mentioned in books on null hypothesis testing. Many websites advocate doing tests of assumptions before the actual statistical test and this technique is almost always used by researchers in the social, physiological sciences. Many textbooks with a generic title of "Statistics for ..." even have flowcharts indicating which tests to perform to test assumptions before the actual test.

Comments?

Ravi
  • 23
  • 2
  • 1
    I wouldn't say this is never talked about, but you are right that the approach described is generally a bad idea and is often advocated for on various websites and even textbooks. There are many people who spend a lot of time trying to correct misunderstandings about how to access if data are appropriate for traditional parametric analysis, but they lose more hair than win converts. – Sal Mangiafico Oct 20 '21 at 11:28
  • BTW, I don't think the probabilities will work out as simply as you suggest. Because if one were to get a type-I error on, say, the test for normality, then they wouldn't use, say, the parametric test they were planning on using, but instead choose some other test. What's the probability of choosing a specific test? Or if, say, the parametric and non-parametric test have the same result, did that change the type-I probability at all? Or if the nonparametric test has lower power, does that boost the type-II probability? All this is way above my rudimentary understanding of probability. – Sal Mangiafico Oct 20 '21 at 11:35
  • There are many posts about this topic here, but maybe most in some particular contexts such as t-test. Search! [One paper here](https://link.springer.com/article/10.1007/s00362-009-0224-x), for example https://stats.stackexchange.com/questions/381790/the-assumptions-regarding-the-t-test, https://stats.stackexchange.com/questions/313471/always-use-welch-t-test-unequal-variances-t-test-instead-of-student-t-or-mann, https://stats.stackexchange.com/questions/273193/bayesian-t-test-assumptions/273227#273227 – kjetil b halvorsen Oct 20 '21 at 23:07
  • Thank you for your comments. My estimate of the Type-I error when we test many assumptions was a very simplistic estimate. Yes, we need to worry about the issues raised by Mangiafico. – Ravi Oct 21 '21 at 03:45
  • Just one additional thought. The better textbooks ask one to judge normality by looking at a QQ plot or a histogram. But that just shifts the onus from doing say, a Shapiro-Wilk test to a judgemental test whose probability of a Type-I error we do not know. – Ravi Oct 21 '21 at 03:50

0 Answers0