One test vs many tests

Question

I have a dataset with answers to an online survey from ~800 participants. These participants can be divided into 5 groups. We can call the groups A, B, C, D and E. I'm interested in three different types of comparisons: A vs B; A+B vs C+D+E; C vs D. I will use ordinal logistic regression (OLR) for the analysis.

My question is: should I run only one OLR and then look at the post hoc comparisons I am interested in, or run three different OLR, one for each of the comparisons I'm interested in?

If I run just one test with all groups together, I am accounting for multiple comparisons, but I am also losing power since I'm doing a bunch of comparisons I'm not really interested in.

If I run three tests, one for each comparison, then I'm comparing exactly the right groups, but I don't want to cheat on the multiple comparisons problem.

Answer 1

Generally it is considered better to fit one model using all of the groups, and then do post-hoc tests afterwards. This will give a consistent analysis, and is generally more powerful.

Some similar questions with answers:

• One predictive model for all data vs subject/group specific models

• How to test whether linear models fit separately to two groups are better than a single model applied to both groups?

• Forecasting at individual versus grouped level

• does it make sense to divide training data into sub-groups and fit model for each of them?