I often read that, after obtaining complex (e.g., four-way) interactions in a factorial ANOVA, researchers decide to split their data by a factor (e.g., gender) and run separate ANOVAs for these two groups.
I realize that this can be very helpful in understanding and explaining complex interactions. However, I have the feeling that this is statistically not completely sound. Isn't it better to perform the "big" ANOVA, including contrasts of interest to help understand the effects?
Also, how do you justify the way you split your data? For example, can you only include variables into the separate, "small" ANOVAs that were part of the significant interaction in the "big" ANOVA, or can you chuck in the whole bunch of independent variables again? This seems like effect-fishing to me.
I cannot find any helpful literature on this at all so I am unsure whether my gut-instinct is simply wrong. I'd be very grateful for any insight someone may have into the matter!
