Is it legitimate to reduce the number of pairwise comparison hypotheses based on expectations?

Question

Context

I'm comparing 7 classification algorithms with corrected resampled t-tests in 3 times repeated 10 fold CV. I have educated guesses as to how their performance could line up. (For example, a transductive semi-supervised algorithm will probably perform better than an inductive supervised one which can use only the small labeled part of the same training data etc.)

Problem

Before looking at any classification performance data, I have created a lineup of expected performance and written down my reasons for believing so. I could announce only 6 hypotheses:

H1 algorithm A performs better than algorithm B

H2 algorithm B performs better than algorithm C

...

H6 algorithm F performs better than algorithm G

I could also perform 7(7-1)/2=21 pairwise comparisons irrespective of my intuitions. Since I need to use a Bonferroni correction (or the more powerful Holm approach), it would be advantageous to keep the number of hypotheses to a minimum.

Questions

Is it legitimate to reduce the number of pairwise comparison hypotheses based on expectations?

Do I see this correctly that H1 and H2 combined imply that algorithm A performs better than C without requiring a dedicated hypothesis to test this? If both H1 and H2 have their null hypotheses rejected, then obviously yes, but what if this is not the case?

Also, I would take two-sided tests in spite of my intuitions. Like this, nobody can tell me that I might have adjusted the direction of the one sided test after seeing the data. Doesn't this conservative approach contradict the other decision to reduce the number of hypotheses?

Answer 1

I'll give you my opinion, but I'm sure that others will have different thoughts. I'll also propose a non-traditional solution that others are free to argue with.

The correction entirely depends on what conclusion you wish to draw from your dataset. For example, in your article will you state:

Algorithm A performed the best out of all examined classifiers.

or will you state

When successively compared in order of mention, algorithm A was significantly different than algorithm B, with the trend continuing across all classifiers.

As you probably know, Bonferroni and derivatives correct the family-wise error rate (FWER). Obviously then, the error rate $\alpha$ to which you correct will depend on the family which you have defined. What you're implicitly asking in your question, then, is whether your set of 6 hypothesis is a reasonable approximation of the total family of 21 tests. At this point it becomes a field specific question.

I see your reasoning in saying that if $H_1$ is rejected indicating that $A > B$ and $H_2$ is rejected indicating that $B > C$, then $A > C$. However, there are two issues with this.

• If you are testing two tailed hypothesis, what you're really testing is if $A = B$ and if $B = C$, and if $A \neq B$ and $B \neq C$ then it does not follow that $A \neq C$.
• Assuming you switch to one tailed tests, then if you fail to reject any one of your chain, then the conclusion is put into question. For example consider tests which indicate $$ \begin{aligned} A > B \\ B > C \\ C \ngtr D \end{aligned} $$ Does it follow that $B > D$ or that $A > D$?

A possible (and contestable) solution to this issue may be stratefication. This is analogous to identifying groupings within you hypotheses and correcting them separately. In practice, this brings in prior knowledge about the probability of a test being a true null. Your case would have stratum 1 comprising of your 6 hypothesis and the other comprising the remaining pairwise comparisons. This would allow you to validate your assumptions while also retaining the weighting on your identified 6 hypotheses.

You may wish to consult Sun et al. for full details and instructions.

I would like to add though, that the most conservative course would be doing the pairwise comparisons and drawing conclusions from that.