hypothesis test for a boolean combination of alternatives

Question

I have a student (for a high school project) who has conducted an experiment and collected data in the following (simplified) way:

id|situation|scoreX|scoreY

where id identifies the subject; situation only takes the nominal values A or B; and scoreX and scoreY are numerical measurements. Each subject has two entries, one where situation=A and the other when situation=B.

Originally she planned two hypothesis tests, using a t-test, with the alternatives: mu_scoreX_A > mu_scoreX_B and seperately mu_scoreY_A < mu_scoreY_B.

My thinking is that if the two original tests were to give p-values of 0.1, individually they are not significant, yet we are seeing two 'almost' effects, which together could be significant.

So my question: isn't it better to do one test against the alternative: mu_scoreX_A > mu_scoreX_B AND mu_scoreY_A < mu_scoreY_B. If so, what is the test one should use and do we need to adapt the significance too?

Taking my thinking one step further, are there rules and tests for any boolean combination of alternatives?

Answer 1

Originally she planned two hypothesis tests, using a t-test,

Warning: the data are paired, so you should use a paired test (a paired-t test is probably fine).

with the alternatives: mu_scoreX_A > mu_scoreX_B and seperately mu_scoreY_A < mu_scoreY_B.

That's not silly, if that's what you're interested in finding out (though it's more usual to use two-tailed tests unless there's a particularly good justification for one-tailed tests).

My thinking is that if the two original tests were to give p-values of 0.1, individually they are not significant, yet we are seeing two 'almost' effects, which together could be significant.

Quite right; indeed, if scoreX and scoreY are correlated, it's quite possible for a combined test to do much better.

So my question: isn't it better to do one test against the alternative: mu_scoreX_A > mu_scoreX_B AND mu_scoreY_A < mu_scoreY_B.

If you want power against the joint alternative, you could perhaps construct a test like this (you'll gain power where both are the case, but at the expense of being much less able to pick up the situation where only one of the variables is consistent with the alternative).

However you should probably consider the possibility of a multivariate test, for the reasons discussed here.

If so, what is the test one should use and do we need to adapt the significance too?

a) If the scores are expected to be similar in variability, one way to do it would be to construct a contrast consisting of their difference.

b) If you can't assume that, you might combine the tests using (for example) Fisher's method.

c) a third alternative might be to construct a custom test statistic which will be sensitive to both variables being in the anticipated region and perform a permutation test. For example,
$\quad T=\max(\bar{X}_A - \bar{X}_B,0) \times \max(\bar{Y}_B-\bar{Y}_A,0)$

Taking my thinking one step further, are there rules and tests for any boolean combination of alternatives?

The above suggested approaches may work for other combinations (such as additional variables)