2

I am thinking about p-values correction in case of multiple test.

In general if the complex test 'contain' several simple test then probability of first type error increases. In this case I conclude that complex null hypothesis is rejected if at least one of the simple hyphothesis is rejected. To reduce 1-type error of complex hypotheisis testing due to cumulation of propability of 1-type error several method vere proposed : https://en.wikipedia.org/wiki/Multiple_comparisons_problem

The my question is what if we want to reject complex null hypothesis if all simple hyphothesis can be rejected.

In first case first type error probability can be calculated as: $$ \alpha_{complex} = 1 - (1-\alpha)^k $$ So to have the same $ \alpha $ value corrected p-value by Sidak correction is: $$ p_{corrected,i} = 1 - (1-p_i)^{k} $$

I assume that in case when we want reject null hypothesis if all $ k $ hypothesis are rejected then first type error can be calculated as folow:

$$ \alpha_{complex} = \alpha^k $$

So as well

$$ p_{corrected,i} = p_i^{k} $$

Is my assumption correct?

What do you think about other correction factor :

$$ p_{corrected,i} = \dfrac{p_{i}}{k} $$

I think that this factor can be better in some cases, but this just my 'personal feeling'. I do not have any evidence for that :)

PS. please correct me if I am completly wrong, I've just started my adventure with statistic:)

mdewey
  • 16,541
  • 22
  • 30
  • 57
Staszko90
  • 21
  • 4
  • 1
    I think by complex hypothesis you want to simultaneously test several hypotheses and by simple hypothesis you mean the individual hypotheses. You want to make a statement about rejecting all hypotheses. For that you want to assign a p-value to the simultaneous inference. This means that you need to require smaller p-values for each hypothesis in order to safely reject them all. – Michael R. Chernick Dec 21 '16 at 18:44
  • 1
    This is a strong requirement. Now there is a theory allowing out of a large number of hypothesis tests allowing a few false rejections. This is called false discovery rate (FDR). In the strict formulation there are several ways to adjust the p-value. Bonferroni takes the original choice for significance level and divides it by n where n is the number of hypotheses. This is fine if n is small. But it is overly conservative so there are several other ways of doing the correction. You should look into these. – Michael R. Chernick Dec 21 '16 at 18:56
  • 1
    Peter Westfall's book shows how bootstrap and permutation methods can be used and he compares them to the standard ones. – Michael R. Chernick Dec 21 '16 at 18:58
  • I think that for FDR analysis I have to have some knowledge about distribution of alternative hypothesis. If I do not have such a knowledge then i think that I shoud use lets say 'safety' mechanism for p-value correction (or $ \alpha $ correction). I think that the safest mechanism is in exponential equation. – Staszko90 Dec 22 '16 at 07:39
  • 1
    You might be interested in this Q&A http://stats.stackexchange.com/questions/243003/can-a-meta-analysis-of-studies-which-are-all-not-statistically-signficant-lead which discusses a related issue. – mdewey Dec 23 '16 at 09:55

0 Answers0