2

Case 1

Suppose I wanted to test the association between variable 'a' and variable 'b1' (i.e., I did only one hypothesis test), and I reported that 'a' is significantly associated with b1 with 95% confidence.

Case 2

Now suppose I wanted to test the association between variable 'x' and hundred variables (b1-b100; i.e., I did 100 hypothesis tests). Among the 100 hypothesis tests in case 2, I found that x is associated with only one of these hundred variables—lets call it 'bx'. So, I reported that x is significantly associated with bx with 95% confidence.

Are the two cases the same? (By intuition, case 2 should be less reliable because I choose my result between 100 probabilities.) How do you measure the reliability in this case (does it simply mean that the probability that case2 is correct is 1/100)?

Community
  • 1
Elmahy
  • 467
  • 1
  • 3
  • 12
  • 1
    Look into familywise type I error rate. If the tests were independent, then when there in truth no association for any of the variables, then for each there would be a 5% probability of some association with a $p\leq 0.05$ under the null hypothesis and the probability with $m$ comparisons would be $1-0.95^m$ under the global null hypothesis. – Björn Jan 11 '16 at 17:32
  • 1
    maybe this can help: http://stats.stackexchange.com/questions/167289/is-there-a-multiple-testing-problem-when-performing-t-tests-for-multiple-coeffci/167338#167338 –  Jan 11 '16 at 17:39

0 Answers0