1

Suppose that one runs 8 correlation analyses but in two separate batches: Would the alpha-correction apply to all 8 tests as a family or each of the 4 tests as a family?

For example, if there are two research questions and both require correlation analysis, the researcher conducts the first 4 tests (for research question 1) and then conducts the second 4 tests (for research question 2). In this case, should the alpha be corrected for a family of 4 tests or for a family of 8 tests? (Note that the same statistical test--Pearson's $r$--is used in both cases.)

I understand there isn't a consensus on this matter (as with many other matters in statistics as I am learning fast!) but some broad guidelines would be most helpful.

chl
  • 50,972
  • 18
  • 205
  • 364
Adhesh Josh
  • 2,935
  • 16
  • 50
  • 67
  • There was [similar question](http://stats.stackexchange.com/questions/19572/what-might-be-a-clear-practical-definition-for-a-family-of-hypotheses-with-r) on what might be considered a family of tests. – ttnphns Jan 11 '12 at 11:27
  • Thanks! The context above is slightly different, I believe, though. As always, many thanks for the responses. – Adhesh Josh Jan 11 '12 at 22:30
  • @AdheshJosh: I think you have to click some symbol (I don't remember what) to award the bounty to MånsT. – j.p. Jan 16 '12 at 13:12
  • I did click "+50" but nothing happened! – Adhesh Josh Jan 17 '12 at 06:01

1 Answers1

6

I'd say that it depends on your research questions. Are the two research questions related to the same data set? Are they completely independent from eachother? Would a false negative be worse than a false positive for either of these questions? If you have jellybeans type questions, correct for all of them. If your first question is about the link between smoking and cancer and your second question is about the link between a father's length and his son's length, correct for 4 in each batch. If you're somewhere in between, I don't think that there is a "right answer" to this.

You could correct for all 8 tests. But then again, you could correct for all 400 tests that you run this year, to get at most a 5 % (say) risk of making a type I error in 2012. You have to draw the line somewhere...

Also, as you run multiple correlation tests, it sounds like you might have multivariate data. Have you considered using some multivariate generalization of Pearson's correlation coefficient, such as canonical correlations?

MånsT
  • 10,213
  • 1
  • 46
  • 65
  • There can't be a better explanation than this! Many thanks. I am giving you the bounty points! – Adhesh Josh Jan 13 '12 at 16:53
  • A very well-argued response. For a similar but more detailed one, I recommend Geoffrey Keppel's Design and Analysis: A Researcher's Handbook. I'll find references to specific page no.s (1991 edition) if anyone would like that. – rolando2 Jan 13 '12 at 23:19