Should false discovery be controlled at the data acquisition level, or should this be at the data interpretation level?

Question

Should false discovery be controlled at the data acquisition level, or should this be at the data interpretation level?

I have an experiment in which microarrays were used to quantify the expression of about 30,000 genes (variables) in two groups of biological tissues (group-sizes of 75 and 76). Raw array data was pre-processed to remove background signals and genes whose expression levels are undetectable, and to normalize values across arrays. The final data was then examined using the Mann-Whitney U test to compare gene expression between the two groups to identify differentially expressed genes, with false discovery rate (FDR) controlled with the Benjamini-Hochberg procedure. At FDR <5%, no gene is identified as differentially expressed, and I formally conclude that 'there is no differential expression of genes between the two tissues.'

Now, suppose someone is interested in the expression of only one specific gene. Using my pre-processed gene expression data-set and the U test, they compare the two groups for the expression of only this gene and notice a P value <0.05, the commonly used significance threshold in my field of study. As this does not involve multiple testing, there is no false discovery control. Can this observer formally conclude that 'the gene is differentially expressed between the two tissues,' contradicting my conclusion?'

Or should the observer have applied false discovery control because such a control has to be applied at the data acquisition level (as per which, data on multiple variables were collected) and not at the data interpretation level (as per which, data for only one variable was analyzed)?

Answer 1

I would argue strongly that is should apply only at the interpretation level. Multiplicity implicitly involves the definition of an investigation by an investigator(s) (i.e. the study wise error rate to be controlled) and needs to accurately reflect the intentions that drove the process of generating inputs to the inference/decision. (This is a bit slippery and for instance Wittgenstein admitted late in his career that he regretting not realizing intentionality early in logic.)

For instance, if someone intended to do all the comparisons but stopped with the first one because it was so good – this is a multiplicity to be dealt with. On the other hand if that comparison was credibly documented as the only one to be made – there isn’t. It should not matter if the data entry clerk who was taking a statistics course without permission ran all possible comparisons as an exercise. That sounds like your situation to me. ( This judgement can very slippery and thanks to user603, I can point to Jake's birthday as a good example http://www.johndcook.com/blog/2012/09/07/limits-of-statistics/ )

Something like this happened to an early colleague. They want to test A versus placebo but someone wanted them to includ B as well. They thought B was silly but being a nice guy included the B group. The result was A versus placebo was clearly significant but not after adjusting for B. They could never get the study published because of that.

Also, Ed George had a nice talk at the joint meeting this summer where he was in effect arguing for an analysts' posterior for those who have access to the data versus a reported posterior for those who only find out about the study if it is selectively reported to them.

Thinking about his talk afterwards and that slippery intentionality stuff possibly also applying to the analysts, the "Men in Black" movie seemed relevant or at least their use of the Neuralizer - http://en.wikipedia.org/wiki/Neuralizer

It’s as if the Bayesian analyst knows they would have been neuralized as soon as they realized a given data set did not achieve some pre-set goal. So when they get a data set that does meet it, they realize they don't know how often they have been neuralized but they know the selection rule for avoiding being neuralized this time.

Answer 2

As @phaneron has stated, there is no need for multiplicity control if you only consider one gene. I wish to add to that hypothesis testing serves two purposes: (a) convince yourself and (b) convince "the world". For the purpose of (a), recall that the BH procedure controls the "expected proportion of false discoveries over re-tests of the same hypotheses". If you (honestly) consider the test of one hypothesis (snp), there is no multiplicity at hand and the only question left is "do you find frequentist hypothesis testing convincing?".

For the purpose of (b) the difficulty might be to convince that the choice of the snp was "honest". Which technically means that the focus on that single snp was made using different (thus statistically independent) data than the data serving for the hypothesis testing.