17

I'm a pharmacologist and, in my experience, almost all papers in basic biomedical research use Student's t-test (either to support inference or to conform to expectations...). A couple of years ago it came to my attention that Student's t-test is not the most efficient test that might be used: sequential tests offer much more power for any sample size, or a far smaller sample size on average for equivalent power.

Sequential procedures of varying complexity are used in clinical research but I've never seen one used in a basic biomedical research publication. I note that they are also absent from the introductory level statistics textbooks that are all that most basic scientists are likely to see.

My question is three-fold:

  1. Given the very substantial efficiency advantage of sequential tests, why are they not more widely used?
  2. Is there a drawback associated with the used of sequential methods that would mean that their use by non-statisticians is to be discouraged?
  3. Are statistics students taught about sequential testing procedures?
Michael Lew
  • 10,995
  • 2
  • 29
  • 47
  • 3
    Just to be sure, are you talking about ST as found in clinical trials, e.g. http://en.wikipedia.org/wiki/Sequential_analysis? – chl Oct 25 '10 at 14:07
  • Yes. There are quite a few variants of sequential testing, including sequential t-tests, but none are used in basic research. I don't see any impediment to their use. – Michael Lew Oct 25 '10 at 19:19
  • (+1) Just stumbled across sequential testing and asked myself the same questions. – mlwida Jul 15 '11 at 08:02

1 Answers1

5

I don't know much of sequential tests and their application outside of interim analysis (Jennison and Turnbull, 2000) and computerized adaptive testing (van der Linden and Glas, 2010). One exception is in some fMRI studies that are associated to large costs and difficulty to enroll subjects. Basically, in this case sequential testing primarily aims at stopping the experiment earlier. So, I am not surprised that these very tailored approaches are not taught in usual statistical classes.

Sequential tests are not without their pitfalls, though (type I and II error have to be specified in advance, choice of the stopping rule and multiple look at results should be justified, p-values are not uniformly distributed under the null as in a fixed sample design, etc.). In most design, we work with a pre-specified experimental setting or a preliminary power study was carried out, to optimize some kind of cost-effectiveness criterion, in which case standard testing procedures apply.

I found, however, the following paper from Maik Dierkes about fixed vs. open sample design very interesting: A claim for sequential designs of experiments.

chl
  • 50,972
  • 18
  • 205
  • 364
  • 1
    Basic biomedical researchers do interim analyses all the time, they just don't declare them because they don't even know that it matters! I have surveyed researchers at a national congress and found that more than 50% did not know that the control of error rates from Student's t-test is dependent on a pre-determined fixed sample size. You can see evidence of that in the sometimes erratically varying sample sizes used. – Michael Lew Oct 26 '10 at 19:44
  • Some of the disadvantages that come from the complexities of sequential designs come specifically in the design of the analyses rather than in their implementation. Perhaps we could have a set of pre-canned designs for small sample basic experiments. – Michael Lew Oct 26 '10 at 19:49
  • 1
    @Michael About "fake" interim analyses (looking at p-values while study is still in an evolving stage): it looks like it is a misappropriate use of statistics, no more. – chl Oct 29 '10 at 22:24
  • 1
    @Chi On one level, yes, undeclared and uncorrected interim analyses are inappropriate (but it is done in ignorance, an ignorance that I believe points to inadequacies in the methods of teaching statistics to basic biomedical researchers...). However, if we consider it at a meta level, then it is possible to find some partial justifications. Many experiments involve such small samples that an increased false positive error rate may be a reasonable tradeoff for more power. Convention precludes a declared level of alpha higher than 0.05. – Michael Lew Oct 30 '10 at 23:25
  • I note in this context that basic biomedica researchers do not work in an exclusively Neyman-Pearson approach, even if statements that "results where P<0.05 were considered significant" might suggest otherwise. If we stay within the confines of Fisher's significance testing wherein considerations other than the achieved P value can be incorporated into decisions of how to deal with the test results, perhaps interim analyses might not be so bad. However, it is certain that a designed sequential test would be superior to an undesigned one. – Michael Lew Oct 30 '10 at 23:29
  • @Michael Yes, you made a good point. This reminds me of Rothman arguing against a strict correction for multiple testing at the expense of Power considerations. However, planned *vs.* unplanned comparisons, as well as pitfalls associated to what I called "fake" interim analysis, should be part of every courses on DoE or RCTs. – chl Oct 31 '10 at 10:39