11

Reading the answers to this thread, I started wondering about how Hypothesis Testing relates to the Scientific Method. While I have a good understanding of both, I am having a hard time drawing the precise connection between them.

At a high level, the scientific method, comes down to:

  • Make conjectures & hypotheses (theory)
  • Make predictions from this theory
  • Carry out experiments and observations

  • Test and embrace the new theory if

    • the data fit the predictions (more) accurately than alternative theories
    • the new theory is not more complex than other plausible alternatives

At a high level, it looks to me that the scientific method thus follows an "accept-if-fits-well" approach which contrasts with the "reject-if-it-doesn't-fit" approach from statistical hypothesis testing. Is this correct? and if so, why is this the case? Aren't they both fundamentally chasing the same goal; inferring the theory or model that best explains the observations?

Community
  • 1
Amelio Vazquez-Reina
  • 17,546
  • 26
  • 74
  • 110
  • 1
    The new theory doesn't have to be simpler than alternative theories. Also, another feature of new theories is that they usually encompass the old theories. E.g. special relativity covers Newton's motion theory. Maxwell's equations cover Ohm's law etc. – Aksakal Dec 30 '14 at 14:55
  • 1
    But what do people usually use NHST to reject? Usually it is not their own or anyone else's hypothesis. Hypothesis testing is fine if you have a theory/hypothesis to test with it. – Livid Dec 30 '14 at 15:01
  • 1
    HYpothesis testing theory seems to fit nice with Popperian ideas ... – kjetil b halvorsen Dec 30 '14 at 15:22
  • @kjetilbhalvorsen: except that Popper [rejected induction](http://en.wikipedia.org/wiki/Problem_of_induction#Karl_Popper)... – Xi'an Dec 30 '14 at 15:53
  • @Aksakal I have updated the OP. Model simplicity plays a role in science (e.g. with the bias for theories colloquially known as "elegant theories"), and presumably this role is as important as goodness of fit, but I agree that the tradeoff is not as explicit as it probably is in statistical theory. – Amelio Vazquez-Reina Dec 30 '14 at 16:08
  • 1
    @user023472, one problem I have with your exposition is that you seem to put *scientific method* and *hypothesis testing* on the same epistemology level. It's like comparing a *hammer* and *carpentry*, hypothesis testing is one of the tools of scientific method. – Aksakal Dec 30 '14 at 16:12
  • 3
    This characterization of the scientific method does not appear to conform to what scientists actually do, nor to how philosophers write about it. In particular, I am not aware of anyone who has articulated or advocated an "accept-if-fits approach": this sounds almost the opposite of the scientific method, which would be much better (if oversimplistically) characterized as "reject-if-doesn't fit." But maybe I misunderstand: would you have a reference to support your characterization? – whuber Dec 30 '14 at 16:12
  • 2
    @whuber, almost everything in my post I got from the [definition on Wikipedia](http://en.wikipedia.org/wiki/Scientific_method) for the scientific method. The "accept-if-fits-well" vs "reject-if-it-doesn't-fit" is my own characterization to summarize the question. If this characterization is incorrect, an explanation of why that's the case would constitute, IMHO, an answer. I will rephrase the OP to make that clear. – Amelio Vazquez-Reina Dec 30 '14 at 16:16
  • 1
    @whuber is right. In science it is often said that one can prove the theory wrong, but can never prove it's right, which is very similar to hypothesis testing. – Aksakal Dec 30 '14 at 16:17
  • 1
    @Aksakal They are of course similar in the sense that they don't require a proof, but this question is about the perceived difference in the way they conduct theory selection, and the precise way in which these model selection processes relate to each other. – Amelio Vazquez-Reina Dec 30 '14 at 16:25
  • @user023472, in science you're never sure that theory is right, like in hypothesis testing. As the evidence mounts, you get comfortable with theories, of course. Scientists always look for evidence against the theories, that's the most fun part of their job to break a theory. – Aksakal Dec 30 '14 at 16:28
  • 4
    @Aksakal I don't think that is correct. You can only disprove the conjunction of the theory + various assumptions (eg, the equipment was functioning correctly). People will not throw out a theory that seems useful just due to some conflicting evidence. I think almost always people do not believe the theory is 100% correct anyway, the theory needs to be replaced by something more convincing. I'd be interested if you have an example in mind though. – Livid Dec 30 '14 at 16:31
  • 1
    Xi'an: And what does hypothesis testing have to do with induction? HT seems to fit with the standard presentations of hypothetical-deductive method, close to Poppers ideas. No induction in there, in theory ... – kjetil b halvorsen Dec 30 '14 at 16:54

1 Answers1

4

These issues have been known for a long time, it started in education research, psychology and has since spread to even physics. There is no one in particular to blame and apparently nothing can stop it.

We are quite in danger of sending highly trained and highly intelligent young men out into the world with tables of erroneous numbers under their arms, and with a dense fog in the place where their brains ought to be. In this century, of course, they will be working on guided missiles and advising the medical profession on the control of disease, and there is no limit to the extent to which they could impede every sort of national effort.

Fisher, R N (1958). "The Nature of Probability". Centennial Review 2: 261–274.

The usual application of statistics in psychology consists of testing a "null hypothesis" that the investigator hopes is false. For example, he tests the hypothesis that the ex perimental group is the same as the control group even though he has done his best to make them perform differently.Then a "significant" difference is obtained which shows that the data do not agree with the hypothesis tested. The experimenter is then pleased because he has shown that a hypothesis he didn't believe, isn't true. Having found a "significant difference," the more important next step should not be neglected. Namely, formulate a hypothesis that the scientist does believe and show that the data do not differ significantly from it. This is an indica tion that the newer hypothesis may be regarded as true. A definite scientific advance has been achieved.

MATHEMATICAL SOLUTIONS FOR PSYCHOLOGICAL PROBLEMS. HAROLD GULLIKSEN. American Scientist,Vol. 47, No. 2 (JUNE 1959), pp. 178-201

The major point of this paper is that the test of significance does not provide the information concerning psychological phenomena characteristically attributed to it; and that, furthermore, a great deal of mischief has been associated with its use. What will be said in this paper is hardly original. It is, in a certain sense, what "everybody knows." To say it "out loud" is, as it were, to assume the role of the child who pointed out that the emperor was really outfitted only in his underwear. Little of that which is contained in this paper is not already available in the literature, and the literature will be cited.

THE TEST OF SIGNIFICANCE IN PSYCHOLOGICAL RESEARCH. DAVID BAKAN. Psychological Bulletin. VOL. 66, No. 6. DECEMBER 1966.

The puzzle, sufficiently striking (when clearly discerned) to be entitled to the designation “paradox,” is the follow- ing: In the physical sciences, the usual result of an improvement in experimental design, instrumentation, or numerical mass of data, is to increase the difficulty of the “observational hurdle” which the physical theory of interest must successfully surmount; whereas, in psychology and some of the allied behavior sciences, the usual effect of such improvement in experimental precision is to provide an easier hurdle for the theory to surmount. Hence what we would normally think of as improve- ments in our experimental method tend (when predictions materialize) to yieldstronger corroboration of the theory in physics, since to remain unrefuted the theory must have survived a more difficult test; by contrast, such experimental improvement in psychology typically results in a weaker corroboration of the theory, since it has now been required to survive a more lenient test.

THEORY-TESTING IN PSYCHOLOGY AND PHYSICS: A METHODOLOGICAL PARADOX. PAUL E. MEEHL. Philosophy of Science, 1967, Vol. 34, 103–115.

Livid
  • 1,078
  • 6
  • 15
  • 3
    How much of this represents a problem with the **scientific method** itself vs a problem of **malpractice** or **improper** use of the scientific method? – Amelio Vazquez-Reina Dec 30 '14 at 15:46
  • 1
    @user023472. I don't follow you. Those quotes are from people complaining about the replacement of science as it used to be practiced with something else. – Livid Dec 30 '14 at 15:53