3

What is the term for when, through accident or intent, many hypothesis tests are conducted until an 'interesting' result is found? That is, the experiment (or variations on it) are repeated until a statistically significant result is found that other researchers would probably not be able to replicate (not without themselves running the test many times).

Some examples

If you torture the data enough, nature will always confess

You can only use an observation once to confirm a hypothesis. As soon as you use it more than once you’re back to doing exploratory analysis. This means to do hypothesis confirmation you need to “preregister” (write out in advance) your analysis plan, and not deviate from it even when you have seen the data.

What is this phenomena called - does it have a noun or a verb? That is, what is it called when someone does not adhere to statistical best practice, and repeats tests (perhaps with variations) until they find a result that stands out in some way?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
stevec
  • 259
  • 1
  • 7
  • 5
    P-hacking or data dredging are commonly used terms, see e.g. https://en.m.wikipedia.org/wiki/Data_dredging – Michael M Aug 07 '20 at 06:53

1 Answers1

4

As mentioned in the comments, the terminology p-hacking or data dredging are used for such actions of repeating hypothesis tests until through random occurrence a favored outcome or p-value is produced.

Cherrypicking the data is also a term for when you pick or disregard data in a way that favors your hypothesis, often deliberately.

HARKing is when you form a hypothesis after the result of analysis is already known.

Nuclear03020704
  • 730
  • 4
  • 18