12

Nowadays, in the scientific domain, increasingly, people in the top position argue against doing post-hoc analysis and they advise against collecting a bunch of data and just make up a story after data collection and report the significant findings. In psychology, which is my field, recently some journals accept pre-registration of research proposals and those journals allow those studies to be published regardless of the results (see http://www.sciencemag.org/careers/2015/12/register-your-study-new-publication-option)

Yes, I understand people's conceptualisation of how science is done is that:

  1. I have some competing (maybe mutually exclusive) hypotheses
  2. I design a study (maybe an experiment) and collect data
  3. I look at the data and see if it supports my hypotheses
  4. I publish my findings

However, is there really a problem of some people's "post hoc" way of doing research? If there is a hidden pattern in reality and researchers couldn't really have a grasp of it at the moment, what's wrong with doing a exploratory study and collect a bunch of data and examine their relationships in a post hoc way? And why is making up a story to cover up a study a problem?

JetLag
  • 1,105
  • 2
  • 9
  • 22
  • Thank you @Firebug for raising. However, I would prefer a more detailed and technical answer. This potential problem has been especially prominent in recent years. I hope arouse a fiercer discussion. – JetLag Oct 10 '17 at 13:09
  • The [Simpon's paradox](https://en.wikipedia.org/wiki/Simpson%27s_paradox) points to -yet- another potential and serious issue with post-hoc analysis. – leonbloy Oct 10 '17 at 20:39
  • 1
    This is fun: http://www.tylervigen.com/spurious-correlations – John Coleman Oct 11 '17 at 01:35
  • 1
    The jelly-bean [xkcd](https://xkcd.com/882/) cartoon might be appropriate here – Michael M Oct 11 '17 at 06:35

3 Answers3

15

Obviously post-hoc analysis and explanatory studies provide useful pieces to understand and discuss phenomena : There is nothing wrong with them and with publishing them. The moment where such designs become a bad practice is when they are used for conclusions that only a properly dedicated study could support. Care must be taken when interpreting post-hoc analysis, for the data analyst as well as for the person that will interpret the reported result.

Actually in most cases, post hoc explorations of the data fully immerse us in "The garden of forking paths" where it is almost impossible to conduct proper hypothesis testing (at least without increasing drastically type II error) as a large number of choices are then conditioned by the data (e.g. which variables to exclude, how to group data, what relationship to test?). Indeed, in large data set, you can look at the data in a certain way (here I mean with the good grouping, thresholds, fitting function ..) so that a striking pattern appears. A pattern in data does not mean a pattern in the phenomum and only a proper replication can justify a conclusion (as far as a hypothesis test can justify a conclusion...).

peuhp
  • 4,622
  • 20
  • 38
  • I understand that data analysts might have some preconceptions or some hypothesis that they really want to shown others when analysing a big dataset. However, this might not be a strong argument? Observers influence what is observed. It is true regardless of whether they have a large dataset or not. Moreover, if there really is a pattern lying in the dataset, publishing it should not be a problem? However, I suspect that the real problem might lie in the inflated Type 2 error rate? – JetLag Oct 10 '17 at 13:04
  • @Steven This is not related to the size of the dataset but to how it is dealt. The fact is that there are patterns in large datasets and this is the problem. As long as you work on large data set, you can look at the data in a certain way (here I mean with the good grouping, thresholds, fitting function ..) so that a striking pattern appears (even if it means nothing; play with random data frame to convince you). A pattern in data does not mean a pattern in the phenomum. Hope it answer your comment. – peuhp Oct 10 '17 at 13:17
  • Thank you for your explanation. I understand what you mean. Just a clarification: by "The fact is that there are patterns in large datasets and this is the problem", are you referring to a Real pattern or a False Positive? – JetLag Oct 10 '17 at 13:35
  • @Steven False positive. By the way, I guess that this is also true for real patterns, the fact is that we cannot distinguish them. – peuhp Oct 10 '17 at 13:37
  • 2
    Thanks. I agree. If I could combine @Richard Redding's answer with yours, I'm now convinced that while this kind of post hoc study could be exploratory in nature, those results must subject to further formal hypothesis testing procedures to test their validity. Thank you all. – JetLag Oct 10 '17 at 13:41
  • @Steven Nice summary ! – peuhp Oct 10 '17 at 13:43
2

Making up a story would be dangerous but, to your general point, it depends. It obviously complicates any statistical methods you use but in many cases it would be wasteful to not contribute new credible information. I would expect your finding to be treated with greater skepticism but few things are black and white.

Richard Redding
  • 639
  • 5
  • 11
  • Particularly if you work in a field where data sets have to be assembled with great care and difficulty (e.g. waves of election surveys in the U.S.) it would be wasteful not to do more analysis. – zbicyclist Oct 10 '17 at 13:58
1

There's a formula called "Bayes' Theorem" that says that if you start out assigning probability P1 to a hypothesis H, and you see evidence E, then you should adjust your probability to:

P2 = P1*(probability of seeing E given that H is true)/(probability of seeing E).

So if something would normally be very unlikely to see, but probable if the hypothesis is true, then seeing it should significantly increase your confidence in the hypothesis. However, if something is likely to be seen regardless of whether the hypothesis is true, then it should not increase your confidence.

Statistical analysis can tell you what the probability of a particular study arriving at a result given the null hypothesis is, but that is not the same as the probability of it being seen. Unfortunately, the first number is erroneously treated as if it were the second.

This difference is the basis of the Monty Hall paradox: if you pick Door A and are shown that Door B has a goat, then the evidence "Door B has a goat" is equally likely regardless of whether Door A has a car, so it shouldn't cause you to switch. However, "I saw that Door B has a goat" is less likely if Door A has a car, because in that case Monty Hall has only a 50% chance of showing you Door B. Therefore, while knowing that Door B has a goat shouldn't cause you to switch, knowing that you know that Door B has a goat should.

That is, if Monty Hall always shows you Door B, regardless of whether it has a goat, then seeing it's a goat shouldn't make you switch. But if Monty Hall never shows you Door B when it has a car, always shows you Door B when Door C has a goat, and randomly chooses between Door B and Door C when they both have a goat, then seeing that Door B has a goat should make you switch.

Similarly, if someone shows you the result of a study, and you can confidently say "I would have seen this statistic regardless of what it had been", then you can take the naive probability calculations at face value. But if you see a statistic, and you realize that this statistic probably wouldn't have been mentioned if it weren't so impressive, then now you have to adjust for that bias.

So if you have a rigorous, predetermined procedure by which you become aware of results, and the probability of seeing a result is not dependent on the results of that study, then you don't have to worry about this distinction between "Probability of E" and "Probability of knowing E". But once the probabilities diverge, now you have an extra parameter that you have to estimate, and you likely will have only a vague idea of what this parameter should be, and it will be very tempting to simply ignore this issue.

Acccumulation
  • 3,688
  • 5
  • 11