Is this just a consequence of Publication bias or has a name of its own?

Question

Scenario:

In our world, only statistically significant ($p<0.05$) results are published, everything else is rejected or not even handed in.

Let's now assume an experiment is $100$ times conducted and in truth $H_0$ holds. Since the $p$-value is uniformly distributed under $H_0$ (let's assume all conditions are satisfied) this would mean that on average $5$-paper could be handed in which all claim that they have found a statistically significant effect ($p<0.05$) and the academic world might actually believe in the alternative hypothesis $H_1$ although $H_0$ is actually true (of course no one would actually know that).

Question: Is this reasoning correct? I wonder how this reporting bias is called, it's different from the publication bias (or just a consequence and that's why it has name of its own?).

Answer 1

This scenario seems like a straightforward example of publication bias. The definition at the linked Wikpedia article is a good one: publication bias "occurs when the outcome of an experiment or research study influences the decision whether to publish or otherwise distribute it." In your scenario, the 95 experiments that failed to reject the null hypothesis are not published whereas the 5 that succeeded are.

Answer 2

This effect is a form of publication bias, but also an effect of wholesale adoption of significance at P $\leq$ 0.05, ignoring effect size, and the crisis of reproducibility in science.

I add the alpha level of 0.05, as this level is not particularly stringent for the reason the OP states - 5/100 studies will be statistically significant by chance alone.

Effect size is often ignored which elevates trivial effects when p-values are small and may ignore important effects when p-values are large. Read here for more on reproducibility:

Baker, Monya. "Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the 'crisis rocking science and what they think will help." Nature, vol. 533, no. 7604, 2016, p. 452+. Gale OneFile: Health and Medicine, Accessed 13 Jan. 2020.

In aggregate, science doesn’t publish all work equally, abuses p-values at the cost of effect size and doesn’t/can not verify the direction or exent of biases created.