Are smaller p-values more convincing?

Question

I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I apologise for the wall of text, but I felt it was necessary to provide an overview of my current understanding of these concepts, before I moved on to my actual questions.

---

From what I've gathered, a $p$-value is simply a measure of surprise, the probability of obtaining a result at least as extreme, given that the null hypothesis is true. Fisher originally intended for it to be a continuous measure.

In the Neyman-Pearson framework, you select a significance level in advance and use this as an (arbitrary) cut-off point. The significance level is equal to the type 1 error rate. It is defined by the long run frequency, i.e. if you were to repeat an experiment 1000 times and the null hypothesis is true, about 50 of those experiments would result in a significant effect, due to the sampling variability. By choosing a significance level, we are guarding ourselves against these false positives with a certain probability. $P$-values traditionally do not appear in this framework.

If we find a $p$-value of 0.01 this does not mean that the type 1 error rate is 0.01, the type 1 error is stated a priori. I believe this is one of the major arguments in the Fisher vs N-P debate, because $p$-values are often reported as 0.05*, 0.01**, 0.001***. This could mislead people into saying that the effect is significant at a certain $p$-value, instead of at a certain significance value.

I also realise that the $p$-value is a function of the sample size. Therefore, it cannot be used as an absolute measurement. A small $p$-value could point to a small, non-relevant effect in a large sample experiment. To counter this, it is important to perform an power/effect size calculation when determining the sample size for your experiment. $P$-values tell us whether there is an effect, not how large it is. See Sullivan 2012.

My question: How can I reconcile the facts that the $p$-value is a measure of surprise (smaller = more convincing) while at the same time it cannot be viewed as an absolute measurement?

What I am confused about, is the following: can we be more confident in a small $p$-value than a large one? In the Fisherian sense, I would say yes, we are more surprised. In the N-P framework, choosing a smaller significance level would imply we are guarding ourselves more strongly against false positives.

But on the other hand, $p$-values are dependent on sample size. They are not an absolute measure. Thus we cannot simply say 0.001593 is more significant than 0.0439. Yet this what would be implied in Fisher's framework: we would be more surprised to such an extreme value. There's even discussion about the term highly significant being a misnomer: Is it wrong to refer to results as being "highly significant"?

I've heard that $p$-values in some fields of science are only considered important when they are smaller than 0.0001, whereas in other fields values around 0.01 are already considered highly significant.

Related questions:

• Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?

• When to use Fisher and Neyman-Pearson framework?

• Is the exact value of a 'p-value' meaningless?

• Frequentist properties of p-values in relation to type I error

• Confidence intervals vs P-values for two means

• Why are lower p-values not more evidence against the null? Arguments from Johansson 2011 (as provided by @amoeba)

Answer 1

Are smaller $p$-values "more convincing"? Yes, of course they are.

In the Fisher framework, $p$-value is a quantification of the amount of evidence against the null hypothesis. The evidence can be more or less convincing; the smaller the $p$-value, the more convincing it is. Note that in any given experiment with fixed sample size $n$, the $p$-value is monotonically related to the effect size, as @Scortchi nicely points out in his answer (+1). So smaller $p$-values correspond to larger effect sizes; of course they are more convincing!

In the Neyman-Pearson framework, the goal is to obtain a binary decision: either the evidence is "significant" or it is not. By choosing the threshold $\alpha$, we guarantee that we will not have more than $\alpha$ false positives. Note that different people can have different $\alpha$ in mind when looking at the same data; perhaps when I read a paper from a field that I am skeptical about, I would not personally consider as "significant" results with e.g. $p=0.03$ even though the authors do call them significant. My personal $\alpha$ might be set to $0.001$ or something. Obviously the lower the reported $p$-value, the more skeptical readers it will be able to convince! Hence, again, lower $p$-values are more convincing.

The currently standard practice is to combine Fisher and Neyman-Pearson approaches: if $p<\alpha$, then the results are called "significant" and the $p$-value is [exactly or approximately] reported and used as a measure of convincingness (by marking it with stars, using expressions as "highly significant", etc.); if $p>\alpha$ , then the results are called "not significant" and that's it.

This is usually referred to as a "hybrid approach", and indeed it is hybrid. Some people argue that this hybrid is incoherent; I tend to disagree. Why would it be invalid to do two valid things at the same time?

Further reading:

• Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"? -- my question about the "hybrid". It generated some discussion, but I am still not satisfied with any of the answers, and plan to get back to that thread at some point.

• Is it wrong to refer to results as being "highly significant"? -- see my yesterday's answer, which is essentially saying: it isn't wrong (but perhaps a bit sloppy).

• Why are lower p-values not more evidence against the null? Arguments from Johansson 2011 -- an example of an anti-Fisher paper arguing that $p$-values do not provide evidence against the null; the top answer by @Momo does a good job in debunking the arguments. My answer to the title question is: But of course they are.

Answer 2

I don't know what's meant by smaller p-values being "better", or by us being "more confident in" them. But regarding p-values as a measure of how surprised we should be by the data, if we believed the null hypothesis, seems reasonable enough; the p-value is a monotonic function of the test statistic you've chosen to measure discrepancy with the null hypothesis in a direction you're interested in, calibrating it with respect to its properties under a relevant procedure of sampling from a population or random assignment of experimental treatments. "Significance" has become a technical term to refer to p-values' being either above or below some specified value; thus even those with no interest in specifying significance levels & accepting or rejecting hypotheses tend to avoid phrases such as "highly significant"—mere adherence to convention.

Regarding the dependence of p-values on sample size & effect size, perhaps some confusion arises because e.g. it might seem that 474 heads out of 1000 tosses should be less surprising than 2 out of 10 to someone who thinks the coin is fair—after all the sample proportion only deviates a little from 50% in the former case—yet the p-values are about the same. But true or false don't admit of degrees; the p-value's doing what's asked of it: often confidence intervals for a parameter are really what's wanted to assess how precisely an effect's been measured, & the practical or theoretical importance of its estimated magnitude.

Answer 3

Thank you for the comments and suggested readings. I've had some more time to ponder on this problem and I believe I've managed to isolate my main sources of confusion.

• Initially I thought there was a dichotomy between viewing the p-value as a measure of surprise versus stating that it's not an absolute measure. Now I realise these statements don't necessarily contradict each other. The former allows us to be more or less confident in the extremeness (unlikeness even?) of an observed effect, compared to other hypothetical results of the same experiment. Whereas the latter only tells us that what might be considered a convincing p-value in one experiment, might not be impressive at all in another one, e.g. if the sample sizes differ.

• The fact that some fields of science utilise a different baseline of strong p-values, could either be a reflection of the difference in common sample sizes (astronomy, clinical, psychological experiments) and/or an attempt to convey effect size in a p-value. But the latter is an incorrect conflation of the two.

• Significance is a yes/no question based on the alpha that was chosen prior to the experiment. A p-value can therefore not be more significant than another one, since they are either smaller or larger than the chosen significance level. On the other hand, a smaller p-value will be more convincing than a larger one (for a similar sample size/identical experiment, as mentioned in my first point).

• Confidence intervals inherently convey the effect size, making them a nice choice to guard against the issues mentioned above.

Answer 4

The p-value cannot be a measure of surprise because it is only a measure of probability when the null is true. If the null is true then each possible value of p is equally likely. One cannot be surprised at any p-value prior to deciding to reject the null. Once one decides there is an effect then the p-value's meaning vanishes. One merely reports it as a link in a relatively weak inductive chain to justify the rejection, or not, of the null. But if it was rejected it actually no longer has any meaning.