Can someone please offer a nice succinct explanation why it is not a good idea to teach students that a p-value is the prob(their findings are due to [random] chance). My understanding is that a p-value is the prob(getting more extreme data | null hypothesis is true).
My real interest is what is the harm of telling them it is the former (aside from the fact it is simply not so).
I have a different interpretation of the meaning of the wrong statement than @Karl does. I think that it is a statement about the data, rather than about the null. I understand it as asking for the probability of getting your estimate due to chance. I don't know what that means---it's not a well-specified claim.
But I do understand what is likely meant by the probability of getting my estimate by chance given that the true estimate is equal to a particular value. For example, I can understand what it means to get a very large difference in average heights between men and women given that their average heights are actually the same. That's well specified. And that is what the p-value gives. What's missing in the wrong statement is the condition that the null is true.
Now, we might object that this isn't statement perfect (the chance of getting an exact value for an estimator is 0, for example). But it's far better than the way that most would interpret a p-value.
The key point that I say over and over again when I teach hypothesis testing is "Step one is to assume that the null hypothesis is true. Everything is calculated given this assumption." If people remember that, that's pretty good.
I've seen this interpretation a lot (perhaps more often than the correct one). I interpret "their findings are due to [random] chance" as "$\text{H}_0$ is true", and so really what they are saying is $\Pr(\text{H}_0)$ [which actually should be $\Pr(\text{H}_0 | \text{data})$; say, "given what we have seen (the data), what is the probability that only chance is operating?"] This can be a meaningful statement (if you are willing to assign priors and do Bayes), but it is not the p-value.
$\Pr(\text{H}_0 | \text{data})$ can be quite different than the p-value, and so to interpret a p-value in that way can be seriously misleading.
The simplest illustration: say the prior, $\Pr(H_0)$ is quite small, but one has rather little data, and so the p-value is largish (say, 0.3), but the posterior, $\Pr(\text{H}_0|\text{data})$, would still be quite small. [But maybe this example isn't so interesting.]
I'll add a late answer from the (ex) student perspective: IMHO the harm cannot be separated from its being wrong.
This type of wrong "didactic approximations/shortcut" can create a lot of confusion for students who realize that they cannot logically understand the statement, but assuming that what is taught to them is right they do not realize that they are not able to understand it because it is not right.
This does not affect students who just memorize rules presented to them. But it requires students who learn by understanding to be good enough to
I'm not saying that there aren't valid didactic shortcuts. But IMHO when such a shortcut is taken, this should be mentioned (e.g. as "for the ease of the argument, we assume/approximate that ...").
In this particular case, however, I think it is too misleading to be of any use.
Referring directly to the question: Where is the harm?
In my opinion, the answer to this question lies in the converse of the statement, "A p-value is the probability that the findings are due to random chance." If one believes this, then one also probably believes the following: "[1-(p-value)] is the probability that the findings are NOT due to random chance."
The harm then lies in the second statement, because, given the way most people's brains work, this statement grossly overestimates how confident we should be in the specific values of an estimated parameter.
Here is a simple example that I use:
Suppose our null hypothesis is that we are flipping a 2-headed coin (so prob(heads) = 1). Now we flip the coin one time and get heads, the p-values for this is 1, so does that mean that we have a 100% chance of having a 2-headed coin?
The tricky thing is that if we had flipped a tails then the p-value would have been 0 and the probability of having a 2-headed coin would have been 0, so they match in this case, but not the above. The p-value of 1 above just means that what we have observed is perfectly consistent with the hypothesis of a 2-headed coin, but it does not prove that the coin is 2-headed.
Further, if we are doing frequentist statistics then the null hypothesis is either True or False (we just don't know which) and making (frequentist) probability statements about the null hypothesis is meaningless. If you want to talk about the probability of the hypothesis, then do proper Bayesian statistics, use the Bayesian definition of probability, start with a prior and calculate the posterior probability that the hypothesis is true. Just don't confuse a p-value with a Bayesian posterior.
OK another, slightly different take on this:
A first basic problem is the phrase "due to [random] chance". The idea of unspecified 'chance' comes naturally to students but it is hazardous for thinking clearly about uncertainty and catastrophic for doing sensible statistics. With something like a sequence of coin flips it is easy to assume that 'chance' is described by the Binomial setup with a probability of 0.5. There is a certain naturalness to it for sure, but from a statistical point of view it's not more natural than assuming 0.6 or something else. And for other less 'obvious' examples, e.g. involving real parameters it's utterly unhelpful to think about what 'chance' would look like.
With respect to the question, the key idea is understanding what sort of 'chance' is described by H0, i.e. what actual likelihood/DGP H0 names. Once that concept is in place, students finally stop talking about things happening 'by chance', and start asking what H0 actually is. (They also figure out that things can be consistent with a rather wide variety of Hs so they get a head start on confidence intervals, via inverted tests).
The second problem is that if you're on the way to Fisher's definition of p-values, you should (imho) always explain it first in terms of the data's consistency with H0 because the point of p is to see that, not to interpret the tail area as some sort of 'chance' activity, (or frankly to interpret it at all). This is purely a matter of rhetorical emphasis, obviously, but it seems to help.
In short, the harm is that this way of describing things will not generalise to any non-trivial model they might subsequently try to think about. At worst it may just add to sense of mystery that the study of statistics already generates in the sorts of people such bowdlerised descriptions are aimed at.
If I take apart, "p-value is the probability that an effect is due to chance," it seems to be implying that the effect is caused by chance. But every effect is partially caused by chance. In a statistics lesson where one is explaining the need to try to see through random variability this is a pretty magical and overreaching statement. It imbues p-values with powers they do not have.
If you define chance in a specific case to be the null hypothesis then you're stating that the p-value yields the probability that the observed effect is caused by the null hypothesis. That seems awfully close to the correct statement but claiming that a condition on probability is the cause of that probability is again overreaching. The correct statement, that the p-value is the probability of the effect given the null hypothesis is true, does not ascribe cause to the null effect. The causes are various including the true effect, the variability around the effect, and random chance. The p-value doesn't measure the probability of any of those.
