0

I have taken introductory level statistics, but all I'm certain of is that we reject a null hypothesis when the p-value is to "too low" (<0.05) and fail to reject the null hypothesis if p is >0.05 (as an example, if alpha is 0.05).

However, I don't understand what the p-value means or implies. I've heard something like: Given a confidence interval of 95%, for example, a p-value of 0.04 says that if the experiment were ran again "many times", the alternative hypothesis would be true 96% of the time and the null would be true 4% of the time, however this would be purely due to "random chance", and therefore is an acceptable level of possibly rejecting a null that maybe true. The phrases in quotations are the statements that confuse me the most. Don't we only run an experiment once (usually)? And what is another way of saying "random chance"? Is R , by default, computing p-values with methods like bootstrapping?

Sorry in advance if I'm mixing-up different concepts!

Nate
  • 595
  • 1
  • 11
  • Not quite, but thank you. I get stuck here: "if we were to repeat our experiment many, many times". What does the p-value mean mean if we don't repeat our experiment? – Nate Oct 06 '21 at 03:43
  • Confidence intervals $\ne$ hypothesis tests (where $p$ values come in). The former estimates a **plausible ranges of values** to infer about a population's statistic of interest, and is where the "If we were to repeat our experiment many, many times…" comes in. The latter **makes a decision about** evidence for the equality or inequality of a population's statistic, and is where the linked answers Socratic dialog opens (e.g., "the probability of observing a value of the *sample* test statistic, assuming the null hypothesis is true"). – Alexis Oct 06 '21 at 03:49
  • Confidence intervals, got it. But, still unclear about the "repeating an experiment" and p-values :( – Nate Oct 06 '21 at 04:06

1 Answers1

0

I won't address the titular questions, those are well answered by the post Alexis links to. I will try to address the question you make in the comments.

The statement about repeated experiments and the p value are in fact the same. Probability statements are really just statements about the long term relative frequency of events happening or not happening. When I say "A fair coin lands on heads 50% of the time" that is the same as saying "If I were to repeatedly flip a fair coin over and over and over, then roughly half of those flips would be a heads".

The p value is just a statement about repeated experiments were the null to be true. A p value of 0.01 means "If the null were really true, and were I to repeat this experiment under ideal conditions, and assuming all other assumptions about the data generating process are appropriate, then 1% of those experiments would yield results more extreme than what I have seen here today". We don't actually need to repeat experiments to determine the probability of seeing results more extreme than what we say from our experiment (again, assuming the null is the truth); that is what a p value tells us.

Demetri Pananos
  • 24,380
  • 1
  • 36
  • 94
  • Ok, I think that last part is was what I needed to hear. So, we’re saying IF the experiment was repeated - we’re claiming something like: no matter how many times this experiment is done again, the data can change a little with each new trail, but our 1st trial IS already representative of our population, and any (less than 5%) weirdness that contradicts our answer would be purely due to random chance? – Nate Oct 06 '21 at 12:33
  • @Nate, no that is not what we're saying. We're saying that data can change from experiment to experiment. If we repeated this experiment, then the probability of getting something more extreme is $p$. Nothing about representatives. Nothing about weirdness. It is literally a claim about the long term relative frequency of seeing results more extreme than the ones we obtained. – Demetri Pananos Oct 06 '21 at 13:30
  • Got it, thank you. – Nate Oct 06 '21 at 13:33
  • @Alexis It says something about repeated experiments *conditioned on the null being true*. It says nothing about actual repeated experiments. – Demetri Pananos Oct 06 '21 at 15:54
  • @Alexis If you have problems with a statement made by OP that's fine but I feel I've been very clear in my definition of the p value: Under the assumption that the null is true and assuming our other assumptions about the data are correct, the p value is the probability we would observe a result more extreme than the one we observed. Translating that into a statement about long term relative frequencies would mean talking about repeated experiments. Nowhere have I mentioned bernoulli distributions, so I'm not sure why you're addressing me. – Demetri Pananos Oct 06 '21 at 16:02