6

In testing a null vs an alternative, if we don't reject the null (e.g. the p-value is very big, bigger than the significance level), what is our conclusion?

Can we say that we accept the null, can we? I remember I heard that not rejecting the null doesn't mean accepting the null, but I might be wrong.

Or should we simply say that we don't reject the null?

Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
Tim
  • 1
  • 29
  • 102
  • 189

2 Answers2

13

"We fail to reject the null" is the correct answer. Rather than say, for example, "there is no difference" we should write "no difference was detected".

Clearly, with not enough replicates or large measurement error you are likely to not be able to detect even large effects. So maybe the difference is there, but you failed to see it?

As Maarten Buis writes in the comment: "Absence of evidence is not evidence of absence". (Personally, I am careful with the word "evidence".)

January
  • 6,999
  • 1
  • 32
  • 55
  • 1
    I like the phrase "Absence of evidence is not evidence of absence" in this respect. Failing to reject would be a case of absence of evidence. Many null hypotheses are of the form "there is no effect", so accepting such a hypothesis would be a case of finding evidence of absence (of a effect). – Maarten Buis Mar 12 '14 at 14:12
  • I like this formulation as well, but I did not want to start a discussion on whether a large p-value is not *a kind of* evidence in favor of the null (as in, posterior probability of the null increased; as in, Bayes factor). – January Mar 12 '14 at 14:51
3

I would include three things:

  1. The phrase "Insufficient evidence to reject". Shows that with more evidence, e.g. more data, or repeating the experiment with a different random selection of data, you might have rejected.

  2. The significance level. At a higher significance level you might have rejected the null hypothesis.

  3. What your conclusion is. "The data is consistent with" is a good phrase here.

As an example, putting these altogether for an experiment to determine the effect of cognitive behavioural therapy (CBT) on addressing insomnia

"There was insufficient evidence to reject the null hypothesis that CBT increased the amount of sleep at the 5% significance level. The data is consistent with CBT having no effect on insomnia."

TooTone
  • 3,631
  • 24
  • 33
  • 1
    I would also include the study power when failing to reject the null. "There is insufficient evidence to reject the null hypothesis at the 5% significance level. The study's power to detect a 1% difference in means was approximately 90%." is an entirely different statement from "There is insufficient evidence to reject the null hypothesis at the 5% significance level. The study's power to detect a 1% difference in means was approximately 15%." – P Schnell Mar 12 '14 at 16:05
  • 1
    @PSchnell Yes -- the significance level is the chance of rejecting the null hypothesis when you shouldn't (when $H_0$ true), and it could also be worth reporting the power, the chance of rejecting the null hypothesis when you should (when $H_0$ false). However I've done significance tests when I haven't calculated the power, so I wouldn't have been able to report it... Sometimes you just do as many replicates as time/money allows and you only get to choose the significance level. – TooTone Mar 12 '14 at 16:36
  • Pedant mode: "data are consistent". Carry on. :) – Almo Mar 12 '14 at 17:40
  • 1
    @Almo I love this site:) – TooTone Mar 12 '14 at 17:41