Why does a confidence interval including 0 mean the difference is not significant?

Question

Could someone please explain why zero being inside or outside the confidence interval tells you whether or not the difference is significant?

Answer 1

If the confidence interval (with your chosen level of confidence) includes $0$, that implies you think $0$ is a reasonable possibility for the true value of the difference. In general, by 'significant' people usually mean that they no longer believe the null hypothesis ($0$) is a reasonable possibility. Note that if a $95\%$ CI doesn't include $0$, the $p$-value would be $<.05$, which is the conventional cutoff for 'significance'.

Answer 2

"Having zero in one's confidence interval implies that a treatment effect has no effect." : This is often how such confidence intervals are interpreted, but this is a mistake. A confidence interval that contains zero is not certainty that there is no treatment effect, but that it is uncertain whether there is a treatment effect. Having zero in one's confidence interval implies that a treatment effect could have a positive or negative effect on the outcome of interest. (DataCamp).

Answer 3

In the simple scenarios you're probably considering, it's a logical equivalence: the point defining the null hypothesis is inside a confidence interval with confidence level $\gamma=1-\alpha$ if and only if the observed value of the test statistic is outside the critical region of a test with size $\alpha$.

Consider the case of the usual $Z$ test. You have a random sample $X_1,\dots,X_n$ from a normal distribution with unknown mean $\mu$ and known variance $\sigma_0^2$. You want to test the null hypothesis $H_0:\mu=\mu_0$ against the alternative $H_A:\mu\ne\mu_0$. The test statistic is $Z=(\bar{X}-\mu_0)/(\sigma_0/\sqrt{n})$ and, under the null hypothesis $H_0$, $Z$ has a standard normal distribution. The critical region is $$ \mathscr{C}_\alpha = ( -\infty; -z_{1-\alpha/2}] \;\cup\; [z_{1-\alpha/2};\infty). $$ On the other hand, a confidence interval for $\mu$, with confidence level $\gamma=1-\alpha$, is given by $$ \text{CI}[\mu;\gamma = 1-\alpha] = \left( \bar{x} - z_{1-\alpha/2} \frac{\sigma_0}{\sqrt{n}} \;;\; \bar{x} + z_{1-\alpha/2} \frac{\sigma_0}{\sqrt{n}} \right). $$ It follows by simple algebra that $$ \mu_0\in \text{CI}[\mu;\gamma = 1-\alpha] \;\;\Leftrightarrow\;\; z_{\text{obs}}=(\bar{x}-\mu_0)/(\sigma_0/\sqrt{n})\notin\mathscr{C}_\alpha. $$ The connection with the usual definition of a $p$-value for this problem is also immediate.

Answer 4

This has to be explained in the context of interpreting the significance of parameters in building let's say linear regression. Null hypothesis of linear regression says that the predictor coefficients are 0, alternate hypothesis is coefficients are non-zero.Confidence interval tells you the actual coefficient value can lie within that range. If that interval includes 0, that means the actual coefficient value can be zero and that means that the predictor has no relationship with the response variable or it is insignificant in terms of its influence on response variable. Hope that explains.