How to interpret non-significant interaction term in linear regression?

Question

I have difficulties interpreting an interaction term in a linear regression with regards to my hypothesis.

Consider this basic example:

H0: Better school grades lead to higher income and this relation is the same for men and for women.

DV: income

IV: school grade, sex

How do I interpret a non-significant, positive interaction term grade * sex? Would it be correct to accept H0?

Answer 1

"Accepting H$_{0}$" is always a logical fallacy (i.e. lack of significance is always "failed to reject"). Interpretively, this means you did not find evidence of the interaction grade*sex.

The reason why you can only state that you did not find evidence of X with tests for difference is that these tests only provide evidence of how likely you are to see $\hat{\beta}_{\text{grade}\times\text{sex}}$ if H$_{0}$ is true, and your test only yields your desired power to reject for at least as large as one not all (smaller) possible values under H$_{\text{A}}$.

If you want to state that you found or did not find evidence of an absence of X, then you need to use, for example, tests for equivalence (say, using two one-sided tests) where H$_{0}$ no longer takes the form H$_{0}^{+}\text{: }\theta=0$, but rather takes the form H$_{0}^{-}\text{: }|\theta|=\Delta$, where $\Delta$ is a researcher-specified value meaning "too small a difference to care about". (the '$+$' and '$-$' in the superscript indicate null hypotheses for difference and for equivalence, respectively.)

To perform an equivalence test on grade*sex (i.e. you want to provide evidence that there is no interaction), you will need a few things:

• $\theta$: the effect you are estimating for grade*sex (i.e. the coefficient $\hat{\beta}_{\text{grade}\times\text{sex}}$)
• $\Delta$: an effect size that is too small to care about (e.g. we do not care about $-0.1 \leq \beta_{\text{grade}\times\text{sex}} \leq 0.1$. A $\Delta=0.1$ is not magical, and I only use it here as a imaginary value of $\Delta$, you need to decide).
• $s_{\theta}$ the standard error of your estimate (i.e. the standard error of $\hat{\beta}_{\text{grade}\times\text{sex}}$)

Given that, then:

H$_{0}^{-}\text{: }|\beta_{\text{grade}\times\text{sex}}| \ge \Delta$, which gives two one-sided null hypotheses:

H$_{01}\text{: }\beta_{\text{grade}\times\text{sex}} \ge \Delta$, and
H$_{02}\text{: }\beta_{\text{grade}\times\text{sex}} \le -\Delta$

The test statistics corresponding to both of these are:

$$t_{1} = \frac{\Delta - \hat{\beta}_{\text{grade}\times\text{sex}}}{s_{\hat{\beta}_{\text{grade}\times\text{sex}}}}$$ $$t_{2} = \frac{\hat{\beta}_{\text{grade}\times\text{sex}}+ \Delta}{s_{\hat{\beta}_{\text{grade}\times\text{sex}}}}$$

These are both right-side/upper tail tests, so you get the p-values:

$p_{1}=\text{P}\left(T_{df} \ge t_{1}\right)$, and
$p_{2}=\text{P}\left(T_{df} \ge t_{2}\right)$

If both H$_{01}^{-}$ and H$_{02}^{-}$ are rejected with $p\le\alpha$ (not $p \le \alpha/2$), then, taken together with the failure to reject H$_{0}^{+}$ you can conclude you found evidence that the grade*sex is equivalent to zero, given $\alpha$ and $\Delta$.

However, if you reject only one or reject neither of H$_{01}^{-}$ and H$_{02}^{-}$, then, taken together with the failure to reject H$_{0}^{+}$ you can't conclude anything: your results are indeterminate because your data are underpowered.