Holm-Bonferroni correction with two interactions

Question

I try to get a sense of the correct adjusted Holm-Bonferroni significance levels $\alpha$ when dealing with multiple interaction terms. To simplify, assume one has the following model: $$ y=\beta_0 + \beta_1x_1 + \beta_2x_2+\beta_3x_3+\beta_4x_1x_3+\beta_5x_2x_3 + \varepsilon $$

I would publish any of the following statistical findings (Theories T):

T1a: $\beta_1\ne0$ and $\beta_4=0$

T1b: $\beta_1\ne0$ and $\beta_4\ne0$

T2a: $\beta_2\ne0$ and $\beta_5=0$

T2b: $\beta_2\ne0$ and $\beta_5\ne0$

The crucial point is that I would not publish $\beta_1=0$ and $\beta_4\ne0$ and would also not publish $\beta_2=0$ and $\beta_5\ne0$. Thus, I test $\beta_4$ and $\beta_5$ only conditional on $\beta_1$ and $\beta_2$, respectively, being significant.

Let's assume now, we have the following p-values: 0.01 for $\beta_1$, 0.02 for $\beta_2$, 0.04 for $\beta_4$ and 0.05 for $\beta_5$. We test for the overall significance level $\alpha=0.05$.

Using the Holm-Bonferroni approach, I would assume that the significance level for $\beta_1$ is $\alpha/2=0.25$. That is because the interaction terms $\beta_4$ and $\beta_5$ are never tested before $\beta_1$ and $\beta_2$, respectively, are significant. For $\beta_2$, I would put the significance level at $\alpha/2=0.25$. That is because I am not testing $\beta_5$ before $\beta_2$ and we already dealt with $\beta_1$. That means at this point, I only considered $\beta_2$ and $\beta_4$. For $\beta_4$, I would put the significance level at $\alpha/2=0.25$. That is because I already dealt with $\beta_1$ and $\beta_2$ - that is now identical to the standard Holm-Bonferroni test. Lastly for $\beta_5$, the significance level is $\alpha/1=0.05$ as usual.

Thus in this example, I would accept T1 and T3 and reject T3 and T4 using the Holm-Bonferroni correction. Is my dealing with the interaction effects $\beta_4$ and $\beta_5$ correct?

NOTE 1: In my actual problem, I have two separate models $y=\beta_0 + \beta_1x_1 +\beta_3x_3+\beta_4x_1x_3 + \varepsilon $ and $y=\beta_0 + \beta_2x_2+\beta_3x_3+\beta_5x_2x_3 + \varepsilon $ fitted with two different datasets. But I don't think this makes any difference in this case.

NOTE 2: The question How to apply Bonferroni correction when including an interaction term? has some similarities to this one, but I am not sure whether the answers also apply to my question.

Answer 1

So you have the following situation:

Test model 1 on data 1, if significant -> additionally test model 3 on data 1
Test model 2 on data 2, if significant -> additionally test model 4 on data 2

Because these two approaches use different data sets these are two different processes.
The answer will also depend on your objective / primary hypotheses.

If you decide to test both models (say model 1 and 3) regardless of results of model 1, then you would adjust all the p-values from both models together.
If, however, you decide to conduct additional tests based on previous results (if they are significant), this is generally what people call "exploratory analysis" and different rules apply here. You will find different answers here, but in general, you would not adjust the p-values from the second model, since these results would be only "exploratory" and not "confirmatory", but you would still adjust the p-values from the first model.

Also, I am not sure what you mean by $\alpha/2$, if you are using Bonferroni then you divide by $n$ the number of conducted tests not the number of models.