2

There are many similar questions posted on this website but also a wide variety of conflicting answers, so I'm still unsure of the best way to proceed.

I have several independent but similar datasets and completed analyses, which compare multiple treatments on subjects over time. In many of these analyses, some treatments are significant while interactions of treatment*time are not. The interactions are meaningful and not trivial, so I'm apprehensive to simply remove them from my models as some answers suggest.

A typical formula (in R) for me is the following: count ~ treatment * time + block , where blocks are randomized complete block designs and almost always succeed in explaining excessive variation (and are therefore significant).

I generally will produce full models, reduced models, and a null model, and compare them by AIC to select the most parsimonious model for analysis. The full models almost always score the best, and are used in the analyses.

My understanding is that including a non-significant but non-trivial interaction of treatment*time will result in having conditional treatment effects. I've seen some answers recommend replacing these interactions with an additive model instead.

So my main questions are:

  1. Is it improper to report conditional treatment effects absent of significant interaction?
  2. Is it a good option to proceed with a fully additive model when the interaction is not significant?

Any other insights are also appreciated. It might be worth noting that my datasets are always zero-inflated and overdispersed Poisson distributions, so I generally use negative binomial hurdle-GLM's (pscl) and non-parametric longitudinal analyses (nparLD) packages for analysis.

Dieter Kahl
  • 119
  • 6

1 Answers1

1

I think that this is a decision only you can make. You say:

I generally will produce full models, reduced models, and a null model, and compare them by AIC to select the most parsimonious model for analysis. The full models almost always score the best, and are used in the analyses.

But that can produce a different model than you would choose based on standard statistical significance based on $\alpha < 0.05$. As Frank Harrell notes:

If using $\chi^2$ tests, AIC uses a cutoff of $\chi^2 =2.0$ which corresponds to $\alpha=0.157$.

So you already have potentially competing criteria within your own analysis.

For prediction, keeping "insignificant" interaction terms is generally best unless the model is overfit. You might consider acknowledging this situation and presenting results for both models, along with some discussion to help the reader think about the implications.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • What about purely for significance testing and not prediction? – Dieter Kahl Aug 20 '20 at 00:30
  • @DieterKahl the traditional cutoff for “significance testing” is p < 0.05, not an AIC criterion. See [this thread](https://stats.stackexchange.com/q/79289/28500) for extensive discussion of the limitations of such all-or-none declarations of “significance.” – EdM Aug 20 '20 at 03:46
  • I appreciate the feedback, and that thread is quite educational. Can you recommend an alternative approach to feature selection for non-parametric, non-linear models, that uses the same significance criteria of α = 0.05? – Dieter Kahl Aug 20 '20 at 07:48
  • @DieterKahl that's a pretty open-ended question. Much depends on what you mean by "non-parametric, non-linear models" and the type of modeling that is being done. For nested models that are fit by maximum-likelihood methods, chi-square tests based on deviance are a way. If there's a specific type of situation you have in mind, post a new question on this site. It's easier for people to use this site if each question is relatively specific. – EdM Aug 20 '20 at 12:35