Can I remove significant interaction variable in regression model?

Question

I have a regression model, which consists of 4 predictors and 1 interaction - i dont know what are the correct symbols and terms, so I will drop my model here:

Y=B0+B1x1+B2X2+B3X3+B4X4+B5X3X4 

So X4 is insignificant ( p-value is 0.90), while interaction is significant (p-value is 0.00000 ...)

The task asked me to analyse the model and reduce if possible , I have reduced it already, so it is my 2nd model in model building.I am not sure how it works? Can I remove the interaction and see if the variable is still insignificant and remove it as well?

I am non-stats. major, but stats. is part of my degree, so keep terms simple.

Answer 1

You have to be very careful when interpreting the p-values reported for coefficients in multiple regression models that include interaction terms.

You don't specify which software you are using, so let's examine what happens with R under its default settings. With the X3:X4 interaction term included in the model, the coefficients B3 and B4 would represent the coefficients for X3 and X4 when the other of that pair is at its reference level (categorical predictor) or at 0 (continuous predictor). With a highly significant interaction term, it's quite possible that the coefficient B4 (with X3 at its reference level) would not be significantly different from 0 even if the X4 variable is very important when X3 is at some other level. And with the significant interaction, you can't properly interpret the effect of X3 unless you know the value of X4 and its associated coefficients. So you don't want to be omitting the B4 coefficient for X4 or the B5 coefficient for the X3:X4 interaction from your model. Predictions based on values of X3 and X4 will need both the direct and the interaction coefficients to be reliable.

One hint to consider: if either of X3 or X4 is a continuous variable, try centering its values around the mean or median before you perform a regression with the interaction term. Although this doesn't directly change predictions that would be made for any combination of predictor values, or the statistical significance that would be reported by analysis of variance that takes interactions into account, it can change the reported values of the individual coefficients and the individual-coefficient p-values in a way that better represents your intuition if the typical values of the continuous variable are far from the reference value of 0. See this page for some discussion.

Finally, do ask why you are being tasked to reduce your model. For predictive work that is typically counterproductive, and it doesn't really help with inference.

Answer 2

There are two good choices for you. The first, and it is the lesser of the two, is step-wise regression. It will go through your model, gradually altering the model, looking for the best model as measured by some criterion such as the AIC or BIC. The second is better for theoretical reasons, amongst others. It is to solve a combinatoric solution.

You could construct a sequence of do loops or for-next loops that includes or excludes all possible combinations of variables. This is doing the same thing as step-wise, but you are covering every combination. Why this is of added value is that if there is a theoretical reason for some variable to be in the model, then you can force it to be in your model. Instead of using a loop to include it or exclude it, it must be there.

You should then calculate either the AIC or the BIC. The model with the smallest value of the information criterion you chose would win. They differ in how they penalize model complexity. With small sample sizes, the AIC will tend to overfit.

What does matter with the AIC or the BIC is the relative ranking of the set of models. If you had different data sets and different problems, an AIC of -23 for one problem couldn't be compared to an AIC of -22 for another. However, with the same data set facing the same problem, a -23 is a better model than a -22.

The difference in calculation between the two methods is how they penalize added model structure. The AIC adds a penalty of $2k$ to its value, where $k$ is the number of independent variables. A model with two independent variables has a penalty of 4, while one with three has a penalty of six. The BIC adds a penalty of $log(n)k$, where $n$ is the number of observations in the sample. As a rule of thumb, their results will be highly concordant.

A way to think about the trade-off between the AIC and the BIC is in what they are trying to accomplish. The BIC grants equal prior weight to each model, while the AIC grants lower prior weight to models with more parameters. The AIC does this by directly penalizing structure by having a fixed penalty for $k$. The BIC is making a tradeoff between sample size and structure.

If you have a small sample, it may improve the predictive power of the model to add a variable as you are increasing the amount of natural variation for the model to work with. On the other hand, once the sample size starts to become large, two variables that covary may be providing mostly the same information. It can cause the model to be a worse model because the effect of the added information from an added variable is being offset by the effect of colinearity to the point that it turns into noise.

In any case, the reason to use stepwise is that it is almost certainly a built-in function of your software. The reason to be combinatoric is that it is better. In any case, do not use p-values as a criterion even though they will be correlated with the AIC and BIC.