Computing R-squared change, F-, and p-values for the interaction / moderation term

Question

I would like to compute R-squared change for the interaction/moderation term in a multiple regression model, along with the corresponding F- and p-values. Previously, I have worked with the modprobe macro by A. Hayes, which can produce this for SPSS. As I am transitioning to R now, I am trying to find a function/package or a custom-made script in R that does this. In case it helps, my current interaction model looks like this:

m1 <- lm(all_ART~Neuroticism*Agreeableness+Attentional.Control, 
         data=stp2_sub2, na.action=na.omit)

Any pointers on how to compute these values (i.e., $R^2$(interaction), F-value(interaction) and p-value(interaction)) for the interaction term in R would be much appreciated!

@gung Would an answer that provides formulas for $R^2$, $F$, and a $p-value$ be off-topic? (I suspect such an answer exists in some other thread--perhaps we could locate it and redirect this question there.) — whuber, Jun 30 '15 at 14:36
@whuber, if the OP were asking for that, it would be on topic. Note that, in my answer below, I link to an answer that provides one version of the formula. — gung - Reinstate Monica, Jun 30 '15 at 15:02
@ging It's not clear to me that asking for code itself is automatically off-topic. The paragraph at the on-topic [help/on-topic] includes "*if it needs statistical expertise to understand or answer, ask it here*". I know you've expressed a dislike of that phrase, but as I read it, that's the situation we presently work under. I'd prefer to avoid suggesting to people that the actual situation is quite so black and white as that first short sentence. (That's not to state a position on this particular post, only on the general principles involved.) — Glen_b, Jun 30 '15 at 23:53

score 3 · Answer 1 · edited Apr 13 '17 at 12:44

3

$R^2$ change and $F$-change (etc.) are just some of the many names for a nested model test. You can find some information on it and one version of the formula in my answer here: Testing for moderation with continuous vs. categorical moderators. In R (or any other software), you fit a full model with all variables, and a reduced model without the variables you want to test. (If you drop more than one variable, they are tested as a set, not individually, i.e., you get a single p-value for all of them.) Then you perform a nested model test.

m1.full = lm(all_ART~Neuroticism*Agreeableness+Attentional.Control, 
         data=stp2_sub2, na.action=na.omit)
summary(m1.full)  # this will give you the F value for the full model, if you want it
m1.red  = lm(all_ART~Neuroticism+Agreeableness+Attentional.Control, 
         data=stp2_sub2, na.action=na.omit)  # I switched * to +
summary(m1.red)  # this will give you the F value for the reduced model, if you want it
anova(m1.red, m1.full) # this performs the nested model test

edited Apr 13 '17 at 12:44

Community

1

answered Jun 30 '15 at 13:35

gung - Reinstate Monica

132,789
81
357
650

Thank you so much for your comments and apologies for potentially misposting! Strangely I knew of the use of anova() to compare regression models, but hadn't, in my mind, made the connection to doing the same for an interaction model (I somehow assumed a more complex solution was needed for the "change measures"). So, it turns out that my question was indeed more about statistics than R itself. – Tiberius Jun 30 '15 at 22:23
Btw, I just noticed that the p-value for the interaction (model m5, see below) is identical to the p-value resulting from anova(m5, m6). The former is 0.0138 and the latter, as can be seen in my answer below, 0.01383. The same pattern appears for another similar regression model I am running (both p-values are the same). Is this to be expected? – Tiberius Jul 01 '15 at 16:41
@Marcel, yes. If the variable has 1 df, then the F-change test will be the same as the t-test. If you have a categorical moderator with k>2 levels, then you would have 2 t-tests with standard output (that you should ignore) & the single F-test will differ. It may help to read my answer [here](http://stats.stackexchange.com/a/20455/7290). – gung - Reinstate Monica Jul 01 '15 at 16:55
Thank you very much. That clarifies it. Last question, do you agree that the change values in my answer below are as stated in the conclusion (F-change = 6.5014, p-change = 0.01383 and R-sqd-change = 0.1095803). In particular, I still feel a bit uncertain about whether I correctly computed the R-sqd-change value. Any guidance would be much appreciated. – Tiberius Jul 01 '15 at 17:32
@Marcel, you may want to calculate the semi-partial r-squared for the interaction term, see [here](http://stats.stackexchange.com/a/136615/7290). – gung - Reinstate Monica Jul 01 '15 at 18:12
thanks. Given that your comment in concerned with r-sqd, I presume that F-change and p-change in my answer below are already fine. Basically, what I still need, and indeed similar to the link you provided, is an effect size for the interaction effect (i.e. r-sqd-change) so that I can state the percentage of additional variance solely explained by the interaction. Is the way I computed r-sqd-change a possible way to do this or is it wrong/untypical? (...) – Tiberius Jul 01 '15 at 22:31
(...) Also, if I understood you correctly then semi-partial r-sqd is the same as r-sqd-change?! As for computing it, I understand that I need to use the semi.r function and square the output as done in your answer, i.e. sp.x1 = semi.r(y=y, x=x1, given=x2); sp.x1 … sp.x1^2. It is unclear for me, however, how exactly I would need to set up y (y = 4 + .5*x1 - .3*x2 + rnorm(10, mean=0, sd=1)). Could you perhaps explain the idea behind this a bit respectively how I should set it up in my particular case? – Tiberius Jul 01 '15 at 22:31
@Marcel, yes, your F & p is correct. I think that formula is going to be too complicated for you. For the effect size, try partial eta squared (which is a form of R-squared): ( SSE(reduced)-SSE(full) ) / SSE(reduced). – gung - Reinstate Monica Jul 01 '15 at 22:50
thank you! I have computed this as follows now: SSEfull – Tiberius Jul 02 '15 at 08:22
PS: I got the code from here: http://stats.stackexchange.com/questions/49924/how-does-anova-lm-in-r-calculates-sum-sq – Tiberius Jul 02 '15 at 08:45
Hi Gung, if you have a moment, could you perhaps just confirm that I have set up the code for partial eta-squared correctly. It's just that I have a conference poster that I need to finalize tonight and this value is the last missing piece. I'd greatly appreciate a quick yes or no. Thanks. – Tiberius Jul 02 '15 at 18:36

score 0 · Answer 2 · answered Jun 30 '15 at 22:40

I am posting below how I have now resolved this:

Linear models

m5<-lm(data=stp2_sub2, all_ART~Neuroticism*Agreeableness+Attentional.Control, na.action=na.omit)

m6<-lm(data=stp2_sub2, all_ART~Neuroticism+Agreeableness+Attentional.Control, na.action=na.omit)

Computing F-change and p-change

anova(m5, m6)

Analysis of Variance Table

Model 1: all_ART ~ Neuroticism * Agreeableness + Attentional.Control Model 2: all_ART ~ Neuroticism + Agreeableness + Attentional.Control

Res.Df RSS Df Sum of Sq F Pr(>F)
1 51 69011
2 52 77809 -1 -8797.5 6.5014 0.01383 *

Signif. codes: 0 ‘’ 0.001 ‘’ 0.01 ‘’ 0.05 ‘.’ 0.1 ‘ ’ 1

Computing R-sqd-change

rs1<-summary(m5)$r.squared

rs1

[1] 0.140404

rs2<-summary(m6)$r.squared

rs2

[1] 0.03082376

rs.change<-rs1-rs2 # provides R-sqd-change

rs.change

[1] 0.1095803

Conclusion

So, in this particular case: F-change = 6.5014, p-change = 0.01383 and R-sqd-change = 0.1095803. Would be great if you could confirm, if I have understood this correctly.