1

Consider the following multiple regression analysis:

Step 1: enter control variables, which explain a significant 25% of the variance in the DV

Step 2: enter variable of interest, which explains an additional ($R^2$-change) significant 7% of the variance in the DV.

How do I interpret the the $R^2$-change effect size? Is it so small that (although statistically significant) it is pretty meaningless?

Ferdi
  • 4,882
  • 7
  • 42
  • 62
pomodoro
  • 723
  • 5
  • 15
  • 2
    Try to do an F-test? – SmallChess Oct 25 '17 at 04:39
  • The f-test is significant (as stated above, the variable in step 2 explains a significant additional 7% of the variance). My questions relates to the effect size and it's interpretation rather than the significance - is an additional 7% variance explained meaningful? Are there rules of thumb for interpreting r2change? – pomodoro Oct 25 '17 at 05:03
  • In order to make you even more confused about the interpretation of 7% (which means 7% **in addition**). It would be worthwhile to read on types of sums of squares. https://stats.stackexchange.com/questions/20452/how-to-interpret-type-i-type-ii-and-type-iii-anova-and-manova – Sextus Empiricus Oct 25 '17 at 10:30

1 Answers1

1

You already found that the additional variance explained is statistically significant. This of course is different from so-called "clinical significance".

Here is a possible strategy to see whether the additional variance explained is actually "meaningful". Run cross-validation on models with and without your variable of interest. Save the out-of-bag prediction accuracy of both models. Compare. Is the difference in predictive power meaningful in your context?

A small increase in predictive power can be very meaningful indeed, if you are predicting stock prices. Or a large difference can be meaningless. For instance, maybe you are predicting demand for a particular grade of steel and managed to halve prediction errors - but the production process still needs to fire up the blast furnace to make a full batch in make-to-stock, and your better prediction doesn't change the subsequent process in any way. (If so, best to look for a different job.)

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357