Estimating the effect size in polynomial regression

Question

In polynomial regression, estimating the effect of an independent variable on the outcome seems to be quite tricky (to me). For example: I want to compare the influence of $x$ on $z$ with the influence of $y$ on $z$ (I am satisfied with a statement like "the influence of $x$ is bigger than $y$").

In a linear model I would simply look at standardized regression coefficients (betas). But comparing residuals and fitted values in the linear model lm1 <- z~x+y makes me want to include quadratic terms, and an interaction, too. Ergo: lm2 <- z ~ x + y + x² + y² + x:y

Since there are collinearities, standardized regression coefficients (betas) will become $>|1|$. An interpretation of these is not easy, because since $x$ and $x^2$ are related, the betas are related. Thus, I am simply not able to give a statement about the size of influence of $x$ and $y$ (compared to another).

I can calculate the same equation using orthogonal polynomials. This gives me betas $<|1|$. But, does it make sense to interpret them orthogonal? Since in reality $x$ and $x^2$ are not independent.

I read about Cohen's $d$, but this seems to have the same problems.

Now I thought about $\eta$'s in ANOVA. As I understand it, $\eta^2$ gives: "Out of the total variation in $z$, the proportion that can be attributed to a specific $x$." Moreover, ANOVA and linear regression are close. So my questions are:

1. Can I use $\eta^2$ in collinear models to estimate the effect size of $x$ and $y$ on $z$? Or will $\eta^2$ automatically become "invalid" since there are more than one independent variable?
2. Can I use the betas from the linear model (lm1) to estimate the influence of an independent variable on the outcome even if I use the polynomial model (lm2)? (Is the combined influence of $x$ and $x^2$ in lm2 related to the influence of $x$ in lm1?)

Answer 1

Here are my suggested answers:

1. The use of $\eta^2 = R^2$ is not unreasonable. If you obtain the model without any $x$ variables (model lm.1) and the model with all of the variables containing an $x$ term (model lm.2), you can calculate $\Delta R^2$ for the amount of additional variance explained by the two models.

2. There was another post about this issue (but I don't have the time to locate it now...maybe in a follow-up), and I have been thinking about it a bit. One possible work-around might be the following. Calculate the model for the effect with the higher power: $$y = \alpha + \beta_1 x + \beta_2 x^2 + \epsilon$$ Now, transform $x$ to $z$ via $$z = x + \frac{\beta_1}{2\beta_2}$$ If I did my math correctly, the new model will be $$y = \gamma + \zeta_1 z + \zeta_2 z^2 + \epsilon$$ but...$\zeta_1$ should be zero. If you do something comparable with the $y$ predictor, you may be able to compare the corresponding $\zeta_2$ terms. (I'm very curious to hear what others think about such an approach.)

Answer 2

Let me make two preliminary points. First, if you have, say, $X$ and $X^2$ in a model, when you form an interaction, you need product terms with both of these variables. That is, in your example model you should have, lm2 <- z ~ x + y + x² + y² + x:y + x²:y + x:y² + x²:y² (or, in a simpler R formula, lm2 <- z~poly(x,2)*poly(y,2)). Second, it makes no sense to ask about the effect size associated with $X$ in a model that includes interactions involving $X$.

---

Let's thus resituate your question in a model without interactions (i.e., lm2 <- z ~ x + y + x² + y²). Now, how could we get a measure of effect size associated with, say, $X$? This is not that hard, and multicollinearity isn't really an issue here. From the model's point of view, $X^2$ and $X$ are just two different variables, even if you know better¹. Thus, you are asking for an effect size associated with multiple variables. In fact, $\eta^2$ is just that sort of effect size. It is typically used for multi-level categorical variables (factors) that are represented by multiple variables in the model. To compute it, you need to know the sums of squares that are associated with those variables in your model. Most likely, you want the 'type III' sums of squares, or the sums of squares after having accounted for the other variables in your model². You can do this by dropping all of the $X$ variables (in your case, $X$ and $X^2$) and refitting the model. Call this lm3. Get the sum of squared errors from both lm2 and lm3. The latter will be larger. Compute the difference, $SSE({\rm lm3})-SSE({\rm lm2})$. Those are the sums of squares that are associated with all of the $X$ variables. From there you can compute $\eta^2$ or $\eta^2_\text{partial}$³.

<sub>1. Why is polynomial regression considered a special case of multiple linear regression?
2. How to interpret type I, type II, and type III ANOVA and MANOVA?
3. How to compute η2 in ANOVA by hand?</sub>