1

I analyze a dataset consisting of responses to two conditions with a multiple regression analysis. I dummy code the two conditions to retrieve the contribution of each of them to the data. Up to here, no problem, I know what I am doing.

Second, I want to check that the difference between the two conditions (Co1 and Co2) is not confounded by another variable of no interest (NoI). I know that in a multiple regression model, the estimates of each condition correspond to the expected change in the data when a change of 1 is observed in its predictor, while all others are held constant. I thus add a covariate to the analysis with NoI and look at the estimates. Provided NoI does share some variance with Co1 and Co2, should I consider that NoI has taken all of the variability of the data that it can account for?

If yes, then, Imagine I'd be looking at it the other way around: I would want to know the impact of NoI on my data, but would have also two confounding conditions Co1 and Co2. I'd run the same model and observe that the estimate for NoI is smaller when Co1 and Co2 are accounted for. I would say with the same model that Co1 & Co2 have taken all of the variability of the data that they can account for... If NoI shares some variance with Co1 & Co2, this is contradictory with the previous paragraph.

So I guess that each NoI and Co takes its share (half?) of the variability it can account for in the data? but how is that share determined?

I've read this question and the answer and thought I understood everything: that it depends on the way the sum of squares is split, until I read this comment to find out that we're talking of sum of squares only when testing for effects, not when estimating. This confuses me a lot. I don't understand why the problem of sharing variance or sum of squares is different in ANOVA and when doing multiple regression. I get all the more confused when I read elsewhere that ANOVA and multiple regression are one and the same thing.

I also came across the idea that we could orthogonalize regressors sequentially, as described here, and done with this code so that they are completely independent and only account for their own share of variance. This is appealing but does not seem like a wide spread practice.

In sum, I need help for

  1. Understanding how variability in the data is shared/split/not between two correlated regressors.
  2. What aspects of ANOVA and multiple regression are one and the same, and what is different?
  3. Knowing if it is best practice to sequentially orthogonalize regressors as mentioned above.

I hope this question makes sense. I'm really struggling to understand what I'm doing here...

Community
  • 1
Max
  • 131
  • 4
  • Each variable takes all the sums of squares "available" to it when it is fitted, and every subsequent variable takes what is left conditional on prior variables having already taken what they could. Coefficients in the final model (and their standard errors) are effectively as if fitted last. – Glen_b May 11 '14 at 01:35
  • Thanks for your comment. Fitting does not proceed sequentially, but all at once, so I don't think you're right (entering regressors in one order or another doesn't change the fitted coefficients). I think that you're describing ANOVA with type I sum of squares. This is explained [here](http://stats.stackexchange.com/a/20455/39506). I don't understand why the question of sum of squares partitioning only arises when testing, and not when fitting... – Max May 12 '14 at 07:19
  • Please carefully note the import of "as if" in the final sentence. The intent of my comment is *exactly* in accordance with your comment of "doesn't change the fitted coefficients". It's a way of seeing both the coefficients and the sum of squares in a single framework. I am talking about type I sum of squares, since that's what you asked about. Is your final sentence there an additional question or an attempt to clarify your question? – Glen_b May 12 '14 at 08:35
  • I see. Now I was wondering why [in this comment](http://stats.stackexchange.com/questions/24827/where-is-the-shared-variance-between-all-ivs-in-a-linear-multiple-regression-equ#comment45321_24833) user gung (who seems quite savvy, but I don't know how to involve here) said: "The overlap is in the test, not the betas--I'm not sure how else to put that. Each beta denotes the effect on the response variable of a 1-unit change in the covariate, with everything else held constant; a given beta would almost certainly not be the same if the other covariates were removed from the model." – Max May 12 '14 at 12:28
  • Everything after the first sentence of the quote is incontrovertible. The first sentence seems to be gung's description of what that fact means. You can't pull anyone into a thread they're not in, but you can try @gung in chat, which gung has used, so it will notify him. You're correct in thinking gung is quite savvy. – Glen_b May 12 '14 at 21:21

0 Answers0