9

Let's say I have two regression models, one with three variables and one with four. Each spits out an adjusted r^2, which I can compare directly.

Obviously, the model with the higher adjusted r^2 is the better fit, but is there way to test the difference between the two adjusted r^2 and get a p-value?

I know you can do Chow test to test the difference between slopes, but this is variance, so I don't think that's what I'm looking for.

Edit: One model does not simply contain a subset of variables from the other model, or else I would probably use stepwise regression.

In model 1, I have four variables: W, X, Y, and Z.

In model 2, I have three variables: W, X, and (Y+Z)/2.

The idea is that if Y and Z are conceptually similar, the model may make better predictions by grouping these two variables together prior to entering them into the model.

Jeff
  • 3,525
  • 5
  • 27
  • 38
  • 3
    Are the models nested (i.e. are the models the same except for the one variable in the four variable model?) – Andy W Apr 22 '11 at 02:49
  • Good Q.. No they aren't, but close. One model uses four variables, W X Y and Z. The other model has three variables, W X and (Y+Z)/2. Although Y and Z may or may not be weighted equally in the second model. – Jeff Apr 22 '11 at 03:15
  • 1
    you should update your question with this information, attempt to write out the models you are fitting mathematically, and be as explicit as possible about the transformation to "Y and Z" and what you are trying to accomplish with that transformation. – Andy W Apr 22 '11 at 03:26
  • Well let's just stick with a simple average for now... Q has been updated, thanks! – Jeff Apr 22 '11 at 03:40
  • 1
    Yes, the models are nested. To see this, you can rewrite model 1 in terms of W, X, (Y+Z)/2, and (say) (Y-Z)/2, showing that model 2 just eliminates one variable. – whuber Apr 22 '11 at 14:35

3 Answers3

9

As whuber stated this actually is a case of nested models, and hence one can apply a likelihood-ratio test. Because it is still not exactly clear what models you are specifying I will just rewrite them in this example;

So model 1 can be:

$Y = a_1 + B_{11}(X) + B_{12}(W) + B_{13}(Z) + e_1$

And model 2 can be (I ignore the division by 2, but this action has no consequence for your question):

$Y = a_2 + B_{21}(X) + B_{22}(W+Z) + e_2$

Which can be rewritten as:

$Y = a_2 + B_{21}(X) + B_{22}(W) + B_{22}(Z)+ e_2$

And hence model 2 is a specific case of model 1 in which $B_{12}$ and $B_{13}$ are equal. One can use the likelihood-ratio test between these two models to assign a p-value to the fit of model 1 compared to model 2. There are good reasons in practice to do this, especially if the correlation between W and Z are quite large (multicollinearity). As I stated previously, whether you divide by two does not matter for testing the fit of the models, although if it is easier to interpret $\frac{W+Z}{2}$ then $W+Z$ by all means use the average of the two variables.

Model fit statistics (such as Mallow's CP already mentioned by bill_080, and other examples are AIC and BIC), are frequently used to assess non-nested models. Those statistics do not follow known distributions (like the log-likelihood does, Chi-square) and hence the differences in those statistics between models can not be given a p-value.

Community
  • 1
Andy W
  • 15,245
  • 8
  • 69
  • 191
0

Given the setup in Andy W answer, if one estimates the model

$Y = a_3 + B_{31}(X) + B_{32}(W+Z) + B_{33}(Z) + e_3$

the test associated with $B_{33}$ gives you the test that model 1 is different from model 2. The reason is that $B_{33}$ is exactly (a part from the sign) the difference between $B_{12}$ and $B_{13}$. Thus, if their difference is not significant, keeping W and Z in the model (model 1) does not help in terms of variance explained as compared with combining them in one variable (model 2). If $B_{33}$ is significant, model 1 is better.

mcfanda
  • 31
  • 1
0

Take a look at Mallow's Cp:

Mallow's Cp

Here's a related question:

Is there a way to optimize regression according to a specific criterion?

Community
  • 1
bill_080
  • 3,458
  • 1
  • 20
  • 21