3

What is a correct way to compare two $R^2$? I have dependent variable $Y$ and $X_1, X_2, X_3, X_4.$ I run two regression models, namely with $X_1$, $X_2$ and $X_3$, $X_4$. Both $R^2$ values are close. So how I determine if they are statistically different (or the same)? How to get F-statistic for these two?

Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
Donatas Gudauskas
  • 157
  • 2
  • 7
  • 1
    Perhaps you could bootstrap confidence intervals for each $R^2$ and see whether they overlap, and if so, by how much? – Richard Hardy Jan 31 '15 at 19:46
  • Ya, that might be one of the possible solutions, but I'm really looking for a way to find F-statistic and determine p-value for them being equal. – Donatas Gudauskas Jan 31 '15 at 19:51
  • Another option would be to test restrictions on a nesting model. You would estimate a regression of $Y$ on all $X$'s and test restrictions of $X_1$, $X_2$ and of $X_3$, $X_4$. If you reject one and fail to reject the other that would be some indirect evidence for which of the pairs is more important. Another comment (somewhat off-topic): recall that statistical significance depends on sample size. In a large-enough sample anything will be significant (while you can almost always avoid finding statistical significance if you use a small-enough sample). – Richard Hardy Jan 31 '15 at 20:10
  • I have found F-statistic and p-values for restricted/unrestricted regression. But that only adds incremental information. I mean I test Y=x1+x2+x3+x4 against Y=x3+x4. And it just proves that x3 and x4 adds some explanatory information to the regression, it does not show if that information is the same. – Donatas Gudauskas Jan 31 '15 at 20:21
  • If the information is added both ways then you cannot conclude much. However, if it is added only one way that would say that one pair of regressors sort of encompasses the information in the other pair of regressors, but not vice versa. But I admit this does not answer your original question. By the way, what is the real goal of answering the question? What do you intend to do next? – Richard Hardy Jan 31 '15 at 20:26
  • I am testing different performance measures against adjusted stock returns. Want to find F-statistic and p-values to either reject or fail to reject null hypothesis that all measurements are not the same. – Donatas Gudauskas Jan 31 '15 at 20:31
  • 1
    I doubt that a simple F-test exists for non-nested models... Anyhow, here is one more alternative: calculate the AIC values of the two competing models and see how large the difference is. There have been some suggestions of what values signal statistical significance [here](http://stats.stackexchange.com/questions/81427/aic-guidelines-in-model-selection) (see answers by Metrics and Stat). For another reference, check out the last page of [this](http://limnology.wisc.edu/courses/zoo535/handouts2010/week04/Comparing_Models.pdf). – Richard Hardy Jan 31 '15 at 20:56
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/20721/discussion-between-richard-hardy-and-donatas-gudauskas). – Richard Hardy Jan 31 '15 at 21:03
  • bootstrapping is the only way I know of, or to compare models by AIC. I there there is a psychology-related R package that can bootstrap R2, or the boot package could be used. – N Brouwer Feb 01 '15 at 14:22

0 Answers0