12

I'm currently working on my thesis, more specifically I'm analyzing some data collected from researchers about the project's they're working on.

In the end, I have performed a multiple linear regression to verify which determinants (budget, team size,...) significantly influence the external impact of a research project. While the dependent variable (project impact) has been assessed from a survey (conducted among the institute's researchers), the independent variables are based on objective measures.

In such a scenario, what would be an acceptable r-square percentage? Could a low percentage be justified by the "subjectiveness" of the dependent variable? I've heard that the scientific area matters... i.e. in physics you rather need to have a r-square > 95% while in sociology > 5% might be already interesting...

Do you have any references about this?

Oliver Lockett
  • 103
  • 3
Steffi
  • 165
  • 1
  • 1
  • 5
  • 7
    (1) You might find [R-squared: useful or dangerous?](http://stats.stackexchange.com/q/13314) informative. (2) Concerning the phrase "significantly influence," you should also read some of the threads found by searching this site for "causality," including [Statistics and causal inference?](http://stats.stackexchange.com/q/2245). – whuber Aug 21 '11 at 19:45
  • 2
    @whuber: (+1) this question originated on math.SE, and I pointed the OP to the same question you have linked to. I think that prompted the deletion of the question over there and the move here. – cardinal Aug 21 '11 at 19:57

6 Answers6

12

There are no absolutes. Different problems have different difficulties. The only thing that would make one reject a certain $R^2$ is another approach getting a much bigger $R^2$ from the same dataset, using a pre-specified or overfitting-corrected model.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
9

The fact that your outcome variable is subjective suggests that it will be measured with quite imperfect reliability. The lower the reliability, the more dampened correlations with other variables will tend to be, so one has to lower one's standards.

I'll take a stab at giving you a more concrete reaction. Knowing only the small amount that I know, I'd guess that an RSQ near .35 would get me very interested as a reviewer or other reader, and fairly well impressed at this model's explanatory power; that one near .20 would get me mildly interested; and that one near .10 would seem on the border between indicating a useful and useless model.

rolando2
  • 11,645
  • 1
  • 39
  • 60
  • so...my r^2 of about 0.38 would be fine then :) btw, do you happen to have any papers at hand which somehow state that such r^2 could be fine for the kind of study I'm conducting?? It's just that I could reference something "official" – Steffi Aug 21 '11 at 19:23
  • I don't, but as @Frank Harrell suggests, there's not much need or use for anyone's imprimateur saying "this rsq is sufficient." Normally there's much more one wants to accomplish with such research, such as estimating coefficients, specifying standard errors of estimate, and so on. – rolando2 Aug 23 '11 at 02:01
3

I'm assuming your dependent variable is some sort of discrete Likert-style score. If so, I'd just make sure the R-squared wasn't near either extreme and put more attention on your regression coefficients.

By the way, what kind of regression model did you use? Ordinary least squares can be frowned upon for presenting final results, but can provide a good first indicator of a relationship. For a final paper I'd go with an ordered probit or logit. You want to be very clear about the distribution you're imposing on the dependent variable -- for example, using least squares assumes that the numbers on the scale are equidistant, when in reality people are pretty inconsistent when providing subjective scores.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Dave Owen
  • 31
  • 1
2

Like you I have often heard that in sociology an R Square of 0.05 is acceptable. I sometimes wonder if that truly can be correct, or does it reflect the lackluster scientific rigor within sociology (at least based on this one benchmark). Granted an R Square of 0.05 corresponds with an absolute R level of 0.22 which would seem to be telling you something.

However, before I would make myself comfortable with such low R Square, I would do a couple of things. Run your model using different data sets. Use hold out datapoints within your data and see how well your model predicts the dependent variable. In other words, can your model results be replicated with different data sets? Is your model predictive? This will define whether your model has a minimum of directional robustness to be useful much more than an arbitrary R Square threshold would tell you.

I would intuit that with an R Square of 0.05 a related model results are overwhelmed by the Standard Error of the model. And, the simple methods used above could assist you in documenting this phenomenon.

Sympa
  • 6,862
  • 3
  • 30
  • 56
  • Sometimes it's less important to predict than it is to understand the overall effect of the coefficient. This is very true of models like this. – Brandon Bertelsen Aug 21 '11 at 19:07
  • 7
    I don't think this has to do with scientific rigor, but rather with the difficulty of the task. If predicting how many days someone has to live, the $R^2$ will be quite low (thank goodness). – Frank Harrell Aug 21 '11 at 20:25
  • 1
    +1 @Frank, but I can't resist pointing out that $R^2$ will be quite high for simple predictions of death dates (based on actuarial tables), *unless you control for age*! This shows nicely why $R^2$ of itself is almost a useless measure of quality or "acceptability" of a model. – whuber Aug 13 '12 at 21:26
  • 1
    You are thinking of group predictions, which are easy compared with prediction of individuals' outcomes. – Frank Harrell Aug 14 '12 at 12:11
1

There seem to be much confusion surrounding the importance of RQS. RSQ does not imply that you have a valid model, rather that there is some "goodness of fit".

See my blog for a breif overview.

http://www.excel-with-data.co.uk/blog-1

user34889
  • 41
  • 2
  • 2
    It would make you answer self contained if you would summarize the main points of your blog post that relate to the OPs question. Also, by RQS and RSQ do you mean the same thing and what exactly do the abbreviations stand for? – Momo Nov 16 '13 at 14:03
-1

You definitely cannot use Multiple linear regression for the analysis of a categorical response variable. In this case you should rather use the logistic model with a logit or probit link (also matters on whether the response is ordinal or nominal). This might be a really late reply though.

Jordache
  • 1
  • 1
    Yes you can, for example the linear probability model for any 2 state categorical variable taking on 1 or 0 – kirk Jun 05 '13 at 12:32
  • Or for ordered categorical variables with some assumption of linearity – kirk Jun 05 '13 at 12:34
  • This relates to http://stats.stackexchange.com/questions/29469/ols-vs-logistic-regression-for-exploratory-analysis-with-a-binary-outcome . Often a "you definitely can't" statement can be met with "maybe I can't do it and satisfy all the assumptions that are important to you; but I can do it and learn something of value to me." – rolando2 Dec 24 '13 at 04:11