4

This is a very general question about R-squared or the coefficient of determination. I found a couple of threads on CV but none that answers my question in a straightforward way.

In short, what is a ‘low’ R-squared when running multiple linear regression? From which minimum value should we conclude that our model does not make better than the baseline?

I sometimes see R-squared values that are as low as 0.15, yet the models are significant. I guess this depends on size, on whether R-squared is used for prediction or inference, etc., however I still do not have a good intuition for it.

It also seems to me that in the ‘hard’ sciences, R-squared tend to be high (say, 0.8 or higher in classic cases), whereas in the social sciences, from what I can see, it tends to be lower (say, under 0.5). I know this might be a gross generalization, however.

Any thoughts much appreciated.

johnjohn
  • 143
  • 6

4 Answers4

3

The answer to your question is no - without more context, we can never simply look at the $R^2$ value of a model and say "this is good enough" or "this is bad" aside from the (somewhat trivial) case of $R^2 > 0$. As Dave point's out, $R^2$ contains no more information than the MSE.

As an example that helps illustrate why this is the case, see section 3.2 of these notes, which points out that among other annoying properties of $R^2$, we have:

$R^2$ can be arbitrarily low when the model is completely correct. ... By making $Var[X]$ small, or $\sigma^2$ large, we drive $R^2$ towards 0, even when every assumption of the simple linear regression model is correct in every particular.

So a linear model which captures a true linear relationship correctly may still have an arbitrarily low $R^2$ if the residual noise is large. This is one of the reason we see different $R^2$ in different settings, like physics laboratories vs econometric analyses - there are (among other differences) probably different levels of residual noise, even when we have a model that is correctly specified.

Louis Cialdella
  • 1,404
  • 12
  • 17
  • many thanks @ Louis Cialdella, this is very helpful. Thanks for the notes as well, these are very interesting. – johnjohn Feb 17 '21 at 18:03
3

Statistical significance (rejecting the null hypothesis of zero regression coefficients) involves the observed $R^2$ and the sample size $n$. Under certain assumptions, for small $R^2$ and large $n$, the significance approximately depends on the product $R^2 n$; this can be seen for the correlation coefficient significance in simple regression and for the $F$-test in multiple regression. This indicates that even subtle effects (small $R^2$) can be measured with significance given a large enough sample. Intuitively, the larger the sample, the more averaging occurs in calculating $R^2$ and the more the "noise" is reduced in comparison to the "signal".

nanoman
  • 481
  • 3
  • 6
2

Consider what $R^2$ means: proportion of variability explained, compared to a baseline model that always guesses the average value of the pooled response variable.

If you’re higher than $R^2=0$, which you probably will be with in-sample data when you use an intercept, then you’re beating the baseline performance.

Dave
  • 28,473
  • 4
  • 52
  • 104
  • Many thanks @Dave. This I get, yes. My question is more about what we can consider to be a low/bad R-squared? Likewise, can we really say that our model is better than the baseline when R-squared is, say, 0.1? I know 0.1 is better than 0, yet can we say that such a model is helpful to answer a research question, for example? I was also wondering why in tutorials we see R-squared of 0.9 or higher, yet in real life (say, when doing actual research), we rarely see such as high R-squared. But maybe my question is too naive? – johnjohn Feb 17 '21 at 16:52
  • 1
    $0.1>0$, so a model getting $R^2=0.1$ is better than the baseline. “Good enough” is going to depend on the task, but you probably aren’t comparing your performance to the intercept-only model but to competing models made by rival groups (think Citi vs Bank of America). // I don’t care for $R^2$ for this reason. For model comparison, it contains no information beyond what you get from MSE (assuming the same data), and it puts us in a position to equate scores to letter grades in school where $R^2>0.9$ is an $A$, $0.9>R^2>0.8$ is a $B$ etc, when $R^2=0.2$ might be excellent (consider t-testing). – Dave Feb 17 '21 at 16:59
  • thanks @Dave, indeed it contains same info as MSE (which is also why I was wondering what the rationale of R-squared is). The idea that it can be used for comparisons between models/competitors rather than per se or with intercept-only model is helpful. – johnjohn Feb 17 '21 at 18:05
2

The $R^2$ can be interpreted as a percentage of variation in the $y$ that is explained by the regression. This is not in the first place about the question whether the $x$-variables have an influence or not (although there is a mathematical relation, as pointed out in another answer) as opposed to significance tests, and neither is it about whether linearity holds. What can be said is that a model with low $R^2$ will not allow to predict very precisely, and it indicates that the modelled influence of the $x$-variables is rather weak compared to variability coming from whatever other sources. There are no fixed cut-off values, as it depends on the application how much precision is required. A model with $R^2=0.1$ can be good if a substantial practical advantage can be achieved by predicting $y$ even very roughly from $x$, whereas $R^2=0.7$ may be low if there is a requirement to control $y$ strongly given $x$. You can give yourself more intuition just looking at a number of two- or low dimensional data sets and their $R^2$-values, where you can see how precisely the $y$ are fitted by the model, and how this translates into $R^2$.

Note that $R^2$ is computed on the training data, and its relation to prediction quality will (apart from the linearity assumption) depend on the number of $x$-variables, for which reason there is "adjusted $R^2$".

Christian Hennig
  • 10,796
  • 8
  • 35