To what extent are the different versions of $R^2$ comparable, with and without an intercept

Question

I have been trying to fit some data with a linear regression. I don't have any theoretical assumptions on the regression, and from what I know both an intercept or no intercept can be plausible results.

In the evaluation of the goodness of fit, I came across two different definitions of $R^2$, the most general one

$$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} $$

where $SS_{res}$ is the sum of squares of residuals and $SS_{tot}$ is the total sum of squares, and another definition that is justified only in the case of a fit with intercept

$$ R^2 = \frac{SS_{reg}}{SS_{tot}} $$

where $SS_{reg}$ is the regression sum of squares and the condition $SS_{tot} = SS_{reg} + SS_{res}$ is verified.

I know that some softwares (like R) use the second definition, with a "modified" version of $R^2$ in the case of no intercept, that can give conflictual results and this is a problem that has been already addressed. In another question I read that also for the first definition I should use a modified version of $R^2$ in the case of no intercept.

What I want to understand is if there is a general definition of $R^2$ that is always applicable? If there isn't one, in what way can I compare the goodness of a fit with intercept and a fit without intercept, and ultimately to choose between the two? Thank you in advance

Answer 1

$R^2$ compares your model to a naïve model that always predicts the same value. When your model includes an intercept, it makes sense that the naïve guess, ignoring the features, of the conditional mean would be the pooled/marginal mean of all $y$ observations: $\bar y$.

If we ignore the features, the linear regression equation is:

$$ y = \beta_0 + 0x_1 +\cdots + 0x_p =\beta_0 $$

OLS regression gives us $\hat\beta_0=\bar y$.

This is how we wind up with the denominator of the usual $R^2$ equation. Every prediction is the same $\bar y$ value.

$$ R^2_{usual} = 1-\dfrac{ \sum\bigg( y_i - \hat y_i \bigg)^2 }{ \sum\bigg( y_i - \bar y \bigg)^2 } $$

If we exclude an intercept from the model, then the model with no features is: $$ y = 0x_1 +\cdots + 0x_p = 0 $$

That is, the naïve model always predicts zero, and the comparison to the naïve model would reflect that fact.

$$ R^2_{no\_intercept} = 1-\dfrac{ \sum\bigg( y_i - \hat y_i \bigg)^2 }{ \sum\bigg( y_i - 0 \bigg)^2 } $$

If you want a general way of thinking about $R^2$, I would go with the idea of comparing to a naïve model that has no features.

It doesn't stop there. While this will wreck the typical interpretation of $R^2$ as the "proportion of variance explained" (which does not apply to the no-intercept version, anyway, and this interpretation vanishes in many settings with intercepts, too), you can use any baseline model in the denominator. Want to compare your fancy neural network to a linear model?

$$ R^2_{sorta} = 1-\dfrac{ \sum\bigg( y_i - \hat y_{i, neural} \bigg)^2 }{ \sum\bigg( y_i - \hat y_{i, linear} \bigg)^2 } $$

This is equivalent to comparing the models on mean squared error, $MSE = \frac{1}{n}\sum\big( y_i - \hat y_i \big)^2$, however, and I do not see an advantage to $R^2_{sorta}$.

Viewing the problem in terms of $R^2_{sorta}$, the $R^2_{usual}$ and $R^2_{no\_intercept}$ equations are comparing to baseline models that are totally naïve.

Cardinal wrote a rather excellent answer related to this topic.