4

So I've gone through this SE question and all the answers where the general consensus is that you should never remove the intercept of the linear regression model. The most upvoted answer says:

The shortest answer: never, unless you are sure that your linear approximation of the data generating process (linear regression model) either by some theoretical or any other reasons is forced to go through the origin. If not the other regression parameters will be biased even if intercept is statistically insignificant

However, I'd like to know if it's a good idea to remove the intercept if you get better prediction results (Adjusted r-squared: 0.82) ?
I've trained the model without specifying the intercept ( extra details: used OLS regression in python statsmodels package) and then tested the results.

I cross checked the results and saw that the predicted values and actual values was real close (+/-2 difference in average). So is it a good enough reason as to foresake the intercept?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Nuhman
  • 175
  • 7
  • 2
    Generally, I'd avoid dropping the intercept except in the case referred to in the quotation. But if you have strong statistical evidence (say, by AIC) that the model is better when you drop the intercept, droppinig might be reasonable. That said, it's not clear exactly how you are evaluating the difference in models and you need to be careful about that. – mkt Apr 05 '18 at 11:02
  • 4
    I would say it's almost never a good idea because the situation in which the regression line must run through the origin is very rare. What's the harm in keeping the intercept anyway? – prince_of_pears Apr 05 '18 at 11:21
  • 1
    Maybe you have run into an example of: https://stats.stackexchange.com/questions/26176/removal-of-statistically-significant-intercept-term-increases-r2-in-linear-mo – kjetil b halvorsen Apr 05 '18 at 12:21
  • 2
    @prince_of_pears, the harm is if we cannot reliably estimate the intercept such that 0 is a better guess than, say, an OLS estimate. – Richard Hardy Apr 05 '18 at 15:26
  • 1
    I will defer to your expertise @RichardHardy, but to me it appears that including the intercept should be done if theoretically/substantively meaningful. If not, then fitting the model without the intercept could be justified. One problem I see is how you can tell if 0 is a better guess -- better in what sense? Perhaps the intercept is unreliable because it is extrapolated, in which case a re-parameterization of the model may be a better approach. – prince_of_pears Apr 05 '18 at 19:30
  • 1
    @prince_of_pears, if a model without the intercept gives better predictions (as is the case here), this is an indication that 0 is a better guess than the OLS estimate. – Richard Hardy Apr 06 '18 at 04:24

1 Answers1

4

If you're only interested in predictive performance, then a proof-of-the-pudding argument perhaps can't be gainsaid. But bear in mind that there are three procedures to think about here: (1) fitting a model with intercept by ordinary least squares; (2) fitting a model without intercept by OLS; & (3) fitting both models by OLS; & selecting the one with the higher adjusted coefficient of determination ($R^2_\mathrm{adj}$).

The predictive performance, on fresh data, of models resulting from the the first two procedures can be estimated by $R^2_\mathrm{adj}$. The fewer & the noisier the data, & the closer the intercept is, in fact, to zero, the better you'd expect the predictions to be when constraining the intercept to be zero rather than estimating it from the data.

If you've got no idea how close the intercept is, in fact, to zero, then the third procedure might seem tempting. But you can't have your cake & eat it—applying a constraint only when it appears you don't need to isn't really applying a constraint. See Algorithms for automatic model selection. $R^2_\mathrm{adj}$ is a statistic, subject to sampling variation, not an infallible oracle: the higher of two will be an optimistic estimate of predictive performance— use an independent test set or resampling validation for a less biased estimate.

Scortchi - Reinstate Monica
  • 27,560
  • 8
  • 81
  • 248