2

I'm constructing a linear model from a data set with 10 variables and my current "best" model uses 4 variables. I've tested the variables and not all of them show significance, so the most that I might add to the model might be 5 variables overall. However, with 4 and 5 variables, eventhough I get very good Pr(>|t|) values for all the variables that I've added and while my $R^2$ value has been improving, it's still just "only" $0.4287$ and the scale is $[0,1]$.

Should I be happy with $< 0.5$ $R^2$ value of should I aim to improve it in order to improve my model even more? What can I do to increase the $R^2$ value now that I have already added all variables that I can add as predictors? Should I look into interaction terms? Or something else?

mavavilj
  • 3,167
  • 4
  • 20
  • 44
  • 1
    A high Rsquared is not everything. You may have a look at a discussion R-squared versus adjusted Rsquared. – Ferdi Oct 15 '16 at 15:20
  • Use another criterion to adjust your model, like, AIC. This won't improve R^2 but will help variable selection. – conv3d Oct 15 '16 at 16:03
  • @jchaykow How to do that in R? – mavavilj Oct 15 '16 at 16:51
  • @mavavilj I've done this method in my blog: http://rcode.io/work/2016-04-04-FF/ Scroll down to forward selection and backward selection. – conv3d Oct 17 '16 at 00:39

1 Answers1

1

One common approach in these cases is to plot the train and test errors as a function of the size of your dataset. This might give some insight as to how the specific bias variance tradeoff of the algorithm you're using, is working for the particular problem.

For a specific value of n, generate several subsets of size n from your dataset. For each one, calculate the train error and test error (the latter, using the hold-out data), and record the average.

Repeat this for several values of n, and plot the two curves:

  1. If the algorithm is suffering from high bias, then the test error will initially decrease, then settle at a relatively-fixed high value.

  2. If the algorithm is suffering from high variance, there will typically remain a large gap between the train and test error.

In your case, you mention that you're using a linear predictor, so high bias is somewhat more of a suspect. If the graphs corroborate this, a good guess would be to switch to one of many higher-variance algorithms.

Ami Tavory
  • 4,410
  • 11
  • 20