If the model fits well, nothing can be done?

Question

I am playing a data without any background information. First, I try multiple linear regression. The model fits well, since the $r^2$ is larger than 90%. I deleted several variables by AIC. The fits improves a little bit. I am interesting what are possible directions that I should look into, if the model fits well at the very beginning.

The target of the analysis is find the model that make the most accurate prediction.

What I can come up with:

1. Try other models like regression tree, random forest, SVM or whatever data mining models. The prediction may or may not become better.

2. Perform cross validation.

3. Check overfitting.

4. Diagnostic residual.

Answer 1

Your ideas what to do next are perfectly fine and in fact, if some statistic measure says that your model "fits fine", that is not enough to say that you are done with it. First of all, there are multiple criteria for assessing model fit, $R^2$ is just one of them and it is not perfect (see this discussion). In fact, checking for overfitting and model diagnostics is a must if you want to be sure your result is not garbage.

For example, Leonard A. Stefanski provides lots of funny examples of made-up data that gives "significant" estimates for linear regression, but if you check the residuals, you get... well... a bison (?!).

So sometime you could get "significant" results with a rubbish data. That means that you have to check your data, residuals etc. The important thing to check is outliers - sometimes a single observation that is very extreme could influence your results (especially with a small data). Check also if your data is fine - if there is no mistakes in coding. Generally, it is good to check descriptive statistics and plots of all your variables included in the model and of the values predicted by the model.

Another thing that could be done is posterior predictive checks. Gelman and Hill give in their book a very readable tutorial on regression models fitting and selection, so I would recommend you their book.

However, in general, you should ask yourself if the model is useful for you, i.e. check it's practical significance. Sometimes the model could fit well but give you results that are no help for you to improve your understanding the problem you are dealing with, so you try different models.

What you should remember it the rule of parsimony, i.e. the simpler model in most cases is the better one. So if your simple model fits fine, that is fine! Consider that if you have complicated model, also interpreting the results you obtained is more complicated. Try to describe your results in plain English - with complicated models it gets tricky, and if statistics is a tool for providing you help with understanding things, then is a model that does not provide you intelligible information useful?

Finally, you could check different methods of estimation (e.g. OLS, ML, Bayesian) so to check if they give you similar results. If not, ask yourself why and try to diagnose the problem. Trying different statistical methods is also a good idea.

Check also this discussion.

Answer 2

I'm not sure what the context of your data is, but you should beware of relying on $R^2$ for determining goodness of fit. As you add more explanatory variables to your model, $R^2$ can get inflated, making it look like you have a 'good fit', but really all your doing is lowering your total sum of squares(since the goal is to minimize sum of square residuals, which is a component of total sum of squares). You should check out Wikipedia's page on Coefficient of Determination and scroll down to "Inflation of $R^2$" and "Notes on Interpreting $R^2$". You could also perform statistical inference on the coefficients of your regression model to see if they are statistically significant.

Methods like Decision Trees and SVM are used more or less for classification, but can also be used in regression as well.

What is the data you are working with?