In predictive modeling I see typical error metrics like RMSE, MAE, RAE be used all the time. Even though they all seem so popular, I have trouble finding academic reference for those error-metrics.
Does anyone know a good paper/book that summarizes all the different metrics that are out there? Preferably one that goes beyond just RMSE, so I can get an idea of which error metrics I could use for evaluating predictive accuracy besides the standard ones.
Pretty much any standard textbook on forecasting will have a section on error metrics.
Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman, specifically section 3.4.
Principles of Business Forecasting (2nd ed.) by Ord, Fildes & Kourentzes, section 2.7.
Demand Forecasting for Managers by Kolassa & Siemsen (sorry about that self-promotion), chapter 11.
We also have a couple of threads here at CrossValidated, and people do indeed cite them (click on the "cite" link underneath a post to get a BibTeX entry). For instance:
Finally, most academic articles that introduce a new error metric will have a review of existing metrics (and explain why their new one is better), for instance:
This particular paper introduced the MASE and is quite legible. They treat most of the common error measures. The IJF is usually a good source for papers on new error metrics. (The editors and reviewers know a bit about forecasting, so they will weed out some low-quality stuff that does slip through the cracks and gets published elsewhere.)
In terms of when to use what error metric, you may be interested in yet another piece of shameless self-promotion:
The R package hydroGOF is very well documented and has functions to calculate many different fit statistics including RMSE, MAE, but also Nash-Sutcliffe efficiency (R2), Pearson Correlation coefficient, percent bias, etc. The help files list the formulas used in the calculations along with the appropriate academic journal article. You can check out the package page for all the information in one place.
