0

ok, so I have a memory-based model to predict the values of variable X and I've also measured the real values of X. I have calculated both the RMSE and MAPE for the difference Xreal-Xpredicted. I know that there is absolutely no point in saying a model is good or bad if the RMSE value is less than a particular value. I was wondering though if there is a rule of thumb providing insights on the subject. For example my data range from 0 to 4 and I have an RMSE of 1,1. Is there any empirical rule suggesting this is a good/bad estimation?

thank you in advance

ETA: X may take all values within the range{-2,2}

Anastasia
  • 83
  • 8
  • 1
    The ratio of the RMSE to the standard deviation of the data is directly related to $R^2$, so for (extensive) discussion about this point, please review questions on [tag:r-squared] such as http://stats.stackexchange.com/questions/13314/is-r2-useful-or-dangerous/13317 and http://stats.stackexchange.com/questions/14585. – whuber Apr 23 '12 at 18:38
  • thank you. However the prediction is not the result of a regression process therefore I was wondering if some other rules apply to it. – Anastasia Apr 23 '12 at 18:59
  • 1
    I am curious why we should care how a model is derived when we are assessing its quality. (Putting aside the problems with assessing model fit using the same data employed in computing that fit...). It seems to me that if you view RMSE as a valid measure of fit, then that's all that matters: the inner workings of the model should be immaterial. – whuber Apr 23 '12 at 19:32
  • What is the nature of your outcome, $X$? Is it a continuous, ordinal, polytomous, or even binary variable? My concern is whether classification accuracy could be an issue. It's always useful to, at least, describe what metric you're trying to predict. In general, there is no global measure of predictive accuracy. The most adequate substitute is a prespecified minimally sufficient, expected, and superior level of predictive accuracy based on input from an expert in the field. This is a different story if you have a competing predictive model. – AdamO Apr 23 '12 at 20:01
  • @AdamO: yes, X is a continuous variable. I just wonder whether, since it takes values within a certain range, that would give us any insight regarding the quality of estimation. I mean, if we have and RMSE of 2 for the range [-2,2] is quite different from having the same RMSE for [-100,100], no? – Anastasia Oct 22 '12 at 13:59

0 Answers0