-1

I am trying to find a metrics to evaluate performance of one regression based ML method. Currently we are observing difference in RMSE is very less to conclude anything. So I thought of one AUC based method 1. I will divide my test observed and predicted values into 2 labels based on different threshold(let say 100 different different thresholds). 2. I will compute TPR/FPR at each of above threshold values and take average of these TPR/FPR.

I want to know if there is any statistical mistake in this metrics

Paras Malik
  • 9
  • 3
  • 2
    Honestly, this is a terrible, uninterpretable metric... Why wouldn't you use one of many available metrics for continuous outcomes? What is the problem with RMSE? See e.g. https://stats.stackexchange.com/questions/131267/how-to-interpret-error-measures-in-weka-output/131273#131273 – Tim Sep 26 '17 at 19:29
  • @Tim for MLE the RMSE might be inapplicable. A deviance-based measure would be worth considering. – whuber Sep 26 '17 at 20:05
  • @Tim I have tried using MAE and RMSE. Problem is I am trying to predict continues value rare event in sparse data. If I use MAE prediction close to median, which is always 0, so I decided to use RMSE. the difference in RMSE is of the order of .01%, so its really difficult to conclude how good/bad a particular method is doing. intution of above mentioned TPR/FPR is "on average how good method will predict given any threshold. i.e. Given random threshold what is the expected predictions are in the same side as observed values divided by how many of them which are on the other side." – Paras Malik Sep 26 '17 at 20:12
  • @Tim I will check how RRSE will behave. – Paras Malik Sep 26 '17 at 20:19

1 Answers1

2

First, it would be helpful if would explain why RMSE has not been useful. Are you saying that between two models, the difference in RMSE is insignificant?

Secondly, TRP/FPR are concerns of a classification model, not regression. Please confirm.

B.Frost
  • 171
  • 5
  • Yes difference in RMSE is very small its of the order of .01%. – Paras Malik Sep 26 '17 at 20:13
  • TRP/FPR is for classification model but I converting my continues variables to classification variable many times using different thresholds and computing TPR/FPR everytime and taking average at the end. – Paras Malik Sep 26 '17 at 20:15
  • If you "are trying to predict continuous value rare event", perhaps you should try imputing a binary flag for all observations that fall within the value range of interest, and approach this as a classification problem. – B.Frost Sep 26 '17 at 21:29