1

This question may seem a bit odd but here we go.

I have a supervised-learning pipeline that I am using to forecast a continuous variable. The model displays reasonably good evaluation metrics across the board with much of the variance being explained by lags / other autoregressive features (I've implemented both ML and traditional time-series econometric methods for this).

The model often fails to predict the continuous variable to the exact number (as one would expect). Given the model will be used by a non-technical party, I'd like to translate the model output to be interpreted as a risk or vulnerability index (where higher projected values = higher risk/vulnerability). The most obvious way to do this may be to group the y feature into bins and apply supervised classification (this I've already done). What I'd like to do is translate the resulting projected probabilities or predicted continuous values into some sort of standardized metric of "risk."

Do you have any creative thoughts on how to do this? Does this sound reasonable? Essentially, I am looking to transform predicted values into an interpretable risk metric for comparison across observations. Happy to elaborate further if the above is confusing.

Cheers

creekjumper
  • 11
  • 1

1 Answers1

0

The model displays reasonably good evaluation metrics across the board with much of the variance being explained by lags / other autoregressive features

"Variance explained" sounds like you are evaluating in-sample fit. This is usually not a good guide to out-of-sample forecast accuracy. Instead, use a holdout sample and assess the mean squared error on this.

The most obvious way to do this may be to group the y feature into bins

Don't do this. Discretization wastes information for no good reason and introduces spurious steps.

Instead, I would calculate a . Econometric methods do this out of the box, and ML can often be taught to do this. (If nothing else, you can always resample residuals.) Here is how you can evaluate your prediction intervals. Wide prediction intervals indicate higher risk - how exactly this mapping works should be informed by your domain knowledge.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Sorry for late response - just wanted t say big thanks for sharing this - will definitely check out the prediction interval route (and your point n discretization makes a ton of sense)> – creekjumper Oct 17 '21 at 10:54
  • Also sorry just one follow up question. So your recommended process would be to calculate prediction intervals and than I could just scale the range to zero to 1 and this could be translated as a risk metric? Cheers – creekjumper Oct 17 '21 at 11:03
  • I would not just blindly scale the prediction interval, but rather include some domain knowledge, depending on what you are doing the exercise for. For instance, I do forecasting for retail replenishment. Suppose my expectation forecast is 1, my 95% quantile forecast is 5, and I can only replenish in packs of 8. No problem, because I will order one pack. Same if my quantile forecast is 6, 7 or 8. But if it is 9, then I would need to order *two* packs, so perhaps I want to look at this forecast a little more closely. – Stephan Kolassa Oct 18 '21 at 06:44