Machine learning metric: essential accuracy vs. incidental accuracy

Question

I guess I could find the answer to my question if I knew the right Google search words.

If you use a better model in a classification problem then you will get a better accuracy (if you use that metric to judge). But even the "best" model will reach a ceiling, e.g. if the data is not telling you enough about the problem.

Is there a way to somehow separate the "essential accuracy" (the max accuracy the best model could achieve) from the "incidental accuracy" (the accuracy you happen to achieve due to your potentially imperfect model)?

Which search terms would I have to use to learn more about this topic?

Thanks! Christian

[How to know that your machine learning problem is hopeless?](https://stats.stackexchange.com/q/222179/1352) may well be a duplicate. — Stephan Kolassa, Feb 25 '20 at 16:24
Does this answer your question? [How to know that your machine learning problem is hopeless?](https://stats.stackexchange.com/questions/222179/how-to-know-that-your-machine-learning-problem-is-hopeless) — Stephan Kolassa, Feb 27 '20 at 10:00

score 0 · Accepted Answer · answered Mar 15 '20 at 10:25

After reading more around this topic I believe that the paper Understanding predictive information criteria for Bayesian models answers my original question best and gives great references to further reading. It also clarifies terminology and vocabulary and gives some concrete examples and comparisons.

In the end, my conclusion is that my question was really around having a "measure of predictive accuracy" and comparing that to the "true data-generation processes" "measure of predictive accuracy". The paper shows what is possible and what is not.

Machine learning metric: essential accuracy vs. incidental accuracy

1 Answers1