4

Background

I'm currently reading the book Data ming: practical machine learning tools and techniques and came across the following ROC curve: enter image description here The text corresponding to this figure:

Method A excels if a small, focused sample is sought; that is, if you are working toward the left-hand side of the graph. Clearly, if you aim to cover just 40% of the true positives you should choose method A, which gives a false positive rate of around 5%, rather than method B, which gives more than 20% false positives. But method B excels if you are planning a large sample: if you are covering 80% of the true positives, method B will give a false positive rate of 60% as compared with method A’s 80%.

Question How does a ROC curve relate to sample sizes? (in the context of the text mentioned above)

KingBoomie
  • 683
  • 4
  • 7
  • 20

1 Answers1

4

A ROC graph does not relate to sample size directly; small or large samples can have the same ROC graph. The methods (classifiers) used to create the ROCs have different performances and these performance characteristics can relate to the samples size of the results as a proportion of the overall sample examined.

What the author tries to convey is that the performance of method A is "preferable" if we aim detect a subsample of sample with true positives and we want to strongly limit our number of false positive instances (e.g. patients suggested to have an invasive medical procedure). Similarly, if we want to ensure that vast majority the true positive instances is detected (even to the expense of having a substantial number of false positives), method B would be preferable (e.g. patients who are suggested to take a relatively inexpensive screening test).

Different applications may require an exchange between high recall and high precision. As @Frank Harrell correctly notes, if we have a well-defined utility function we can use to maximise our expected utility. If we do not have one (as in the case of this arbitrary ROC graphs) we can pick a methodbased on an overall goal (e.g. medical diagnostics (method A) vs medical screening (method B)).

usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • 3
    If you examine decision theory you'll see that none of this matters. For example, if you use maximum likelihood estimation to obtain probability estimates, these plug directly into utility functions so that you can make the decision that optimizes expected utility. ROC curves do not play a role in that. – Frank Harrell Apr 01 '18 at 12:01
  • @FrankHarrell I agree what what you say how does it relate to explaining the passage about ROC to the OP? Clearly we use utility functions and I specifically mentioned the example with the patients to allude to that. – usεr11852 Apr 01 '18 at 12:03
  • To do so you need to completely ignore the ROC. – Frank Harrell Apr 01 '18 at 14:26
  • (Disclaimer: I admire your work) Given that the OP asked how a statement *based on the ROC presented* was derived, we have to answer that by making a mention the ROC. Please note that I (nor the OP) mention anything about the AUC-ROC, the focus is on what the curve's shape convey and how this relates to the text quoted. – usεr11852 Apr 01 '18 at 14:49