2

I am trying to create a graph representing a trend in binary value distribution. I thought of my own solution but it somehow doesn't feel right. Here's what I'm trying to convey with my graph:

I have a list of items that are either correct or incorrect, and their order is random. I am comparing a couple of scoring algorithms, where higher score means more certainty that the item is correct. So when the list of items is sorted by score, items at the high-score end of the list should have higher correct frequency.

My graph: It would be a 2-axis graph, where x-axis would be increasing score values and y-axis would be correct/incorrect values. The graph itself would be a series of points and their trend line. How steeply the trend line rises would indicate how good is the algorithm at guessing correct items.

The graph feels a bit weird to me, especially the 2 value y-axis, but I can't think of anything better. Is the graph okay or is there a better way to represent the data?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
querti
  • 23
  • 1
  • 3
  • How about putting time on the x-axis and aggregate score trends on the y-axis? – Mike Hunter Mar 27 '17 at 16:02
  • Time? I don't collect any time data, so I don't understand what you mean. – querti Mar 27 '17 at 16:09
  • Then how can you have a trend? Trends are defined by time, are they not? – Mike Hunter Mar 27 '17 at 16:10
  • Oh, I'm sorry, english is not my first language and I'm not very familiar with the terminology. What I mean is a linear trendline that you can create in excel and it can be created on basically any x-y graph – querti Mar 27 '17 at 16:13
  • No problem. What's the issue with this visualization as you've described it? Seems ok to me. – Mike Hunter Mar 27 '17 at 16:20
  • I guess the y axis seems kind odd to me. All the points will be either at very top or very bottom of the graph, so it almost seems kind of redundant. I just wondered if there was a better way to represent it. – querti Mar 27 '17 at 16:24
  • 1
    Sounds like you might want to look at the concept of lift: (https://en.wikipedia.org/wiki/Lift_(data_mining)) and (http://stats.stackexchange.com/questions/17119/lift-measure-in-data-mining). – Wayne Mar 27 '17 at 16:24
  • @wayne is definitely on to something there. Following on that suggestion, one approach might be to create deciles or ventiles based on partitioning the ranked scores into groups and examining the relationship between the groupings and the actual averages of the raw binary data. If there is "lift," then the relationship you seek should be represented. – Mike Hunter Mar 27 '17 at 16:29
  • Yeah, that definitely looks like something I was looking for. Thanks! – querti Mar 27 '17 at 16:34
  • Now that it looks like the direction you might want to go, I've moved my comment to an answer so you can accept it and so it's easier to find for others. – Wayne Mar 27 '17 at 17:02

2 Answers2

2

Sounds like you might want to look at the concept of lift. Here's another thread on Cross Validated that shows a lift chart in the accepted answer.

The Wikipedia entry first mentions a lift over random selection, but ultimately you'll probably want to look at a lift over the process that is currently used -- which probably isn't random, and therefore sets a higher bar.

Community
  • 1
Wayne
  • 19,981
  • 4
  • 50
  • 99
1

Since you are dealing with only binary data, I would do Receiver Operating Characteristic curve (ROC) analysis (which by the way is related to lift analysis).

ROC is a chart of sensitivity (y-axis) vs 1-specificity (x-axis) and is quite common in the medical literature.

In your case, you would construct your ROC out of n classifiers and you would label each point by the name of the classifier (like the image). The closer you are to the upper left, the better your classifier is (see wikipedia image below). Since you have a bunch of random classifiers, I imagine many will be below the 1:1 line

ROC Space from Wikipedia

user918967
  • 281
  • 1
  • 3
  • 13