17

I'm looking for way to visualize subjective rankings, separate from my non-parametric tests.

I've asked 12 participants to rank 8 different items according to different subjective criterion (separate rankings for each one). For any individual set of rankings, I'm looking for a good way to visualized the high-level trends of the rankings.

I've tried both bar and radar plots on the average rankings, and I've seen one other person use a scatter/balloon plot over the number of responses per ranking, but I'm not quite sure what conveys the best overview. Either I can use the 8 mean rankings, or the the 8 counts of each ranking per item.

Edit:

For example: Each column is an item, each row is a person's ranking of each of the eight items. Not a particularly strong agreement in this example, but in general would like to understand the best way to convey the overall trends.

                        Item:
            A   B   C   D   E   F   G   H
Rater:  
  1         6   8   1   7   3   4   2   5
  2         1   3   8   7   6   5   2   4
  3         5   8   7   6   1   4   2   3
  4         5   8   7   6   4   2   1   3
  5         1   2   8   7   4   3   5   6
  6         1   7   8   5   6   2   4   3
  7         5   1   8   4   7   3   6   2
  8         4   2   8   7   6   1   5   2
  9         6   3   8   4   7   1   5   2
  10        3   2   8   7   4   1   5   6
  11        2   3   7   8   1   5   4   5
  12        8   5   6   7   2   3   1   4
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Jesse
  • 171
  • 1
  • 1
  • 4
  • It might help if you could list some of your data. Also, what software are you likely to use? – gung - Reinstate Monica Jun 23 '12 at 17:25
  • I've added an example matrix, and I'll likely be plotting in Excel. – Jesse Jun 23 '12 at 17:33
  • Are you trying to display the item ranks or the inter-rater agreement (or both)? – whuber Jun 23 '12 at 19:59
  • 1
    I was asking specifically about the item ranks, but if there's a good way to show inter-rater agreement other than something than Kendall's W, all the better too. – Jesse Jun 23 '12 at 20:39
  • You could plot the empirical CDF of each column. This would look something like a (whole mess of) ROC curves. Not sure what would jump out if you did this, though. – shabbychef Jun 24 '12 at 04:27

4 Answers4

8

I've done something similar for visualizing similar rankings. The method I used gave me a quick snapshot of how the rankings related-nothing more. My solution used Excel 2010 sparklines to create a small-multiples view of the rankings (this can be done in other Excel versions, but it takes a bit more work). Also, I generally use Excel's Table functionality just to speed things along.

  1. Transpose your matrix so the rows hold all the values per rated item.
  2. If rank 1 is best, invert your ranks so 1=8,2=7... this just helps you visualize the chart's columns (rather than blank space) as better.
  3. Sum the results of each question's raters. Then sort by this total. This will put the item with the best overall scores first and worst overall scores last. This ranking will help you visually rank the charts when the overall pattern may not be obvious (as was the case in your sample data.
  4. Insert Sparklines using your ranking data (without the totals column) next to your data table. Visually, the more/bigger bars there are, the better the overall ranking.

There's nothing particularly analytical about this approach, but it's a pretty quick way to visualize the data.

Subjective Rankings

Note, if you don't have Excel 2010, you can create stripped down, cell sized column charts for each row that look about the same. Or, you can use a third-party add-on to create them.

EDIT: Table and Chart utilizing Gung's suggestion for average measure. As pointed out, since the scale is similar, it was added to the chart as an additional point of comparison (I used gray to help differeniate it from the raw data plots).

enter image description here

dav
  • 1,551
  • 2
  • 15
  • 23
  • 3
    +1, this is a really nice, simple solution for this situation. One small suggestion I would make is to use *averages* instead of sums in step #3. This won't really change anything, but puts the total back into the original scale, which can make the interpretation of the value easier. – gung - Reinstate Monica Jun 26 '12 at 14:44
  • @gung, thanks for the feedback. I think using the averages really works well (as above). I never liked using the totals, but hadn't put too much thought into a good alternative. – dav Jun 26 '12 at 18:59
2

I would do this in the form of a bumps chart. It looks pretty cluttered, so I would work with the colors to try to help this out.

The sparklines @dav suggested don't really allow a comparison between products.

enter image description here

Jon Peltier
  • 381
  • 3
  • 6
  • I think this chart would be excellent if the dot sizes were scaled in reverse order from 8 (smallest) to 1 (largest). I think the trends might jump out more that way. – Brandon Bertelsen Sep 22 '17 at 07:04
2

I know this question is seven years old and see the poster is AWOL, but I do appreciate a question that includes data...

It sounds like product rating trends are the issue rather trends of the individual raters. In which case, you can get a sense of the high-level trends by plotting the means and confidence interval for each product.

enter image description here

At least we can see that C and D seem worse than the others (assuming 8 is the worst ranking). Maybe A, B and C have more disagreement than the others.

If individual raters are important in convey, you could use different marks and colors for each rater's dots.

xan
  • 8,708
  • 26
  • 39
1

Dot plots can be very useful for this type of analysis.

This is what I came up with your data. You can also add connecting lines.

This is what I came up with your data. You can also add lines

I would also think about adding a more prominent series to show the average for each option.

Some links:

http://peltiertech.com/Excel/Charts/DotPlot.html

http://peltiertech.com/WordPress/charting-survey-results/

guitarthrower
  • 111
  • 4