4

I attempted to run a study where people had to rank order 12 statements from best to worst. However, it was extremely messy and difficult for people to rank that many items. Now, my professor and I have decided instead to show people 2 statements at a time and just rate which one is higher. Since it would be way too much to ask people to make every possibly comparison within the set of 12, we'd give each person a subset, maybe 10 from the full set of possible pairs.

I've tried to look up methods to analyze them, but everything I've checked into is not quite right. Some require each person to have rated the full set, like Borda's method or other voting ranking systems. Others don't seem take into account the within-subjects factor, like the Bradley Terry model. And searching for the issue of ranking usually gives me ways to test whether two groups are different, as opposed to one population. For some of these though, I'm not sure if they test/model can't do what I need or if I just don't understand how it is applied. Eventually I'll have the data in R, so for each participant I'll have a set of 10 pairs and how they ranked each one; I haven't figured out how to apply the models to that, though the BradleyTerry2 package seems closest.

Can anyone help me figure out if this analysis is possible before I run the study? Will I be able to get a single, ordered list of the 12 statements with the method I have sketched out? Thank you in advance!

Bofstein
  • 61
  • 3
  • There is no perfect Pepsi. There are only perfect Pepsi's. [link](https://www.ted.com/talks/malcolm_gladwell_on_spaghetti_sauce?language=en). If you have demographic differences in the rank-givers, then you might be able to not only give ensemble order, but split it by demographic or by demographic interactions. You should also consider replication, so give the same choice, reversed order, to each ranker. – EngrStudent Apr 25 '16 at 14:41
  • 1
    It isn't completely clear just what you mean by "analyze them" or "this analysis." What is this "single, ordered list" intended to represent? – whuber Apr 25 '16 at 14:42
  • By analyze them, I mean put them into an order and test against the null hypothesis (i.e., that there is no true order and thus the order isn't significantly different from random choosing). E.g., if there were only 3 statements (A, B, and C), and there was no defined order, each would win about 50% of the time against the others. A statistical test of the rank would be non-significant. On the other hand, if there was a true order A > B > C, maybe we'd find that A wins 90% of the time against C and 60% of the time against B, while B wins 70% of the time against C (and the rest are inverse). – Bofstein Apr 30 '16 at 05:48
  • I am now working with a statistics professor who studied this in his dissertation and though I still din't understand the procedure behind it, he is helping me analyze it. We are using the Bradley Terry method for the basic rank, and he is working on adding in a within-subjects component. Basically each step is a logistic regression predicting the likelihood of choosing each statement over the next one, and we can look at the confidence intervals of each rank to see if they are statistically different from one another. – Bofstein Apr 30 '16 at 05:54

1 Answers1

-1

If you're interested in use (more than in development), you should give a try to rankade, our ranking system. Rankade can manage small to large playing groups (composed by players or 'items', as per your needs), and it features rankings (and stats, and more). It doesn't cover all your tasks, maybe, but it should be a useful tool to reach your target (rank order 12 statements from best to worst, both with global and partial rankings).

Rankade's algorithm (that is not in public domain, for the time being - here's a comparison between most known ranking systems - but rankade is free and easy to use) can manage, within different scopes, any kind of match. Due to faction structure, you can record outputs for 1-on-1 comparison, or for 2+ items as well (even mixing both kind of 'matches'), while Bradley-Terry model or other ranking system (like most known Elo or Glicko) don't. It's surely true that it's extremely messy and difficult for people to rank that many (twelve!) items, but a multiple comparison (e.g. 3-7 statements) should be suitable and useful, in your work.

Tomaso Neri
  • 766
  • 5
  • 18
  • 3
    Thank you for your reply. In its current state it reads like an advertisement and appears to offer little information. Would you be able to enhance it to say something about how your method works? – whuber Apr 25 '16 at 14:25
  • 1
    Rankade builds a ranking allowing 'matches' with multiple comparison, so I think it should be useful in @Bofstein needs. I just did some edits, indeed, thank for your hints. – Tomaso Neri Apr 25 '16 at 14:39
  • 1
    -1 Because of the advert-oriented nature, because of the lack of transparency in a scholarly field, and because you imply rent-seeking behavior on mathematics, which is generally already in the public domain. – Alexis Apr 25 '16 at 17:39