7

I have a data set of N people, T items. Let's say N=100, and T=10.

Each person goes through the following exercise.

  1. She is shown 2 random items from the set of T=10, and ranks them as rank 1 and 2.
  2. She is next shown 2 more random items from the remaining 8 out of 10 items, and ranks as rank 1 and 2.

At the end, the data set is of size 100x10, where each row has 4 numeric entries (two of which will be 1, and the other two will be 2) and 6 empty entries.

My goal is to compare the 10 items against one another, and come up with an estimated rank value for a given item.

What is the best way to analyze such data ?

Thank you.

datddd
  • 71
  • 2

3 Answers3

2

I think that using a rating system, like Elo or Glicko, is a good choice.

  • Do the experiment with subject A, then repeat with subject B, subject C, and more.
  • Randomize matches' (i.e. comparisons) order and insert results in a rating system engine.

If you're interested in use more than in development, rankade, our free ranking system (for sports, games, items, and more) is another option. It allows matches with both 2 and 3+ factions, while Elo and Glicko works just for one-on-one (here's a comparison). Due to your items, it should be easy - and useful - comparing 3 or 4 types in each test.

Tomaso Neri
  • 766
  • 5
  • 18
0

Here one possible answer, although I imagine a better one exists:

Take the row means (ignoring blanks).

Oreotrephes
  • 213
  • 2
  • 9
  • 1
    I suppose an issue might be that, by chance, one of your items which, if you explored all possible head-to-head comparisons, would be ranked first more often, might in any given hundred head-to-head comparisons happen to come up against an especially strong contender, and thus have a lower row mean than it 'deserves'; maybe there is a way to test for this by subsampling your data? Also see Elo systems for a way to deal with this issue. – Oreotrephes Mar 18 '15 at 11:41
  • Also see http://stats.stackexchange.com/questions/89012/aggregate-ranking-from-multiple-ranked-lists and http://stats.stackexchange.com/questions/71297/converting-a-list-of-partial-rankings-into-a-global-ranking?rq=1 – Oreotrephes Mar 18 '15 at 12:02
  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. – Andy Mar 18 '15 at 12:03
  • I actually meant it as an answer, not a question. Maybe a bad answer, but an answer. I'll rephrase. – Oreotrephes Mar 18 '15 at 12:06
0

This might be an odd approach but logistic regression might be useful. For example, if person 1 compared items 1vs2 and 3vs4 and person 2 compared 2vs3 and 5vs6, and if the lower # items were always rated as "2", the data could be entered in R as:

data.frame(
T1=c(1,2,3,4,2,3,5,6),
T2=c(2,1,4,3,3,2,6,5),
Y =c(1,2,1,2,1,2,1,2) 
)

All 3 variables are categorical. You can predict the probability of getting a "2" in Y from T2, controlling for T1. The p-values and standard errors will be meaningless because nesting was not taken into account but the estimates could be useful.

Then, you can order the Items based on the probabilities that they will be ranked higher than another item.

Hotaka
  • 1,114
  • 7
  • 13