1

I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of items from a document); and I do not want measures which show how related these items are. The list of items is fixed.

Is there some non-parametric way of showing some form of agreement in the ratings between the respondents? I've seen a study which uses Kendall W, but there are a lot of ties and perhaps this is not the most useful way.

Just to clarify, I'm looking for some form of agrement measure for each item within the data, not specifically one item.

Cesare Camestre
  • 699
  • 3
  • 15
  • 28
  • 2
    Any reason to ask for a 'non-parametric' measure of inter-rater agreement. What's wrong with a weighted [Kappa](http://en.wikipedia.org/wiki/Cohen's_kappa), or some [ICC](http://stats.stackexchange.com/a/3546/930) or [generalizability](http://stats.stackexchange.com/a/64447/930) indices? – chl Jul 18 '13 at 10:59
  • The problem is the small sample size which tends to lead to issues with the assumptions of parametric measures. – Cesare Camestre Jul 18 '13 at 11:01
  • I note in the other post "Inter-rater reliability for k raters can be estimated with Kendall’s coefficient of concordance, W. When the number of items or units that are rated n>7, k(n−1)W∼χ2(n−1). (2, pp. 269–270). This asymptotic approximation is valid for moderate value of n and k (6), but with less than 20 items F or permutation tests are more suitable (7). There is a close relationship between Spearman’s ρ and Kendall’s W statistic: W can be directly calculated from the mean of the pairwise Spearman correlations (for untied observations only)." Wouldn't there be an issue with the ties – Cesare Camestre Jul 18 '13 at 11:03
  • @chl See also my clarification above. – Cesare Camestre Jul 18 '13 at 13:42
  • Hi @CesareCamestre, 3 years later, did you have a solution for this agreement measure? I have a similiar study (rate items on 1-5). – Bruno De Freitas Barros Feb 07 '17 at 02:44

0 Answers0