5

What is the best inter-rater agreement test for Likert scale type questions? As far as I see, Cronbach's $\alpha$ is for internal consistency, or it shows how good items are related to describe the main question. I want to measure inter-rater agreement. Is Cronbach's $\alpha$ the right metric for doing this?

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
user2868483
  • 71
  • 2
  • 3
  • 2
    Alpha is not the right tool for this. One method is weighted kappa, but there are others as well see [this page](http://www.john-uebersax.com/stat/kappa.htm) for a lot of resources – Peter Flom May 23 '14 at 19:07
  • 1
    A very nice article which might be useful is Hallgren (2012): http://www.tqmp.org/Content/vol08-1/p023/p023.pdf – Patrick Coulombe May 23 '14 at 19:26
  • Paper was very helpful (ICC is the right metric). Thank you – user2868483 May 24 '14 at 11:46

1 Answers1

3

Krippendorff's alpha, originally developed in the field of content analysis, is well-suited for dealing with ordinal ratings such as Likert-scale ratings. It has several advantages over some other measures such as Cohen's Kappa, Fleiss's Kappa, Cronbach's alpha: it is capable of dealing with more than 2 raters; it is robust to missing data; and it can handle different types of scales: nominal, ordinal, etc.

It also accounts for chance agreements better than some other measures like Cohen's Kappa.

Calculation of Krippendorff's alpha is supported by several statistical software packages, including R (by the irr package), SPSS, etc.

Below are some relevant papers, that discuss Krippendorff's alpha including its properties and its implementation, and compare it with other measures:

  1. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89.

  2. Krippendorff, K. (2004). Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Human Communication Research, 30(3), 411-433. doi: 10.1111/j.1468-2958.2004.tb00738.x

  3. Chapter 3 in Krippendorff, K. (2013). Content Analysis: An Introduction to Its Methodology (3rd ed.): Sage.

There are some additional technical papers in Krippendorff's website.

ynagar
  • 123
  • 1
  • 8