What are typical approaches to express the reliability of a categorical scale?

Question

I have categorical scales consisting of items with each three answer options: 1) very much; 2) somewhat; 3) never. What would be a suitable approach to express reliabilities for scales like this?

I am aware that previous questions have touched upon this topic. But I have not found any satisfying answer yet. For example, here, it was only suggested that "if you have unordered categorical data (i.e., three or more unordered categories; which you do), then you don't use Cronbach's alpha". However, no recommendation was given for what to do if you have three (ordered) categories. Any help appreciated!

Answer 1

Composite reliability can be thought of as a generalizing at coefficient alpha, that relaxes some of the assumptions. See, e.g. https://www.statisticshowto.com/composite-reliability-definition/.

You can calculate composite reliability for ordinal measures by doing confirmatory factor analysis of a polychoric correlation matrix. Example; https://journals.sagepub.com/doi/full/10.1177/2515245920951747

Alternatively, you can use item response theory, which thinks about reliability in a different way - I like https://www.researchgate.net/publication/232569066_The_new_rules_of_measurement. Summary here: https://www.rasch.org/rmt/rmt132e.htm .

IRT considers test information, rather than reliability - and there can be different values of information at different levels of ability (or whatever is being measured). E.g. a scale might be very good at differentiating people with mild depression from people without depression, but not good at differentiating people with moderate and severe depression -the reliability of the scale varies (and reliability can be calculated for any person who has taken the test).

However, if you're going to add up the items to get a score, you might as well use coefficient alpha.