Should I use an average to summarize ordinal data?

Question

I need to work out a "average" (for lack of knowing a better word) of ratings or perhaps I could call them labels. Basically, I have a list of words that have been rated 1 - 3 for difficulty. 1 being easy, 2 moderate and 3 difficult. The list is marked by 5 individuals. I need to use the words that are "mainly" rated as 2. So the individuals did not always agree on the difficulty. If a word had the following "score": 1, 1, 2, 3, 3 then it's safe to say that it's agreeably a 2.

If most reviewers rated a word as 3 then it's obviously not on average a 2. Basically, when taking 5 reviewers' rating for a word, what is the average rating?

Now it makes sense that I want to add it all up and divide it by 5. This will mostly give me decimal values such as 1.6 or so on.

Now my question finally is. If doing so, getting values such as 1.6, what do I make of it? Round it to the nearest integer and take that as the deciding rating? Is it as simple as that?

Answer 1

This is largely an issue for you to decide based on your theoretical assumptions about the data and what lies behind them. When you calculate an arithmetic average, you are assuming that the intervals are reasonably similar. (That is, you are implicitly stating that $3-2 = 2-1$ and $3-1 = 2\times (3-2)$.) If you believe that is a reasonable assumption, and others in your field (e.g., reviewers) are likely to agree with you, then it's fine. Using means with ordinal data tends to be more defensible when:

1. There are a larger number of ordinal levels (a rule of thumb is $\ge 12$);
2. the ordinal levels are composed of many components (e.g., ratings for many related questions are aggregated into a composite); and/or
3. the raters were instructed / tried to make the ratings equal interval.

It isn't clear to me that those hold in your case, but it is for you to decide.

You also should think hard about what you mean by '"mainly" rated as 2'. Again, that is for you to decide. However, I would not think of the set of ratings $\{1,1,2,3,3\}$ as "mainly" being $2$, despite the fact that the mean is $2$. I would interpret that as being a somewhat polarizing word, with some thinking it's 'easy' and some thinking it's 'hard'. But again, this is a theoretical issue for you to decide.

---

For what it's worth (almost certainly very little), if it were me, I would think your ratings were not amenable to be described by means. I think I would interpret '"mainly" rated as 2' as the majority of raters gave this word a 2. That is, I would select words that received $>50\%\ \rm ``2\!"$s.

By contrast, I suspect that you don't only want to select individual words '"mainly" rated as 2', but also want the entire set of selected words to be rated $\approx 2$. To check that aspect, I would feel more comfortable using the mean of all the ratings for all the selected words (or the mean of the words' means). At this point, you are averaging over many more ratings and I think the mean would be more defensible.

Answer 2

It is often the case with discrete data that the mean is non-discrete. That doesn't mean that it isn't a worthwhile statistic to report. A value of 1.6 could be interpreted as easy for most, but intermediate for some - pretty much what you see in your table.

If you want integers, you can calculate the median, which is the value such that half of all observations are below it and half above it. For example for kanarie, the median of (1,1,2,2,2) is 2, since half are below 2 and half are above (or equal).

Alternatively is the mode, which is just the most common value. It is often useful to get a feel for the overall population's choice.

Answer 3

You state that you "need to use the words that are 'mainly' marked as 2."

I'd argue that you need to define what 'mainly' actually means to you.

One way is to take the mean, as you indicate intuitively makes sense to you. If you do, then, indeed, if the mean is: 1.5 ≤ mean < 2.5. Then you might interpret it as 'mainly' 2.

An alternative is to use the mode, as @ssdecontrol suggested. You would, however, need to decide on how to handle a situation where there are two modes: e.g. (1,1,2,2,3) where the modes are 1 and 2. Would you consider this to pass your criteria? Or your example of (1,1,2,3,3) where the modes are 1 and 3. The mode is not 2 in this case, but the mean is exactly 2. Does it pass your criteria?

A third option is to use the mean of the mode. Where the pass criterion could be: 1.5 ≤ mean of mode < 2.5.

Answer 4

I think you have a few options depending on what you care about. You can go with the mean and see if it falls within some range of 2. This would be like averaging your votes. The advantage here is that you can check on how balanced people's votes are.

If you care more about how the votes land with at least 1/2 voting more or less at some point you would want to go with the median. The advantage of the median is that it is based more on vote distribution.

You might be interested in where the most voters voted. This would be the mode. You used the word mainly so you might be interested in this metric. Advantage is where the most voted, disadvantage is that as you get more options this can mean less.

One thing to keep in mind is also that you may have some controversial words where most voters are 1 and 3. In this case you should be careful since most would not say 2, but some methods would give you 2. I would thus suggest you use mode, in the case of two modes you would reject or use another method such as median.

e: You could also look into percent of votes that are 2. If it is above 50% then mainly people believe the question is a 2.

E: If you really want to add complexity you could also try to address the issue of voters not agreeing and having a bias. You would then need to do some transformation of the vote and do a mean/median.