Parametric tests and Likert Scales (Ordinal data) - Two different views

Question

Following articles reach quite different conclusion and I start to believe that there is no clear answer to this problem. The conclusions are below and the first author reacts on second author. My question here is, what approach is appropriate (given following situation) when we want to analyse Likert Scales in Social research, MANOVA fits our research design (two or more DV based on Likert Scale), we have N = 180 but these two contradictory opinions?

First Article:

Parametric statistics can be used with Likert data, with small sample sizes, with unequal variances, and with non-normal distributions, with no fear of ‘‘coming to the wrong conclusion’’. These findings are consistent with empirical literature dating back nearly 80 years. The controversy can cease (but likely won’t).

• Norman, Geoff. "Likert scales, levels of measurement and the “laws” of statistics." Advances in health sciences education 15.5 (2010): 625-632. 10.1007/s10459-010-9222-y

Second article:

(...)the researcher should decide what level of measurement is in use (to paraphrase, if it is an interval level, for a score of 3, one should be able to answer the question "3 what?"); non-parametric tests should be employed if the data is clearly ordinal, and if the researcher is confident that the data can justifiably be classed as interval, attention should nevertheless be paid to the sample size and to whether the distribution is normal.

• Jamieson, S. (2004). Likert scales: how to (ab)use them. Medical education, 38(12), 1217-1218. DOI: 10.1111/j.1365-2929.2004.02012.x

Answer 1

One way I approach this is to not take people's word for it, based on what appears to be either their beliefs, or precedent, but to try it out and see if (in your case) it matters in a way that you care about.

Here's a simple example: A 5 point Likert scale, with a uniform distribution. 100 people per group, and we'll do a two sample t-test. I'll repeat this 10000 times when the null hypothesis is true (i.e. there is no difference).

> mean(sapply(1:1000, function(x) { 
    t.test(sample(1:5, 100, TRUE), sample(1:5, 100, TRUE))$p.value 
  } ) < 0.05)

[1] 0.0499

It appears that I get a significant value 4.99% of the time. Given that I expect a significant value 5% of the time, it does not appear that violating the assumptions of normality and interval measurement has had any effect on my results - at least in terms of type I errors. (There might be power issues, of course.)

If someone has a specific criticism, you can investigate and see if it's an issue.

Here's another example: Now I have 5 people in one group, and 100 in the other.

>   mean(sapply(1:10000, function(x) { t.test(sample(1:5, 5, TRUE), sample(1:5, 100, TRUE))$p.value } ) < 0.05)
[1] 0.0733

Now I have a 7.3% type I error rate. This is probably enough to worry about.

What about 5 per group?

 mean(sapply(1:10000, function(x) { t.test(sample(1:5, 5, TRUE), sample(1:5, 5, TRUE))$p.value } ) < 0.05)

Now a 4.5% signifance rate - indicates a slight loss of power, but I prefer that (a lot) over an inflated type I error rate.

Answer 2

Some general comments:

I would be wary of over-reliance on blanket pronouncements like "everything is fine". In such situations it's often easy to come up with plausible counterexamples where everything is demonstrably not fine ... so in the absence of what would be effectively a proof of the robustness of the tool against almost any deviation from assumed distributional characteristics (e.g. robustness of level and power in the case of a hypothesis test) you can't assume as a matter of course that you're okay.

Similarly, I would be wary of blanket assertions that you're not fine. In many cases, the traditional procedures perform well enough on a wide range of cases that it may be perfectly reasonable for your situation.

---

One thing to keep in mind is the distinction between a Likert item (a single question with an ordinal response) and a Likert scale -- properly, Likert scales are composed of multiple items.

If you have a response that's a Likert scale, you have already treated its components as interval when you added the individual items that make up the scale. There's no reason you should step back from a choice you already made.

I agree with Jeremy that you can figure out quite a bit about how sensitive your tests are (significance level and the power properties) to population deviations from the assumptions, and this kind of understanding of "how bad things might be" is important to have. Simulation can be a very important aid in understanding the properties of our inferential tools.

The issue lies somewhat in choosing cases (population distributions) to look like that tell you something about the situation you're in. Rather than obsessing over having a population exactly like your sample (as people sometimes tend to do), I think it's useful to consider a few simple possible population assumptions that might reasonably have produced something like your sample. One reason is that by focusing too much on the specific sample characteristics you're going to tend to be replicating the noise in the sample rather than the underlying structure. The other thing is if such knowledge is to be at all generalizable (so you gain understanding that can be applied to other similar situations), you want a less specific approach. I tend to think of it as trying to investigate the neighborhood of distributions that might have produced a sample a bit like this and then push things about a bit -- that kind of investigation takes a bit longer but tends to lead to information you can generalize.

Knowing how some proposed procedure performs under general situations -- say a uniform distribution on the scale (as discussed in in part of Jeremy's answer) is even more basic information (it's unlikely to look much like any sample you see, but it can provide information that's useful to understanding the behavior -- if you're dealing with some data you can think about questions like "what would happen if the distribution was less skew and more evenly spread?" and that uniform is a very important anchor point, since it's as evenly spread as you can go). So to my mind that just the sort of thing to look at when you first start trying to analyze (say) Likert scale data and (say) MANOVA. I think that sort of general information is fundamental to using these tools sensibly across a variety of situations. Those give you context into which more specific analyses can be placed.

[Note, however, for a Likert scale to end up looking uniform would require a pretty bizarre distribution across the Likert items that make it up. So it would be a rather extreme case to consider if we're looking at a scale composed of more than one or two items.]

One could, if one chose, supplement such understandings by literally resampling the data (if your sample size is not small) -- i.e. bootstrapping. Now you're looking at a population that looks exactly like your sample, which (naturally) incorporates as much of the "noise" as it's possible to include into the signal and is harder to generalize from. Nevertheless it also provides a natural anchor point for distributions that could generate your sample and so provides relevant information.

Answer 3

"Findings are consistent with empirical literature dating back nearly 80 years" is a vey bad, bad reason. Latest empirical literature suspected of questionabile research practice.

Normal distribution is not requested for the use of non-parametric tests: "nonparametric statistics make no assumptions about the probability distributions of the variables being assessed" (Wikipedia, and also any textbook of basic statistics)

For suggestions on analysis, take a look at this blog: http://statisticscafe.blogspot.it/2011/05/how-to-use-likert-scale-in-statistical.html?m=1

Better than nothing.