10

Context:

I have two data sets from the same questionnaire run over two years. Each question is measured using a 5-Likert scale.

Q1: Coding scheme

At the moment, I have coded my responses on a [0, 1] interval, with 0 meaning "most negative response", 1 meaning "most positive response", and other responses spaced evenly between.

  • What is the "best" coding scheme to use for the Likert scale?

I realise that this might be a bit subjective.

Q2: Significance across years

  • What is the best way to determine whether there is statistically significant change across the two years?

That is, looking at the results for question 1 for each year, how do I tell if the difference between the 2011 result and the 2010 result is statistically significant? I've got a vague recollection of the Student's t-test being of use here, but I'm not sure.

Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
Mac
  • 203
  • 2
  • 6

2 Answers2

7

1. Coding scheme

In terms of assessing statistical significance using a t-test, it is the relative distances between the scale points that matters. Thus, (0, 0.25, 0.5, 0.75, 1) is equivalent to (1, 2, 3, 4, 5). From my experience an equal distance coding scheme, such as those mentioned previously are the most common, and seem reasonable for Likert items. If you explore optimal scaling, you might be able to derive an alternative coding scheme.

2. Statistical test

The question of how to assess group differences on a Likert item has already been answered here.

The first issue is whether you can link observations across the two time points. It sounds like you had a different sample. This leads to a few options:

  • Independent groups t-test: this is a simple option; it also does test for differences in group means; purists will argue that the p-value may be not entirely accurate; however, depending on your purposes, it may be adequate.
  • Bootstrapped test of differences in group means: If you still want to test differences between group means but are uncomfortable with the discrete nature of dependent variable, then you could use a bootstrap to generate confidence intervals from which you could draw inferences about changes in group means.
  • Mann-Whitney U test (among other non-parametric tests): Such a test does not assume normality, but it is also testing a different hypothesis.
Community
  • 1
Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
  • so, in a nutshell, you see nothing wrong with what I proposed (equal distance coding, t-test significance test), other than there are other options that may be more accurate? – Mac Apr 12 '11 at 04:51
  • @Mac In my opinion, coming more from an applied perspective, it's a simple, easy to understand, easy to communicate, and generally reasonable approach. However, it's often worth considering measuring constructs of interest using scales rather than individual items. – Jeromy Anglim Apr 12 '11 at 05:20
  • agreed. However, I believe for what I need this will do. Many thanks! – Mac Apr 12 '11 at 06:07
  • just to note that the t test can be quite sensitive to differences in variance, so that would probably be something to check before you make your final decision – richiemorrisroe Apr 12 '11 at 08:21
  • @Mac /cc @richiemorrisroe In my view, apart from the homoscedasticity assumption (which is somewhat circumvented with Welch's t test), the problem is mostly with asymmetric response distributions (ceiling or floor effect), which often arise when using Likert items. – chl Apr 12 '11 at 10:47
3

Wilcoxon Ranksum Test aka Mann-Whitney is the way to go in the case of ordinal data. The bootstrapping solution is also elegant albeit not the "classic" way to go. The Bootstrapping method might also be valuable in case you aim for other things like factor analysis. In case of regression analysis you might chose ordered probit or ordered logit as a model specification.

BTW: If your scale has a larger range (>10 values per variable) you might use the results as a metric variable, wich makes a t-test a safe choice. Be adviced that this is a little dirty and may be considered devil's work by some.

stephan

  • 1
    Could you expand on how would bootstrap provide a more interesting approach for factor analysis? – chl Apr 12 '11 at 20:04
  • I would be interested in learning more about *why* the Mann-Whitney test would be favored over a t-test here. – whuber Apr 13 '11 at 02:35