10

I am using a survey that contains several questions about various dimensions of performance for policy research institutes. Here, performance in the policy arena is unpacked into things like:

  • quality of research,
  • overall ability to engage with policy stakeholders,
  • quality of recommendations, overall support to, and influence on, policymakers,
  • etc.

Each question has a typical 5-level ranking (i.e., ordinal response from "strongly disagree" to "strongly agree", or "very bad" to "very good").

I am thinking of pursing either of two options:

  1. Creating a sort of composite measure, where all dimensions of performance are aggregated together so as to have one composite variable of "performance". I could then use this composite variable as the dependent variable (perceptions of performance).

  2. Combining 20 of these questions to create a performance index. For each dimension, responses range between 1 and 5. So total scores on the index would thus range from a minimum of 20 up to a maximum total of 100 points. This index could also be used as a dependent variable. Or perhaps only for descriptive statistics.

Does this make sense? Any advice and reference would be greatly appreciated.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Philippe
  • 101
  • 1
  • 1
  • 4
  • 1
    Welcome to the site, @Philippe. I don't understand the difference between your conception of "composite measure" & "index". To me they both seem like putting individual question responses into one new variable. Can you clarify that? – gung - Reinstate Monica Jun 18 '13 at 16:50
  • 1
    You can't sum ordinal items; if you sum them it means they are scale and not ordinal for you. – ttnphns Jun 18 '13 at 17:49
  • 1
    @ttnphns This is done all the time is areas like psychological research (but much more widely than that); that's after all exactly what a [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) is - a sum of items from ordinal items to produce a composite scale. It makes an implicit assumption (that the category intervals are equispaced), but in practice it's done when there's no particular reason to think it's the case. If there are many related items it seems to work quite well. – Glen_b Jun 19 '13 at 01:00
  • @Glen_b. `It makes an implicit assumption ... when there's no particular reason to think it's the case` The assumption of (equi)intervality is usually made _without_ grounds, indeed, and it's normal. But that simply means you don't take items as ordinal, when you do it. Likert (summative) construct is the constellation of interval-scaled items. – ttnphns Jun 19 '13 at 06:25
  • Thanks for your useful comments. @gung I think I was seeing composite measure as still being an ordinal variable (5-point ranking) and therefore amenable to logistic regression, and index as scale (20 to 100) to be tested with OLS. But in effect this is putting individual question responses into one new variable. – Philippe Jun 19 '13 at 15:10
  • @ttnphns Would you say that the idea of combining question responses into a new variable to be abandoned? – Philippe Jun 19 '13 at 15:17
  • @Glen_b Thanks. Would you then use OLS on the composite scale? Any key, understandable, literature on creating an index/composite measure? – Philippe Jun 19 '13 at 15:18
  • 1
    You may wish to search on this site for "survey validity". There are about 60 articles here. If you add the questions together, the issue is whether you have something meaningful at the end. In many fields assumptions are violated due to the robustness of the analysis to such violations. The issue of meaningfulness will have to rest with you, but you can check for the internal consistency of the items you sum by running one of many estimates (Cronbach's alpha is one example). You could also determine if your construct is unidimensional. – doug.numbers Jun 19 '13 at 17:01
  • @Phillipe I'm not the right person to point you toward literature on this; I'm more of an interested bystander on the whole area relating to Likert-scales and similar instruments. I try to understand (not least because as a statistician with a little experience and ongoing contact with academia I am often asked questions by people in the areas that tend to use such instruments), but I really don't know much relevant literature. I hope to learn something (we may have already exhausted my knowledge of any relevance to your question). – Glen_b Jun 19 '13 at 23:17
  • Can you still create a composite variable if a "don't know" response option is included? – Philippe Jun 20 '13 at 13:45

2 Answers2

4

Yes, both your points make perfect sense, and are indeed a standard practice - at least in an area called psychometry.

But I cannot agree with the title question: it is not always valid to ordinal variables. In general case one cannot add nor subtract values measured on ordinal scale and hope, that the result would be independent from arbitrariness that come in the notion of ordinal variable.

Ordinal variable is a special case of interval variable; one in which we cannot say how far away from each other are adjacent levels of the variable. For instance, the education (which in many contexts is a valid ordinal variable) can be measured in 3 levels:

  • Primary education
  • Secondary education
  • Higher education

These 3 levels are usually mapped internally into numeric values "1", "2" and "3" - but this mapping is completely arbitrary. One can equally well map these levels as "1", "10", "100", or "8", "12", "17" (the last example would be a rough estimate of years of education) or employ the procedure from the Witkowski's paper. All statistical procedures that are designed for ordinal variables are invariant with respect to any injective function applied to the values associated to the levels. Imagine now, that we asked the subject to state the education level of mother and father. And now we want to build a parents' education index - by simply averaging parents' education level.

Now the outcome will become highly dependent on the mapping done between education levels and numbers, that represent them internally. For the most typical case ("1", "2" and "3") the process of averaging yields the same level "2" if one parent has Primary education and the other has Higher education, and if both parents have Secondary education. This feature might be correct, or might not, depending on how well the assigned numerical values represent the actual value each education has in our view.

The typical 5-level ranking you mentioned (a.k.a. Likert scale) was specially crafted in such a way, that the semantical distance between consecutive levels is kept roughly constant. Because of this property, such variables can be classified as interval, hence we can proceed with addition (or arithmetic mean, or any other mathematical manipulations).

Adam Ryczkowski
  • 2,092
  • 2
  • 18
  • 33
3

I assume here that your study's requirement is something along the lines of:

Given the ordinal responses to n questions for each candidate (a policy research institute in your case), rank/sort order the candidates by functionally combining the n-dimensional response tuple's elements into a score/metric.

Then perhaps you can look into the work of Wittkowski et al. (2004) and the references therein, on combining multiple ordinal variables for scoring.

Reference:

Sven Hohenstein
  • 6,285
  • 25
  • 30
  • 39
stochastically_metaskeptical
  • 31
  • 4