4

I am doing peer review on a paper submission for a journal, and I am somewhat out of my depth.

The authors are developing a psychometric scale. They did a pilot study with 45 items. They measured Cronbach alpha and found that it is 0.89 overall.

Then they found that six items have Cronbach's alpha if item deleted of 0.89 or very similar (between 0.89 and 0.893). They show a table with all six. They then write

A number of modifications were made according to what Table 2 suggested

And go on to explain how the wording of each item was changed to make it clearer. Obviously, they didn't want to have items whose Cronbach alpha if deleted is the same as the overall Cronbach alpha.

Why are they doing this? Is this a common technique in constructing psychometric scales? Are there guidelines for doing it?

They also don't cite any literature about the method they followed to develop their scale. It does look sound overall (pilot study, exploratory factor analysis, then refining the questionnaire based on the results), but I'm not sure whether the decisions they make about details (such as the above change of six items) are justified. Is the knowledge how to make a scale so common that there is no need to cite sources?

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
rumtscho
  • 1,669
  • 13
  • 23
  • 1
    If the questionnaire involves Likert ratings, there might be a whole other can of worms to consider; see "[Factor analysis of questionnaires composed of Likert items](http://stats.stackexchange.com/q/2374/32036)" if you dare... – Nick Stauner May 14 '14 at 21:49
  • I am wondering if any they have shown any additional tests for reliability or determining strength of agreement, like concordance correlation coefficient , Interclass correlation coefficient etc? – user39768 May 14 '14 at 21:33
  • 1
    @NickStauner very informative link. But I must say that, considering the discipline (software engineering), I am already delighted that they bothered to do a factor analysis and calculate reliability at all. I can't imagine requiring them to address such issues before the article is accepted. Especially with their scale, which doesn't even claim to be generalizable (it is specific to their software product). – rumtscho May 16 '14 at 15:51

1 Answers1

6

Cronbach's $\alpha$ tends to increase with the number of questions. If after deleting an item your Cronbach's $\alpha$ is the same, then you haven't gained any reliability by having that item. All other things being equal we tend to prefer shorter questionnaires... so we drop the item. One possible issue here is that the "Cronbach's $\alpha$ if deleted" will tend to all change any time an item is dropped. So perhaps after dropping one item a different decision would be made as to which items to drop. Regardless, these sorts of approaches to Cronbach's $\alpha$ are sufficiently common in my field that I wouldn't expect for them to be cited (or for them to take the ardious approach I just described); nor would I expect anybody to cite the source of the Cronbach $\alpha$. However, these standards do vary depending on area.

Ultimately, I think your duty as a reviewer in this case would be to let the editor know which part of the manuscript you feel goes outside of your expertise and focus on providing comments on the parts you can provide insight into. Alternatively, if you haven't been sitting on this for too long you might request that the editor assign this paper to another reviewer. Regardless, those issues are probably left to a different SE site.

russellpierce
  • 17,079
  • 16
  • 67
  • 98
  • So, why didn't they delete items which *lowered* Cronbach alpha? Or is it implicit that there were no such items, and they would have deleted them too, if there had been any? Or is it mathematically impossible for an item to lower the overall Cronbach alpha? – rumtscho May 16 '14 at 15:47
  • $\alpha$ if deleted can be lower if deleting the best item, but one would only want to delete items with higher $\alpha$ if deleted, following classical test theory, according to which multidimensionality is inherently bad. I may be caricaturizing it somewhat in saying that, but it is a natural consequence of devaluing items that reduce the average inter-item correlation. – Nick Stauner May 16 '14 at 16:34