Measures of association between nominal variables with the same categories (paired nominal data)

Question

Scenario: I am conducting a study in which I am comparing a ‘gold standard’ assessment protocol with an experimental assessment protocol for a health variable of interest, over a common set of study participants. Observations will be paired, which is to say, for each participant in the study, I will have observations from both the ‘gold standard’ and experimental assessment protocols. Both protocols will generate assessments using a common nominal categorical variable (which takes the same three possible values, which I’ve coded as 0, 1, and 2 in the following example).

Analytic topic of interest: I would like to be able to meaningfully characterize the extent to which the results of the experimental assessment protocol correspond (i.e. agree) with those of the ‘gold standard’ assessment protocol.

Statistical question(s) of interest: What measure(s) should I consider using to characterize the level of agreement across the two protocols? Would a measure of inter-rater reliability, such as Cohen’s Kappa, be appropriate here? Are there other measures that I should consider? Measures that can also be associated with a hypothesis test and p-value would be ideal. As a partial aside, I’ve had colleagues propose that I use chi-square to test for independence of observations from the two protocols and Cramer’s V to characterize levels of agreement, but I am concerned that the paired nature of the observations would contraindicate those measures. Is that concern misguided?

I will have on the order of 1000 study participants. My data are currently structured as illustrated in the following sample table, and I expect to use R or STATA to run analyses.

Suggestions (or pointers) greatly appreciated.

subject Protocol1 Protocol2
1       2         2
2       0         0
3       2         1
4       0         1
5       2         1
6       1         1
7       1         0
8       0         0
9       0         1
10      1         2
11      2         1
12      2         0
13      2         0
14      2         2
15      0         1
16      0         0
17      0         2
18      2         0
19      1         1
20      0         0

Answer 1

The most natural and intuitive measure of association/similarity between two nominal variables with the same categories is Dice coefficient which, for nominal data, is equal to the average co-occurence aka average match occasions akа relative agreement size.

Here discussing similarity measures between individuals when attributes are nominal I explained it - how it can be computed from dummy data or from counting co-occurence instances. Presently we want to compute Dice between variables, not individuals. This situation is simply that one transposed. Let us borrow the example data from that linked answer and compute Dice between columns A fnd B rather than between every two rows. For speed & economy, we'll use co-occurence or match counting approach, not dummy data approach. So, simply flag, in each row, whether the two variables codes match (1) or doesn't (0). Then sum and divide by the number of rows.

ID   A    B       Match
1    2    1         0
2    1    2         0
3    3    2         0
4    1    1         1
5    2    1         0
                   ---
                Sum=1
                Dice = 1/5

Equivalent way to compute Dice will be through AxB frequency crosstabulation. Sum the diagonal frequencies (the trace) and divide by the total table frequency.

Crosstab        B
             1  2  3
             _______
          1| 1  1  0
        A 2| 2  0  0
          3| 0  1  0
               trace=1, total=5
               Dice = 1/5

The association measure is the matching association/similarity measure between nominal variables. I.e. it is based on the obvious fact that since we have two nominal variables with the same categories, the ground of association is only whether two values in a row are same or not same.

Note please that a number of other popular measures for binary data, when applied to dummy data (and nominal data are representable as dummy data) are linearly related to Dice.

Measures based of crosstable Chi-square statistic (Cramer's V or Phi coefficient) won't do because they treat boosted count in any cell as the sign of that association is present. But we are interested only by boost in diagonal cells because they are "matches".

Cohen's kappa is equal to $[Dice-Pr(e)] / [1-Pr(e)]$, where Pr(e) is the hypothetical value of random relative agreement (whereas Dice is the observed relative agreement), $Pr(e) = \sum(R_iC_i)/N^2$, where Ri and Ci are marginal counts in row i and in column i and N is the total count.