Simultaneous Z-test for the equality of two proportions (binomial distribution)

Question

Is there a simultaneous test for the equality of two proportions when $k$ binomial properties are being tested?

Example:

Two groups of large size $n_1$ and $n_2$ are asked if they agree or disagree on 10 different questions. For any one particular question you can do a $Z$-test for the equality of proportion, but that proportion may differ from one question to the next.

For example: $p_1=(.23, .80, .03)$ and $p_2=(.28, .84, .08)$

Is there a way to investigate the assumption that the vector of proportions $p_1$ and $p_2$ from two populations are equal?

Answer 1

There is no simple test for what you are looking for, because you have to account for the unknown correlation of the 10 responses within a subject. Some options, many of which are usually not quite adequate, include:

1. Collapse the 10 answers to "at least one yes", and use a chi-square test - this might be reasonable for rare events

2. Collapse the 10 answers to "the number of yeses" and use Wilcoxon's rank-sum test - this might be reasonable for events with similar probabilities

3. Test each question separately with 10 different chi-square test, and then adjust for multiple testing. The simplest option is Bonferroni adjustment - multiply all your p-values by 10 (the number of tests) in this case. A better approach is a resampling based test for the maximum test statistic (or minimum p-value).

4. Use an appropriate generalized estimating equation model (GEE). There are multiple ways you could set it up depending what you are willing to assume.

5. I think you could use some clustered tests for surveys, but it is not straightforward to set up either.

Answer 2

Perhaps a hierarchical log-linear model would work. Picture your data as a 2 x 10 x 2 array: the first dimension is agree/disagree; the second dimension contains the questions and the third dimension indexes the two groups. You fit a series of models until you get one that fits. Thus

1. condition on the margins only. This assumes independence throughout. In R, the call would be loglin(x, list(1,2,3)).
2. Condition on the margins and the relationship between answer and questions. loglin(x, list(c(1,2),3). This allows relationships between the questions and allows for different numbers in the two groups.
3. Then fit the full model, loglin(x, list(c(1,2,3)).

If the two groups behave the same way, then model 2 would be non-significant. If not, then not -- and you would need model 3, which implies that the groups differ.

This does not account for the fact the questions are nested in subjects, as noted by @Aniko, but the log-linear model accepts the pattern of responses it gets and attempts to model it across both groups. If there are clear differences between the groups with respect to how they respond, this test should pick them up.

Another possibility would be to write down the likelihood under the null hypothesis of no difference between groups and the alternate. Then do a likelihood ratio test. If you are prepared to ignore correlations between questions, the likelihood would be that of a multinomial distribution.

If you are concerned about correlations between the questions, you might want to go for a latent variable model with categorical predictors and look at the difference between the groups using a SEM analysis. You would want a lot of data for this and a plausible reason for a latent variable model.