10

I'm testing the independence of two variables, A and B, stratified by C. A and B are binary variables and C is categorical (5 values). Running Fisher's exact test for A and B (all strata combined), I get:

##          (B)
##      (A) FALSE TRUE
##    FALSE  1841   85
##    TRUE    915   74

OR: 1.75 (1.25 --  2.44), p = 0.0007 *

where OR is the odds ratio (estimate and 95% confidence interval), and * means that p < 0.05.

Running the same test for each stratum (C), I get:

C=1, OR: 2.31 (0.78 --  6.13), p = 0.0815
C=2, OR: 2.75 (1.21 --  6.15), p = 0.0088 *
C=3, OR: 0.94 (0.50 --  1.74), p = 0.8839
C=4, OR: 1.48 (0.77 --  2.89), p = 0.2196
C=5, OR: 3.38 (0.62 -- 34.11), p = 0.1731

Finally, running Cochran-Mantel-Haenszel (CMH) test, using A, B, and C, I get:

OR: 1.56 (1.12 --  2.18), p = 0.0089 *

The result from the CMH test suggests that A and B are not independent at each stratum (p < 0.05); however, most of the within stratum tests were non-significant, which would suggest that we don't have enough evidence to discard that A and B are independent at each stratum.

So, what conclusion is right? How to report the conclusion given those results? Can C be considered a confounding variable or not?

EDIT: I performed the Breslow-Day test for the null hypothesis that the odds ratio is the same across strata, and the p-value was 0.1424.

rodrigorgs
  • 203
  • 1
  • 2
  • 6
  • 3
    Did you not perform the Cochran-Mantel-Haenszel test precisely because the evidence for an odds ratio different from one might be weak for each stratum considered individually, but strong for all considered together? – Scortchi - Reinstate Monica Feb 28 '14 at 21:32
  • I performed CMH because I wanted a single, unified answer, and I wanted to make sure that the effect observed between A and B was not due to C. Am I on the right track? Should I report the statistics for individual strata? – rodrigorgs Feb 28 '14 at 22:24

1 Answers1

11

The first test tells you that the odds ratio between A and B, ignoring C, is different from 1. Looking at the stratified analysis helps you decide whether it's all right to ignore C.

The CMH test tells you that the odds ratio between A and B, adjusting for C, is different from one. It returns a weighted average of the stratum-specific odds ratios, so if these are $<1$ in some strata and $>1$ in others, they could cancel out and erroneously tell you there is no association between A and B. So we must test whether it is reasonable to assume that the odds ratios are equal (at the population level) across all the levels of C. The Breslow-Day test of interaction does exactly this, with the null hypothesis that all strata have the same odds ratio, which need not be equal to one. This test is implemented in the EpiR R package. The Breslow-Day p value of .14 means we can make this assumption, so the adjusted odds ratio is legitimate.

But this doesn't help us decide between CMH and Fisher's exact (or Pearson's $\chi^2$) tests. If the Breslow-Day test was significant, you would need to report stratum-specific odds ratios. Since it's not, you need to ask whether it's necessary to adjust for C. Does C "confound" the association between A and B? The heuristic I learned (not a statistical test) was to check whether the proportional difference between the unadjusted and adjusted odds ratios is more than 10%. Here, $\frac{1.75-1.56}{1.75}=0.108$ so CMH is appropriate.

vafisher
  • 477
  • 3
  • 5
  • I edited my question to add the result from the Breslow-Day test (it was 0.14). Therefore, I can say that it is reasonable to assume that the odds ratios are equal? In this case, should I report the Fisher's or the CMH's odds ratio? – rodrigorgs Feb 28 '14 at 23:50
  • 2
    Breslow-Day's null hypothesis is "homogeneous odds ratios across strata". Since a p value > 0.05 does not imply the null to be true, you can't assume that the odds ratios are equal. – Michael M Mar 01 '14 at 10:43
  • @MichaelMayer: I think you meant to say "the assumption of homogeneous odds ratios isn't discredited, but you shouldn't confuse failing to reject the null with proving the null". – Scortchi - Reinstate Monica Mar 01 '14 at 11:06
  • @vafisher: One thing wrong there - the 3rd sentence: Fisher's test still doesn't become appropriate when odds ratios are different across different levels of C. – Scortchi - Reinstate Monica Mar 01 '14 at 11:10
  • @Scortchi: good point! – vafisher Mar 01 '14 at 14:38
  • @MichaelMayer: As it is you seem to be saying that you can *never* assume the odds ratios are equal across strata, regardless of the p-value of the Breslow-Day test. – Scortchi - Reinstate Monica Mar 03 '14 at 12:43
  • @Scorchi: At least not by looking at a p value. Do you know of any publication investigating *how* inhomogeneous true odds ratios (and group sizes) need to be to get unreliable results from CMH test? This would be very interesting. – Michael M Mar 03 '14 at 16:13
  • @Michael Mayer: Do you object to the use of hypothesis testing to verify the assumptions of any statistical method? – vafisher Mar 03 '14 at 21:43
  • @vafisher: This is clearly too broad a topic to discuss here. Many, but not all (e.g. tests for stationarity in time series) tests of assumptions suffer this deficit. For small sample sizes, they can't reject the null, while for large sample sizes, descriptive investigations are often more focussed. – Michael M Mar 04 '14 at 06:48