0

I have a single response variable (X) and a hypothesis that there exists a relationship between it and two explanatory variables (A, B). I also have five potential confounders (C, D, E, F, G), so a total of 7 explanatory variables.

All my variables are dichotomous and I'm using logistic regression. If the analysis shows that neither A nor B contribute to the model, my hypothesis is not supported.

Is using a Bonferroni N of 2 (i.e., .05/2) adequate? The readings I'm finding suggest I should include all 7, but it seems unnecessary to include variables that aren't associated with my hypothesis? E.g., a post here says, "such a correction should be used primarily when the researcher is looking for significant associations but without a pre-specified hypothesis." I have a pre-specified hypothesis, but also have two explanatory variables, so am more likely to see a type I error than if I only had one explanatory variable. So, adjusting my alpha by 2 seems reasonable?

I'm reading here that "Bonferroni adjustment procedures (generally named "family-wise error rate (FWER)" procedures) are correct if you want to control the occurrence of a single false positive." Unless I'm missing something, I don't care about false positives among my confounders, so wouldn't need to include them in my Bonferroni N?

buttonsrtoys
  • 205
  • 1
  • 8
  • 1
    See https://stats.stackexchange.com/questions/120362/whats-wrong-with-bonferroni-adjustments/ for arguments against using the Bonferroni method. – fblundun Mar 31 '21 at 10:19
  • @fblundun thanks for the link! Lots of great resources in it. I took a quick look and agree with the limitations. However, I do think I am increasing my chances of a type I error by having two explanatory variables in my hypothesis. So, would be more comfortable with a "partial" Bonferroni that uses p=0.025. Or would that be weird? My results meet significance for .025 but don't for .05/7=.007, so my only other alternative would be to address why I did not apply Bonferroni to my results? – buttonsrtoys Mar 31 '21 at 11:16
  • 1
    If you are set on using the Bonferroni correction, and if finding associations between X and the five confounders is out of scope for your investigation, then I think that your p=0.025 approach is appropriate, as it does ensure the Type I error probability is at most 0.05, which is the stated aim of using the Bonferroni correction. – fblundun Mar 31 '21 at 14:03
  • @fblundun Thanks! I'm trying to find some literature to support the approach of adjusting my multiplicity N to only those explanatory variables that support my hypothesis but not finding any :-/ I don't suppose you know of any? – buttonsrtoys Mar 31 '21 at 19:56
  • 1
    https://en.wikipedia.org/wiki/Bonferroni_correction supports it. Your family of hypotheses has size 2 since it doesn't include hypotheses about relationships between X and the potential confounders. – fblundun Mar 31 '21 at 20:24
  • 1
    My take on this is here: https://stats.stackexchange.com/questions/503532/should-i-correct-for-multiple-comparisons-if-i-generate-hypothesis/503538#503538 and here: https://stats.stackexchange.com/questions/468620/correction-for-multiple-testing-in-multiple-regression-analysis/468622#468622 ... and I don't understand the mini markdown help that supposedly explains how to link in comments in a nicer way. – Christian Hennig Apr 01 '21 at 12:39
  • 1
    @buttonsrtoys: In the given situation I'd probably agree with your reasoning. Whether you can "sell" this well in a publication, say, is another matter... – Christian Hennig Apr 01 '21 at 12:44
  • @Lewian Thanks for the thoughts and the links. Your first link where you discuss, ""it may be worthwhile to have a deeper look into this on new data" is very relevant to my study, as its exploratory, so I'm not trying to make bold claims. I agree with your insights about "selling" my approach since it could be flagged. Since I can't meet .05/7 significance, my only alternative would be to ignore multiplicity correction, which doesn't feel great when my family of hypotheses has size 2. – buttonsrtoys Apr 01 '21 at 17:41
  • @Lewian Also, one thing I didn't mention in my OP is my effect sizes look great, so that should give some confidence level to the reader (I hope). – buttonsrtoys Apr 01 '21 at 17:42
  • 1
    If anybody objects, ask them why you should stop at a correction factor of only 7. There are many more possible hypotheses that your data might or might not support - for example, that X is pairwise independent of both C and D but is correlated with |C - D|, or that there is a relationship between E and F. If you can ignore these hypotheses while deciding your correction factor, you can ignore the five simple confounder hypotheses too. – fblundun Apr 01 '21 at 18:36
  • 1
    @buttonsrtoys: In any case if you're doing exploratory work you shouldn't aim at having p-values below a certain threshold, regardless of whether you apply Bonferroni in a reasonable manner or not. If your p-value is between .05/2 and .05/7, you need to acknowldege that you have some moderate but not all too strong evidence against the H0; if you just want to say that this could be looked into on new data, this should be OK with p=0.03 before Bonferroni as well, or even (if there are good subject matter reasons on top of the message from the data) with p=0.065. (To be continued.) – Christian Hennig Apr 01 '21 at 20:11
  • 1
    (Continuation:) Have a look at the ASA statement on p-values, stating "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold." https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108 – Christian Hennig Apr 01 '21 at 20:12

0 Answers0