1

In our study, we applied a bacward stepwise elimination logistic regression analysis between the two groups over a wide variety of variables.

There was such a criticism of the referee.

Your results, although statistically are approximating an odds ratio of 1. Especially considering the amount of variables you are reporting, it would be good practice with a Bonferoni correction instead of assuming the usual p-value of .05 for rejection of the null- hypothesis.

What do you think the referee meant and what kind of solution should be applied?

1 Answers1

0

One of the goals of frequentist hypothesis testing is to maintain the probability of a Type I Error (T1E). When testing many hypotheses we usually maintain the family wise error (FWER), which is the probability of making at least 1 T1E.

Why do we have a problem? Suppose we maintain an the probability of T1E of $\alpha$ for each test and define $V = \sum_{i=1}^{m_0} V_i$ where $V_i$ is the indicator of making a T1E for hypothesis $i$, $m_0$ is the number of correct null hypotheses and $m$ is the number of hypotheses. Furthermore, assume of tests are independent.

$P(V \geq 1) = 1 - P(V = 0) = 1 - P(\cap_{i=1}^{m_0} V_i = 0) = 1 - (1 - \alpha)^{m_0}. $

As you can see, the more hypotheses we test the probability of making at least one T1E increases.

In order to maintain the FWER we can apply a variety of multiple comparisons corrections, one of the most standard corrections is the Bonfferroni correction. It will guarantee that the FWER will be maintained at a specified level. The correction is conservative (meaning it will maintain a FWER at probability smaller than $\alpha$), and depending on the scenario other method might be preferable (such as Holm or Hochberg).

Note, that even with the correction the analysis is still not valid. Since you have used backward stepwise algorithm to choose the variables, and this selection must be taking into account in the analysis. You might be better using Lasso logistic regression with the appropriate correction, see Lee, Jason D., et al. "Exact post-selection inference, with application to the lasso." The Annals of Statistics 44.3 (2016): 907-927, and this question.

Kozolovska
  • 1,027
  • 6
  • 11