0

I've just received peer reviews for an applied paper that reported a table of multiple linear regression coefficients. While I reported 95% confidence intervals for these coefficients, a reviewer has requested I report exact p-values as well. The difficulty is that the confidence intervals were estimated via an ordinary non-parametric bootstrap, given the peculiarities of the data set.

Is there a way I can estimate 'pseudo p-values' from these confidence intervals? If so, under what assumptions/conditions are these p-values reasonable?

fmark
  • 4,666
  • 5
  • 35
  • 51

1 Answers1

1

For bootstrapping, I'm assuming you resampled with replacement, computing the beta value each time to generate a distribution from which you computed the 2.5%ile and 97.5%ile. This gave you 95% confidence intervals.

For permutation, you'd resample from without replacement (shuffling up the case/control status or whatever the main factor is), get a distribution of null p values, and then compute you empirical p value based on wherever your observed falls in this null distribution. For details on the math, check out the book recommended on the link I posted above. Sorry for the piecemeal reply.

Steffen Moritz
  • 1,564
  • 2
  • 15
  • 22
N P
  • 11
  • 2
  • Thanks, this is helpful. Is it reasonable to report both p-values calculated from a permutation test and confidence intervals calculated by a non-parametric bootstrap? – fmark Apr 28 '15 at 06:06
  • Another obvious question (I'm new to permutation tests): Would I "get a distribution of null p values" or get a distribution of beta coefficients under the null hypothesis of no relationship? – fmark Apr 28 '15 at 06:10
  • 2
    Please don't post one-line answers (your other one); they're generally better suited to comments. Please don't post two versions of the same answer from two different accounts. Please delete the shorter one. Please arrange to [merge](http://stats.stackexchange.com/help/merging-accounts) your accounts. – Glen_b Apr 28 '15 at 06:19
  • You can report both the permuted p value and the 95% CI. In fact, I'd say it's always good to report both when possible. By the way, if your regression analysis involves time series or repeated measures, the permutation process can get a little bit more complicated than simple shuffling. Please see the following posts: http://stats.stackexchange.com/questions/20217/bootstrapping-vs-permutation-tests and http://stats.stackexchange.com/questions/3520/can-someone-explain-the-concept-of-exchangeability – N P Apr 28 '15 at 07:01
  • @Glen_b, sorry about that. I am working on resolving the issue and I will go through the site tutorial. – N P Apr 28 '15 at 07:03