4

I have seen several articles and CrossValidated questions on bootstrapping ( this, this or this for example); there are a lot of theoretical and statistical explanations, however since they are so theory based, I am afraid I might be understanding the use wrongly. Hence my questions:

1) When I make a non-parametric bootstrapping (changing the sample for every run) with logistic regression on my data, I basically will end up with several different coefficients for each predictor for each run. Eventually I'll have the confidence interval for each predictor as well. I understand until that point. My question is; assuming that the distribution is normal, when I want to come up with a final model on practice, can I just take the mean of the confidence intervals for each predictor and consider this as my final model coefficient?

2) If the answer to question #1 is yes, is this the only way of choosing coefficients while bootstrapping? If not, what else? I encountered in a few more articles a method called "bagging". This seems to be my main purpose.

3) This one is more of a curiosity question: Can above methodology be applied to the categorical predictors when they are assigned with Weight Of Evidences? I know we can split the categorical predictors into dummy variables; but how would I treat each coefficient if I want to use WOE methodology?

Community
  • 1
agondiken
  • 191
  • 1
  • 5
  • Can you clarify your second question? By "choosing", do you mean arriving at a final value for the coefficients, or do you mean choosing which variables to include in the final model? – D L Dahly Dec 30 '13 at 15:25
  • I mean more for the coefficients for each predictor. Let's say I already reduced my variables to a list where each would contribute to the model, but I want to find the optimum coefficients that will keep an optimum level of bias and variance. If I would bootstrap on many samples, I'll have many different values for these coefficients, as well as different Ginis(or AUC) for my bootstrapped models. Assuming that I want to try my model on a 3rd hold-out sample, I would think that the most reasonable coefficient for each predictor would be the mean from all samples. Is this a valid thought? – agondiken Dec 31 '13 at 10:14

1 Answers1

3

Regarding the first question, you instead want to estimate a shrinkage factor that can be applied to your regression coefficients to arrive at your "final" model.

  1. Draw a bootstrap sample.
  2. Estimate your model (using your full modelling process, whatever this is) in this sample.
  3. In the original data, calcuate the linear prediction using the boostrap coefficients.
  4. In the original data, regress the outcome on the linear predictor you just calculated.
  5. Save the regression coefficent from this model.
  6. Repeat this process many times.
  7. Average all of the regression ceofficients from step 5. This is your shrinkage factor.
  8. Shink your "final" model in your original data using this value.

Exactly how to do this will depend on what software you are using.

See Harrell 1996

Also see Steyerberg 2001

D L Dahly
  • 3,663
  • 1
  • 24
  • 51
  • Thanks for the response! But I think you are referring to parametric bootstrapping methodologies, the second methodology mentioned in [this](http://stats.stackexchange.com/questions/64813/two-ways-of-using-bootstrap-to-estimate-the-confidence-interval-of-coefficients) question. I wanted to learn the first one. Still to clarify if I understood your way correctly; when you say original data, do you refer to "all sample"? Or is it just the initial random sample you took for an initial model development? From Harrell's paper, I got the understanding that it's all sample.. – agondiken Dec 31 '13 at 10:05