2

I have an expensive model (or class of models). My baseline approach to quantify uncertainty re the model parameters are hessian based standard errors, and I use k-fold cross validation for model comparison / validation. While a full bootstrap would be pleasant as a more robust uncertainty quantification, this is quite expensive. I think I should also be able to develop expectations for the variance of the leave-k out estimates, to at least get a rough sense of where the hessian based standard error estimates are not performing well. I wonder if someone knows how to do this, or can point to work that does this? Something like an approximate jackknife?

Charlie
  • 275
  • 3
  • 13
  • Just to be clear, was it a repeated CV or a single repeat? For example, under $K$=3 (~33% test set) we effectively have something that approximates the test set size of a standard bootstrap (~36.6%), if we did say 20x3-fold that is not miles away from 60 bootstrap samples. – usεr11852 May 21 '20 at 17:08
  • The example that led to the question was a single 10 fold -- this is flexible though, to an extent. – Charlie May 21 '20 at 17:27
  • OK, scratch that then. :) Have you consider using [Delta method](https://en.wikipedia.org/wiki/Delta_method)? – usεr11852 May 21 '20 at 17:29
  • That is how the hessian is approximated, yeah. I'm hoping to use the k-fold estimates as a check on this. Increasing the number of folds / repeats is certainly possible, it's just not obvious to me how to use the k-folds to approximate the std error of the parameter estimate. – Charlie May 21 '20 at 17:36
  • Good choice! What model are we talking about here? (I think it would helpful to make it clear you are already using the Delta method because it couldn't tell immediately from the original post.) Also, you can increase the folds/repeats why not just bootstrap things? My original comment was along the lines of reusing the estimates through bias correction (See Vanwinckelen & Blockeel (2012) and the CV thread [here](https://stats.stackexchange.com/questions/185048)) for a more context.) – usεr11852 May 21 '20 at 17:55
  • It's some kind of hierarchical sde / kalman filtering stuff. Bootstrapping is anyway not at all simple, and just because I could run it a lot longer, doesn't mean I want to! Multiplying the cov matrix of the k-fold parameter estimates by k seems to get me roughly what I want, though this will need to be repeated and averaged, or k increased, to get anything like a reliable estimate, and I could still be missing some kind of small correction factor or something... – Charlie May 23 '20 at 14:01

0 Answers0