0

I am trying to model the P300 complex of event-related potentials across conditions. For that, I randomly sample parameters, fit a model with these parameters and keep record of parameters vectors that give a reasonable mean squared error (MSE). Next, I run fminsearch in Matlab with the best parameters vectors and search for best-fitting parameters. However, this results in 20-30 models with highly similar MSE, e.g.

0.0298 0.0302 0.0305

etc.

I think that simply selecting the model with smallest MSE is not perfect, as the difference in MSE between 1st and 10th best model is marginal. What is the rule for selecting best model in such cases? Could you recommend a paper or a book on this topic?

Best, Dawid

Dawid Archibald
  • 1
  • Are your MSEs measures of model fit? If so, then the first recommendation would be to use out-of-sample MSEs via cross-validation. Next, if you can reasonably assign a complexity to your models (e.g., perhaps some models have parameters fixed to zero), you can use the one standard error rule [see here](https://stats.stackexchange.com/q/80268/1352). – Stephan Kolassa Aug 04 '21 at 15:48
  • Hi: Assuming you're interested in which model is better at predicting, check out the diebold-mariano test. it's a test of "which model is better at prediction". The original paper should be google-able. Then, Diebold wrote another paper ( also google-able IIRC ) like 20 years later on how the test is often mis-used. The second is also worth a read. – mlofton Aug 04 '21 at 15:49
  • 1
    What's your goal? For many purposes you might be best off averaging the best models. For that perspective, Burnham and Anderson ("Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach") is a easy (for quantitative researchers, anyway) to read book. – Björn Aug 04 '21 at 15:55
  • @mlofton: [Diebold (2015, *J Bus Econ Stat*)](https://doi.org/10.1080/07350015.2014.983236), I agree, it's a good read. And it presupposes OP is looking at out-of-sample MSEs. The test then assesses whether these OOS MSEs are significantly different. – Stephan Kolassa Aug 04 '21 at 16:00
  • Hi Stephan: It's been a while so thanks for the details. Yes, an interesting paper and the second one is good also. – mlofton Aug 05 '21 at 17:57

0 Answers0