5

I have some models based on stochastic differential equations (SDEs). Because of the definition of these models, I can simulate data, but I cannot compute the likelihood function / distribution function. Therefore, I currently plan to use approximate Bayesian computation (ABC) to fit the parameters of these models.

However, I also need a method to compare different SDEs, which are currently discussed as possible explanations of the data, while accounting for the complexity of the parameters. Normally, I would compare these models based on DIC, LOOIC etc, but all these require the likelihood to be known.

Is there any method for comparing the model complexity, if the likelihood is unknown?

The only way I could think that might work, is to use a Bayesian model selection (i.e. using a categorial variable to switch between the models), but I am not sure if this would work at all.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
LiKao
  • 2,329
  • 1
  • 17
  • 25
  • 1
    Is (cross-validated) out of sample prediction accuracy an option? – Eoin Oct 21 '20 at 13:34
  • 2
    i haven't had a chance to read this article, but this might be relevant.. ill take a look in a few hours. https://www.pnas.org/content/108/37/15112 – user Oct 22 '20 at 00:36
  • 1
    @user228809 Thank you for the link. I've only read the abstract so far, but that seems to indicate that this is just not possible, or at least not well-founded due to unknown information loss. What a pitty... – LiKao Oct 22 '20 at 07:29
  • @Eoinisonthejobmarket I was also thinking about that, but I am not sure how that would work in this case. I am not sure if just drawing from the (unknown) distribution would be sufficient as a prediction, or if I have to add something. Also, I might not have enough data to do cross-validation. – LiKao Oct 22 '20 at 07:31
  • You said you can simulate data though? Isn't that a prediction? Also, if the data is small (and model fitting doesn't take too long), leave-one-out cross validation maybe? – Eoin Oct 22 '20 at 08:04
  • You might also look at [synthetic likelihood](https://arxiv.org/abs/1902.04827), and this method from [Turner and Sederberg](https://link.springer.com/article/10.3758/s13423-013-0530-0). I'm not 100% sure what the relationship between the two approaches is, but Turner and Sederberg do claim theirs can be used for model comparison. – Eoin Oct 22 '20 at 08:06
  • @Eoinisonthejobmarket "You said you can simulate data though? Isn't that a prediction?" I am not sure in this case. That's why I am trying to find more input. Normally I would predict using some typical value from the distribution (mean, median, highest probability), but I am not sure if I can just use any value drawn from the distribution as a prediciton. Thanks for the other links though. – LiKao Oct 22 '20 at 11:12
  • @LiKao i agree it does seem unfortunate and maybe calls for a heuristic approach. One heuristic approach could be to assess, say, total Wasserstein distance of simulated data (given each posterior configuration) versus observed data y. i.e., say you draw N posterior samples. Then simulate M*N observations z(m,n), each of are measured against y, d(z(m,n),y). Then sum [d(z(m,n),y]. That could be a heuristic approach... It would be interesting to see whether we can derive bounds or guarantees based off this heuristic. – user Oct 22 '20 at 17:52

1 Answers1

4

There are examples in the ABC literature of model selection through Bayes factors. An ecological individual based model example is here:
https://doi.org/10.1016/j.ecolmodel.2017.07.017
The paper involves picking between models of different complexity so hopefully it will be useful even if it doesn't deal with SDEs directly.

However, I should also mention that there are those that think that Bayes factors (or odds ratios or any other kind of Bayesian model selection) is just philosophically unsound. This is an argument for example forcefully stated by Gelman and Shalizi.

http://doi.org/10.1111/j.2044-8317.2011.02037.x

Their argument is that models ought to be judged primarily by their simulation output. Rather than a formula then you ought to come up with a series of model checks where you compare simulation outputs with either data you held out or other features of the data that weren't directly part of the fitting (in your case, since it's ABC, that would mean other summary statistics that were not included in the rejection step).
In my view this gives a much better grip at what the model is good and bad and is a lot more convincing than citing some obscure probability measure. Of course though, this is a very context-dependent exercise.

CarrKnight
  • 1,218
  • 9
  • 18
  • I have yet to read the Gelman paper, but if I understand your summary correctly, this shows my attempt is kind of futile. I wanted to test a more advanced model vs. a simpler one, but the more advanced model was specifically designed to capture features of the data, the simpler one cannot represent. I am unsure if this ability is due to higher complexity (more parameters), or if the more advanced model is doing some kind of overfitting, allowing it to capture these features. But I guess I have to understand the question better. – LiKao Oct 28 '20 at 08:37