1

I'm trying to think about WAIC under a multivariate model scenario. Suppose I have one model composed of two relationships:

y1 ~ x
y2 ~ y1

This is one model. Now, I have a second model

y1 ~ x
y2 ~ y1 + x

I would like to construct a WAIC to compare the first case to the second. However, given that y1 and y2 are likely on different scales or might even share different error distributions, their WAIC values could be on radically different scales. How, then, to combine?

Intuitively, scaling WAIC values for each relationship seems like the answer. But how? Given that we're often interested in $\Delta$WAIC scores, simply centering and summing seems intuitive, but, is there a theoretical justification behind that intuition? Or is something more exotic needed?

I know this might be an odd question, but is there any literature on rescaling WAICs or perhaps likelihoods to combine multiple pieces into a more holistic score for model comparison in this type of scenario? I've been sifting through the Structural Equation Modeling literature and have not yet found something adequate to translate into a Bayesian framework.

jebyrnes
  • 835
  • 6
  • 16

2 Answers2

2

If you consider that WAIC is asymptotically same as Bayesian leave-one-out cross-validation, then you can think what is your prediction task and the cost/utility function used. Then you would not consider y1 and y2 as some arbitrary quantities, and knowing what is your prediction task would help you to choose the balance between accuracy of predicting y1 and y2. See Aki Vehtari and Janne Ojanen (2012). A survey of Bayesian predictive methods for model assessment, selection and comparison. Statistics Surveys, 6:142-228, http://dx.doi.org/10.1214/12-SS102 for more details on the decision theoretical assumptions behind WAIC and LOO-CV.

Aki Vehtari
  • 538
  • 4
  • 8
1

The WAIC, as the AIC, is scaled to the likelihood/posterior, so there should be no need to adjust it to the scales of y1 or y2.

Moreover, as the AIC is defined point-wise, it shouldn't matter in which order you sum it up, so you can calculate WAIC(y1, y2), or also WAIC(y1) + WAIC(y2).

Side note: blavaan is doing Bayesian SEMS based on the lavaan syntax and can calculate the WAIC.

Florian Hartig
  • 6,499
  • 22
  • 36
  • In blavaan, though, it's working with the covariance matrix - or so was my understanding - not individual different response variables. Do you have a good reference in summing AICs? Or at least centered AICs (as it's the delta AIC that's really the thing - the more I think about it, the more centering makes sense as you're just shifting constants)? While it feels intuitive, I'm hoping to find something more definitive... – jebyrnes Nov 25 '17 at 00:55
  • you mean AIC or WAIC? – Florian Hartig Nov 26 '17 at 21:50
  • Either or both? It's an intuitive concept, as I say, but haven't seen it in the literature. – jebyrnes Nov 29 '17 at 15:10
  • It's standard to sum likelihoods to a joint likelihood (assuming independence of the different data types), and that should logically translate to AIC/WAIC etc. AICs/BFs are certainly also calculated for multiple data types in the literature. – Florian Hartig Nov 30 '17 at 11:16
  • Huh. Thanks. Really have not encountered this before. If you think of a ref where this is done, let me know, as I'd really appreciate it. My goog-fu has been failing on this topic. – jebyrnes Nov 30 '17 at 16:59
  • If you google "joint likelihood" or "integrated population model", you will find hundreds of papers that sum likelihoods for different data sources - a random example I have contributed to is Panassiti, B et al. Identifying local drivers of a vector-pathogen-disease system using Bayesian modeling BAE, 2017 ... I don't know a foundational reference for this, in my mind this never required a mathematical justification, as the whole point of stats is to define p(D|M), so who cares if D consists of two subsets of different type. You could also subdivide any "homogenous" dataset D into D1 and D2. – Florian Hartig Dec 01 '17 at 13:30
  • Gratzi - IPMs use this? Really. Huh. Thanks - feeling sheepish for missing that detail! Thanks! – jebyrnes Dec 01 '17 at 18:45
  • yes, integrated population models (not integrated projection models btw., therefore I try to avoid the abbreviation IPM) often have several observation types, and a joint likelihood. I don't see anyway treating this joint likelihood as different to a single-data likelihood. – Florian Hartig Dec 04 '17 at 13:37
  • I completely thought you meant integrated projection models. HA! Thanks again! – jebyrnes Dec 04 '17 at 17:18