2

Probabilistic Graphical Models (PGMs) are used to model all sorts of complex decision processes, such as medical diagnoses or robot positions, etc. In common machine learning textbooks, like Christopher Bishops book on pattern recognition or Koller's text, the idea of the "flow of influence" or conditional independence seems to be a big feature/benefit of these methods. In other words, these models can make better decisions when dealing with data where a mixture of generative processes may exist.

I am a bit confused about the inference methods for PGMs versus the ability of these inference methods to capture the conditional independence relationships between variables. In both the Bishop, Koller, and Murphy texts, the decription of Bayesian networks (directed) and Markov random fields, is immediately followed by the discussion about "exact" inference and Belief Propagation. Then variational methods are kept for a subsequent chapter.

I can see how belief propagation or message passing methods execute inference by following the pattern of conditional independence relationships in the network. But my question is, do the approximate inference methods, like maximizing the ELBO, evidence lower bound, also capture this pattern of conditional independence relationships? Also, under what conditions do variational inference methods fail to capture the flow of influence in the original network. My assumption is the variational methods should factor in the conditional independence structure of the network, but there should be issues with the complexity of the underlying distribution, or with the mis-specification of the candidate factorized distribution for VI that could get in the way of accurate representation.

Part of my confusion also stems from the fact that many of the popular "Probabilistic Programming" packages, like Pyro, or Tensorflow Probability, or even PyMC3 don't support any belief propagation/message passing methods, and their tutorials never really discuss conditional independence and the flow of influence. So I was just hoping to find an explicit answer to this question.

krishnab
  • 1,008
  • 7
  • 12

1 Answers1

2

The optimization process casts by variational inference consists in fitting a function to an another (a probability distribution to another (simpler) probability distribution, we call the latter the variational distribution) under a criteria to maximize (ELBO most commonly). I think it is the simplest way to think about the problem.

The most common approach to variational inference is mean field variational inference which uses a variational distribution with fully independent random variables (no structure at all, also called fully factorized). This approach does not try to replicate any conditional independence in the variational distribution. Thus we do not learn conditional independence relations. This leads to a direct inference procedure but might fit the original complex distribution quite badly.

It is possible to model your variational distribution with conditional independence between its random variables. It is called structured variational inference (short review Section 5.3 here). This structured variational inference approach can be seen as a way to try to fit the original distribution by a more refined family of distribution. If your structured variational distribution is well crafted you can fit much better your target distribution than in a mean field approach (but with a possibly more complex inference in the variational distribution this time). If some conditional independence relations are learnt in the process it is because the family of function you allow yourself to use somehow already encodes that relations (to the best of my knowledge :) ).

TheCG
  • 822
  • 4
  • 13
  • thanks so much for the answer. So my suspicions were validated, that is good to know. That paper you reference is really excellent, I read section 5.3 as you suggested. It is interesting that inference from variational approximation is so sensitive to the parameterization of the factorized distribution. I did not see much attention to this point in Bishop or Koller's books. – krishnab Jul 20 '20 at 14:04
  • It is so interesting. If one of the biggest selling points on PGMs is the ability to reason based on this conditional independence--and corresponding flow of influence. If that is the case, then why would anyone use a very vanilla mean field factorized distribution, instead of a factorized distribution that matches the cnoditional independence structure of the specified model? Is it just because of fitting time? – krishnab Jul 20 '20 at 14:07
  • I think structured variational inference is not the norm because (a) you need an original complex distribution that you know in advance and that has a structure over which we can propose a "meaningful" variational distribution (b) the structured variational distributions you can use must remain quite simple otherwise the inference in the variationl distribution itself cannot be exact. – TheCG Jul 20 '20 at 14:27
  • For (b): the most famous example of structured variational inference may be factorial hidden markov models which uses a simple structure for the variational distribution: parallel Markov chains (see [here](http://www.ee.columbia.edu/~sfchang/course/svia-F03/papers/factorial-HMM-97.pdf) for the article that introduces the model) – TheCG Jul 20 '20 at 14:27
  • Excellent. That helps a lot. I get it now, so it seems like finding a mapping between the original model distribution versus a suitable variational distribution with desirable properties. I mean we know the original distribution since we have a generative story on how the data are created--namely the model itself. But finding a factorized distribution to match that model seems like the challenge, as you say. Very interesting. I will read the Ghahramani article, looks interesting. If you know of any other good papers on this topic, please send along links :). Thanks again. – krishnab Jul 20 '20 at 14:55
  • Here are two resources that helped me put into practice structured VI: this phd thesis chapter [here](http://www.johnwinn.org/Publications/thesis/Winn03_chap5.pdf) and this article [here](https://arxiv.org/pdf/1301.3901.pdf) (+ Ghahramani articles on factorial hidden models). – TheCG Jul 21 '20 at 05:36
  • Another interesting approach that goes further is auxiliary random variable variational inference in which you "augment" your probability distribution with auxiliary random variables to try to fit even better the original distribution see [here](http://aivalley.com/Papers/textVAUX_icn.pdf). It can be shown that the bound with auxiliary random variable is worse than the ELBO but again, if the variational distribution is well crafted, results can be even better than mean field and structured vi without auxiliary variables. – TheCG Jul 21 '20 at 05:39
  • 1
    thanks again. I am starting to read the articles you suggested, and trying to setup some simple computer experiments to compare exact versus variational inference. Thanks for setting me straight on the hierarchical or structured variational inference, I would not have found that on my own. The augmented variational inference seems very interesting. I have used augmented variables in MCMC but not in variational methods. Seems like there is some renewed interest in PGMs, so nice to have a guide through the literature. – krishnab Jul 21 '20 at 14:50