5

Are there any software libraries that compute KL divergences in closed form, that also give the derivatives of the KL divergence wrt the distributions' parameters? I'm using Julia, so it's particularly straightforward for me to call Julia, Fortran, C and C++ libraries.

Alternatively, if libraries like this don't exist, is there something I can do that would be easier than manually coding the KL divergences, and perhaps using automatic differentiation? I have to compute KL divergences for about 10 pairs of distribution with closed-form KL divergences, e.g. beta/beta, log-normal/log-normal, mv-normal/mv-normal, wrapped-cauchy/uniform.

Jeff
  • 503
  • 2
  • 11
  • 1
    The package [Distances.jl](https://github.com/JuliaStats/Distances.jl) has code to evaluate KL-divergence for a particular sample. – bdeonovic Apr 17 '15 at 02:30
  • I need it for distributions, not samples, unfortunately. – Jeff Apr 17 '15 at 02:41
  • I figured, thats why I posted it as a comment. Since Julia is a budding open source community, why not just write up the code yourself. If they are closed form it shouldn't be too difficult. It could make a good contribution to the Distributions.jl package. – bdeonovic Apr 17 '15 at 02:43
  • As far as I understand, a **closed form** for _KL divergence_ exists only for Gaussians or mixtures of Gaussians. Can you use numerical approximation? – Aleksandr Blekh Apr 17 '15 at 03:22
  • No, they exist for many pairs of common distributions, e.g. beta/beta: [http://en.wikipedia.org/wiki/Beta_distribution#Quantities_of_information_.28entropy.29] – Jeff Apr 17 '15 at 03:28
  • I see. Thank you for clarification. In the future, please use `@` to mention a specific person (if I wouldn't return to this question, I would have missed your comment). In comments, just type `@` and possible names will appear and be available for selection. Please note that that the pop-up doesn't happen for answers. Also note that SE doesn't allow to mention more than one person per comment/answer. Final note is that you don't have to mention the person whose question or answer you're commenting on - they will be **notified** automatically. Thank you for understanding. – Aleksandr Blekh Apr 17 '15 at 04:14
  • @Jeff If I understand correctly, you need a [computer algebra system](http://en.wikipedia.org/wiki/Computer_algebra_system), which is not that specific to entropy, but to any closed-formula summations, integrations and differentiations. Mathematica IS centered around a CAS. For Python you have [SymPy](http://www.sympy.org/). I don't know Julia ecosystem enough to answer you, but search for `Julia CAS` or something like that. – Piotr Migdal Apr 19 '15 at 20:15

2 Answers2

5

I ended up coding KL divergences and derivatives myself in Julia. I've released it as part of an existing open source project. Future readers may find the code at this page of the Celeste.jl project.

Jeff
  • 503
  • 2
  • 11
1

It's great that you came up with the solution (+1). I meant to post an answer to this question much earlier, but was busy traveling to my dissertation defense (which was successful :-). You are likely to be happy with your solution, but, in addition to possibility to compute KL divergences for certain distributions in R, for example, via function KLdiv from flexmix package (http://www.inside-r.org/packages/cran/flexmix/docs/KLdiv), I ran across another and, in my opinion, much better option, which might be of your interest.

It is a very comprehensive piece of autonomous open source software, relevant to the topic, called Information Theoretical Estimators (ITE) Toolbox. It is written in MATLAB/Octave and supports various information theoretic measures. So, sending thanks and kudos to the author of this software, I'm excited to share it here and hope that it will useful to you and the community.

Aleksandr Blekh
  • 7,867
  • 2
  • 27
  • 93