in one of the commments to this post concerning the application of Kullback-Leibler-divergence between measures that do not fulfill the necessary absolute continuity (e.g. point mass vs. continuous) , as an extension it is proposed to define the KL-Divergence with respect to the sum of measures.
That is, given measures $P,Q$ define $D_{KL}(P||Q) := \int \frac{dP}{d(P+Q)} \ln \frac{\frac{dP}{d(P+Q)}}{\frac{dQ}{d(P+Q)}} d(P+Q) $, (possibly taking value $\infty$)
What I would be interested now: is this continuous in measures in some sense?
As an example take specific situation, where we have some fixed $Q$ and $P_n$ (both absolutely continuous w.r.t. Lebesgue measure) and then some limit $P$ that is eg. a point mass - do we get
$ \underset{n \rightarrow \infty} {\lim} D_{KL} (P_n||Q) = D_{KL} (P||Q) = \infty $
take for example $P_n := \mathcal{N}(0,\frac{1}{n}) \overset{d}{\rightarrow} \delta_0$ and $Q$ arbitrary.
