Probabilistic Interpretation of Radial Basis Function

Question

I was wondering if someone could flesh out the probabilistic interpretation of using the Radial Basis Function to compute the probability between an observation and some reference value.

My question is partially motivated by the top answer of this reddit thread:

The RBF kernel is a standard kernel function in $R^n$ space because it has just one free parameter, $\gamma$, and satisfies the condition $K(x,x') = K(x',x)$. More specifically, one way to think of the RBF kernel is that if we assume $x'$ is characteristic of some gaussian distribution (it is the mean value of that distribution), then $RBF(x,x')$ is the probability that $x$ is another sample from that distribution. In this interpretation, $\gamma$ is related to the tunable variance of that distribution.

Does this mean that if we have an observation $\bf{s}$ and we want to know if $\bf{s}$ is generated by a source $\bf{q}$ if $\bf{s}$ is a noisy version of $\bf{q}$, then we can say:

$$P(\mathbf{s} \text{ generated by } \mathbf{q}) \propto \text{exp}(-\gamma d(\mathbf{s},\mathbf{q}))$$ $$P(\text{belongs to a Gaussian region defined by } \mathbf{q} | \mathbf{s}) \approx \text{exp}(-\gamma d(\mathbf{s},\mathbf{q}))$$

where $d(\mathbf{s},\mathbf{q})$ is the distance between $\bf{s}$ and $\bf{q}$ and $\gamma$ is as described in the above quote.

Does this all seem consistent? That this probability is a direct consequence of the RBF comparing an observation to some mean value (or reference value or source value)?

Any references/links to tutorials are most welcome.

Answer 1

RBF kernel

The radial basis function (RBF) kernel is:

$$k(x, x'; \sigma^2) = \exp \Big( -\frac{\|x-x'\|^2}{2 \sigma^2} \Big)$$

where the parameter $\sigma^2$ specifies the width. This formulation is equivalent to the one you wrote involving the 'precision' parameter $\gamma$ if we let $\gamma = \frac{1}{2 \sigma^2}$.

Isotropic Gaussian distribution

Now, consider a $d$-dimensional Gaussian distribution with mean $\mu$ and covariance matrix $\sigma^2 I$ (where $I$ is the identity matrix). This means the variance is the same ($\sigma^2$) in all directions. The probability density function is:

$$\mathcal{N}(x \mid \mu, \sigma^2 I) = (2 \pi \sigma^2) ^{-\frac{d}{2}} \exp \Big( -\frac{\|x-\mu\|^2}{2 \sigma^2} \Big)$$

Their relationship

Notice that $k(x, x'; \sigma^2) = k(x', x; \sigma^2)$ is proportional to $\mathcal{N}(x \mid x', \sigma^2 I) = \mathcal{N}(x' \mid x, \sigma^2 I)$. That is, the value of the RBF kernel evaluated between $x$ and $x'$ (with width $\sigma^2$) is proportional to the probability density assigned to $x$ under an isotropic Gaussian distribution with mean $x'$ and variance $\sigma^2$. Or, equivalently, to the probability density assigned to $x'$ when the mean is $x$.

The following are not true:

$$P(\mathbf{s} \text{ generated by } \mathbf{q}) \propto \text{exp}(-\gamma d(\mathbf{s},\mathbf{q}))$$

As above, the correct term on the lefthand side would be the probability density assigned to $s$ by a Gaussian distribution with mean $q$ and variance $\frac{1}{2\gamma}$. This is not the same as the probability that $s$ is generated by this distribution (for more detail about this point, see the distinction between likelihood and probability). Similarly, the following statement in the reddit quote is incorrect for the same reason: "if we assume $x'$ is characteristic of some gaussian distribution (it is the mean value of that distribution), then $RBF(x,x')$ is the probability that $x$ is another sample from that distribution."

$$P(\text{belongs to a Gaussian region defined by } \mathbf{q} | \mathbf{s}) \approx \text{exp}(-\gamma d(\mathbf{s},\mathbf{q}))$$

There's no such thing as a 'Gaussian region'. Rather, we have a Gaussian probability distribution (which actually has infinite support, rather than being defined on a compact region). And, the RBF kernel is proportional to the density function, but "proportional to" doesn't imply "approximately equal to" (the numerical difference can be quite large, depending on the value of the normalizing constant $(2 \pi \sigma^2)^{-\frac{d}{2}}$.