Where is density estimation useful?

Question

After going through some slightly terse mathematics, I think I have a slight intuition of kernel density estimation. But I am also aware that estimating multivariate density for more than three variables might not be a good idea, in terms of the statistical properties of its estimators.

So, in what sorts of situations should I want to estimate, say, bivariate density using non-parametric methods? Is it worth enough to start worrying about estimating it for more than two variables?

If you can point to some useful links regarding application of estimation of multivariate density, that'd be great.

Answer 1

One typical case for the application of density estimation is novelty detection, a.k.a. outlier detection, where the idea is that you only (or mostly) have data of one type, but you are interested in very rare, qualitative distinct data, that deviates significantly from those common cases.

Examples are fraud detection, detection of failures in systems, and so on. These are situations where it is very hard and/or expensive to gather data of the sort you are interested in. These rare cases, i.e. cases with low probability of occurring.

Most of the times you are not interested on estimating accurately the exact distribution, but on the relative odds (how likely is a given sample to be an actual outlier vs. not being one).

There are dozens of tutorials and reviews on the topic. This one might be a good one to start with.

EDIT: for some people seems odd using density estimation for outlier detection. Let us first agree on one thing: when somebody fits a mixture model to his data, he is actually performing density estimation. A mixture model represents a distribution of probability.

kNN and GMM are actually related: they are two methods of estimating such a density of probability. This is the underlying idea for many approaches in novelty detection. For example, this one based on kNNs, this other one based on Parzen windows (which stress this very idea at the beginning of the paper), and many others.

It seems to me (but it is just my personal perception) that most if not all work on this idea. How else would you express the idea of an anomalous/rare event?

Answer 2

I guess that the mean-shift algorithm (http://en.wikipedia.org/wiki/Mean-shift) is a good example for an efficient and suited application of kde. The purpose of this algorithm is to locate the maxima of a density function given data $(x_i)$ sampled from that density function and it is entirely based on a kde modeling: $$ f_h(x) \propto \sum_{x_i} \exp( -(x_{i}-x)^{T}\Sigma^{-1} (x_{i}-x)), $$ where $\Sigma^{-1}$ is a covariance matrix (most of the time estimated). This algorithm is widely used in clustering tasks when the number of components is unknown: each discovered mode is a cluster centroid and the closer a sample to a mode the more likely it belongs to the corresponding cluster (everything being weighted properly by the shape of the reconstructed density). The sample data $x_i$ are typically of dimension larger than one: for example, to perform a 2D color image segmentation, the samples can be 5d for (RComponent, GComponent, BComponent, xPosition, yPosition).

Answer 3

Typically, KDE is touted as an alternative to histograms. The main advantage of KDE over histograms, in this context, is to alleviate the effects of arbitrarily chosen parameters on the visual output of the procedure. In particular (and as illustrated in the link above), KDE does not need the user to specify start and end points.