What is a "kernel" in plain English?

Question

There are several distinct usages:

• kernel density estimation
• kernel trick
• kernel smoothing

Please explain what the "kernel" in them means, in plain English, in your own words.

Answer 1

There appear to be at least two different meanings of "kernel": one more commonly used in statistics; the other in machine learning.

In statistics "kernel" is most commonly used to refer to kernel density estimation and kernel smoothing.

A straightforward explanation of kernels in density estimation can be found (here).

In machine learning "kernel" is usually used to refer to the kernel trick, a method of using a linear classifier to solve a non-linear problem "by mapping the original non-linear observations into a higher-dimensional space".

A simple visualisation might be to imagine that all of class $0$ are within radius $r$ of the origin in an x, y plane (class $0$: $x^2 + y^2 < r^2$); and all of class $1$ are beyond radius $r$ in that plane (class $1$: $x^2 + y^2 > r^2$). No linear separator is possible, but clearly a circle of radius $r$ will perfectly separate the data. We can transform the data into three dimensional space by calculating three new variables $x^2$, $y^2$ and $\sqrt{2}xy$. The two classes will now be separable by a plane in this 3 dimensional space. The equation of that optimally separating hyperplane where $z_1 = x^2, z_2 = y^2$ and $z_3 = \sqrt{2}xy$ is $z_1 + z_2 = 1$, and in this case omits $z_3$. (If the circle is off-set from the origin, the optimal separating hyperplane will vary in $z_3$ as well.) The kernel is the mapping function which calculates the value of the 2-dimensional data in 3-dimensional space.

In mathematics, there are other uses of "kernels", but these seem to be the main ones in statistics.

Answer 2

In both statistics (kernel density estimation or kernel smoothing) and machine learning (kernel methods) literature, kernel is used as a measure of similarity. In particular, the kernel function $k(x,.)$ defines the distribution of similarities of points around a given point $x$. $k(x,y)$ denotes the similarity of point $x$ with another given point $y$.