0

I'm studying pattern recognition and I'm at the part about Kernel density estimators. During the introduction of the subject, the book I'm studying (Pattern Recognition & Machine Learning by Bishop) takes for granted something I'm not sure I can understand.

Say we have an unknown pdf $p(x)$ in some D-dimensional space and let us consider some small region $R$ containing $x$. Then, if we make the assumption that $R$ is small enough so that the pdf is roughly constant over the region, we have $$P \approx p(x)V$$ where $V$ is the volume of $R$.

I'm completely unaware of how this formula was derived or how the volume $V$ appeared there. Any help woud be greatly appreciated. Thank you.

thenac
  • 291
  • 3
  • 10
  • See https://stats.stackexchange.com/questions/4220. Your equation is one *definition* of a probability density. – whuber Nov 23 '19 at 22:31
  • 1
    Thanks whuber. I was aware of this definition but as I said in the comments I was confused from the volume term. I think I got it now though. – thenac Nov 23 '19 at 23:10

2 Answers2

2

It's simple integral approximation. First, think in 1D. The area under a curve $f(x)$ in a very small x-axis segment (e.g. $[x,x+\Delta x]$) is $\approx f(x)\Delta x$; because $f(x)$ is nearly constant across this region. Similarly, in 2D, integral of $f(x,y)$ over a small region is $\approx f(x)\Delta x\Delta y$. In multiple dimensions, all the multiplicands near $f$ is called as volume, i.e. $f(x_1,...,x_n)\underbrace{\Delta x_1...\Delta x_n}_V$.

gunes
  • 49,700
  • 3
  • 39
  • 75
  • This is what I thought at first, but I was confused because as you say, when my variable $x$ is a vector of 2 dimensions $x,y$ then indeed the approximation would be $f(x)Δ(x)Δ(y)$. But, isnt this the volume? Since $f(x)$ is the "height" and you multiply it with the other 2 dimensions, say width and depth. Therefore I was confused because I thought, it should be $P = V$ and not $p(x)V$ since the volume requires to multiply $p(x)$ with all the rest dimensions. But its probably an n-dimensional thing that they called volume this product? – thenac Nov 23 '19 at 19:26
  • @Thomas, You misinterpret the term volume. In 1D, length of line segment $R$ is the volume. In 2D, area of $R$ is the volume etc. The volume here is not the area/volume under $f(x)$. Informally, think as if it is referred as *hypervolume* of region $R$. – gunes Nov 23 '19 at 19:33
  • When you say it like that it makes sense. We get off subject here but, isn't volume the space that a 3d object takes up? And it is $A*h$ where A = area and h = height. Is the concept we use in this question somewhat different from the one I mention here in the comment? Of course we generalize it for more than 3 dimensions. – thenac Nov 23 '19 at 19:42
  • 1
    The term volume is used for all dimensions, e.g. https://en.wikipedia.org/wiki/Volume_of_an_n-ball – gunes Nov 23 '19 at 19:43
  • Thanks for the link, but as you said , in a 3d example the approximation would be $f(x)dxdy$. Isnt this whole product the volume of a shape? $dxdy$ is the area of $R$ and $f$ is the height. Your answer explained my question, so sorry I keep asking things, just trying to fully understand what you're saying ! :) – thenac Nov 23 '19 at 19:50
  • $f(x)dxdy$ is the volume under $f(x)$, but $dxdy$ is the volume of $R$. – gunes Nov 23 '19 at 19:51
  • 1
    Ok thing I got this! Gotta let it sink in :) Thanks for all the help, very much appreciated. Have a good day. – thenac Nov 23 '19 at 19:54
1

Basically, this is how probability density is defined. Probability density is the probability per foot. It is normalized by the area, so that over all the area it integrates to unity. More formally, quoting All of statistics by Larry A. Wasserman:

2.11 Definition. A random variable $X$ is continuous if there exists a function $f_X$ such that $f_X(x) \ge 0$ for all $x$, $\int_{-\infty}^\infty f_X(x)\, dx = 1$ and for every $a \le b$,

$$ \mathbb{P}(a < X < b) = \int_a^b f_X(x) \,dx \tag{2.2} $$

The function $f_X$ is called the probability density function (PDF).

Tim
  • 108,699
  • 20
  • 212
  • 390
  • I'm aware of the definition and it was my first thought. However I was confused for the reason I posted in the above comment, if you'd like to take a look ! – thenac Nov 23 '19 at 19:29