10

In a problem set I proved this "lemma," whose result is not intuitive to me. $Z$ is a standard normal distribution in a censored model.

Formally, $Z^* \sim Norm(0, \sigma^2)$, and $Z = max(Z^*, c)$. Then, \begin{align} E[Z|Z>c] &= \int_c^\infty z_i \phi({z_i})\mathrm{d}z_i \\ &= \frac{1}{\sqrt{2\pi}}\int_c^\infty z_i \exp\!\bigg(\frac{-1}{2}z_i^2\bigg)\mathrm{d}z_i \\ &= \frac{1}{\sqrt{2\pi}} \exp\!\bigg(\frac{-1}{2}c^2\bigg) \quad\quad\quad\quad\text{ (Integration by substitution)}\\ &= \phi(c) \end{align} So there is some sort of connection between the expectation formula over a truncated domain and the density at the point of truncation $(c)$. Could anyone explain the intuition behind this?

Heisenberg
  • 4,239
  • 3
  • 23
  • 54
  • 2
    That it turns out that way is a consequence of the fact that the $z$ term is the negative of the derivative of the term in the exponent; it's one of many neat results for the standard normal but it doesn't necessarily have intuition behind it. On the other hand it wouldn't surprise me at all if one of the clever people here could come up with some kind of intuition for it. – Glen_b Feb 12 '14 at 21:29
  • @Glen_b What you are saying is that $\int_c^\infty \left(-\frac{d}{dz}\log(f(z))\right) f(z) dz = -\int_c^\infty f^\prime(z)dz = f(c)$ where $f$ is the PDF of *any* continuous distribution $F.$ – whuber Feb 12 '14 at 22:10
  • @whuber That's certainly the case, and it's worth emphasizing that result, since it's directly relevant to the result in the question, but actually in my comment I was referring specifically to the case where the first of those terms is $z$ (since the term "the expectation formula" was in the question, I took it to be about $E(Z|Z>c)$, which is specific to the normal. – Glen_b Feb 12 '14 at 22:16
  • (at least up to the obvious multiplicative constant, about that conditional expectation). However, $E(g(Z)|Z>c)$ for that particular $g=-\frac{d}{dz}log f$ is probably worth discussing in an answer. – Glen_b Feb 12 '14 at 22:21
  • 1
    Your latest edit asks for a proof (or intuitive explanation) of an incorrect statement. The _conditional_ density of $Z \sim N(0,1)$ conditioned on $Z > c$ is $\frac{\phi(z)}{1-\Phi(c)}\mathbf 1_{\{z\colon z > c\}}$ and the _conditional_ expected value is thus $$E[Z\mid Z > c] = \int_c^{\infty}z\frac{\phi(z)}{1-\Phi(c)}\,\mathrm dz = \frac{1}{1-\Phi(c)}\int_c^{\infty}z\phi(z)\,\mathrm dz$$ and not what you have in your revised title. – Dilip Sarwate Feb 13 '14 at 04:52
  • @DilipSarwate you're correct. My latest edit made it seems like a truncated normal, in which the density must be divided by $1-\Phi(c)$ as you suggested. In fact, the problem is about a censored model, in which the density of $Z$ is still standard normal. I'll add details. – Heisenberg Feb 13 '14 at 05:01
  • Almost the same question--with the same solutions--appears at http://stats.stackexchange.com/questions/166273. – whuber Sep 22 '16 at 04:19

1 Answers1

6

Would the Fundamental Theorem of Calculus work for you as intuition?

Let $\phi(x)$ denote the density function $\frac{1}{\sqrt{2\pi}}e^{-x^2/2}$ of a standard normal random variable. Then, the derivative is $\frac{\mathrm d}{\mathrm dx}\phi(x) = -x\phi(x)$. The Fundamental Theorem of Calculus then gives us that $$\phi(x) = \int_{-\infty}^x -t\phi(t)\,\mathrm dt = \int_{-x}^\infty u\phi(u)\,\mathrm du = \int_x^\infty u\phi(u)\,\mathrm du$$ where the second integral is obtained on substituting $u = -t$ and using the fact that $\phi(-u) = \phi(u)$ and the third upon noting that $\phi(-x) = \phi(x)$. Alternatively, write the second integral as the integral from $-x$ to $+x$ plus the integral from $+x$ to $\infty$, and note that integrating an odd function from $-x$ to $+x$ results in $0$.

Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200