7

tl;dr: There are two commonly reported formulas for approximating $\text{var}(f(X))$, but one is notably better than the other. Since it isn't the "standard" Taylor expansion, where does it come from, and why is it better?

Details: Let $X$ be a real random variable and $f:\mathbb{R}\to\mathbb{R}$. There is a standard way to approximate $\text{var}(f(X))$ using Taylor expansions of moments, e.g. $E f(X)$, $E f^2(X)$, etc. Doing so yields the following second-order approximation: \begin{align} \operatorname{var}(f(X))\approx [f'(EX)]^2\operatorname{var}(X)-\frac{[f''(EX)]^2}{4}\operatorname{var}^2(X) := V_1. \end{align}

For a formal proof, see this post and answer. Oddly, this is not the formula quoted in the corresponding Wikipedia page: \begin{align} \operatorname{var}(f(X))\approx [f'(EX)]^2\operatorname{var}(X) + \frac{[f''(EX)]^2}{2}\operatorname{var}^2(X) := V_2. \end{align}

Note the difference in the coefficient of second term: $-1/4$ vs $+1/2$.

At first, I assumed this was a typo in the Wikipedia page. After running some quick simulations, however, it seems that the second approximation $V_2$ is much better than $V_1$! (Admittedly I did not run exhaustive tests but after a few dozen examples the difference was quite clear.)

In fact, here is a partial explanation for why $V_1$ often fails catastrophically: If $f'(EX)\approx 0$, then $V_1 < 0$. Presumably, this can be corrected by using a third-order approximation.

My question, though, is (a) How do we derive $V_2$, and (b) Why does it outperform $V_1$?

JohnA
  • 642
  • 5
  • 12

1 Answers1

3

I cannot speak to the derivation of the first approximation (which looks wrong to me). However, the second equation is obtained using a second-order Taylor approximation to $f$ for the case where the underlying distribution is centred, unskewed and mesokurtic. In this case, you have $\mu=0$, $\gamma=0$ and $\kappa=3$. Using the general form of the Taylor approximation you obtain:

$$\begin{equation} \begin{aligned} \mathbb{V}[f(X)] &\approx ( f''(\mu)^2 \mu^2 - f'(\mu)f''(\mu) \mu + f'(\mu)^2 ) \cdot \sigma^2 \\[6pt] &\quad - \frac{f''(\mu)(f'(\mu) + \mu f''(\mu))}{2} \cdot \gamma \sigma^3 + \frac{f''(\mu)^2}{4} \cdot (\kappa-1) \sigma^4 \\[6pt] &= f'(\mu)^2 \cdot \sigma^2 + \frac{f''(\mu)^2}{2} \cdot \sigma^4. \\[6pt] \end{aligned} \end{equation}$$

The first approximation does not look correct to me, and I see no evidence that it is a "commonly reported formula". This approximation cannot be derived from the general second-order Taylor approximation for any assumed level of kurtosis, so I find it unsurprising that it performs poorly. (It would require $\kappa = 0$ which is not a valid kurtosis value.) For this reason, I would expect the second approximation to perform better than the first, except possibly in the case where the kurtosis of the underlying distribution is highly platykurtic.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • 1
    There is a link to a derivation of the first approximation, which is correct and cited in textbooks (e.g. 4.3.2 of Benaroya, Mi Han, Nagurka. _Probability Models in Engineering and Science._). Your approach (which I have not seen in any references myself) seems much more complicated than the standard approximation, and it's also a bit strange that your second-order expansion involves higher-order moments, whereas the standard one does not. Can you clarify any of this? – JohnA Mar 04 '20 at 13:47
  • It is natural for the second-order expansion to involve moments up to fourth order. The second order expansion is a quadratic involving $X^2$ and the variance of this random variable is affected by the first four moments of the distribution. – Ben Mar 04 '20 at 22:44
  • Certainly, even $V_1$ also depends on higher order moments through $\text{var}(X)^2$ as well. I should have been more clear -- your claim is that $V_2$ is a special case of a more general formula; but $V_1$ is valid for any $X$ (i.e. not just normal or $\kappa=0$, etc.). So then maybe $V_2$ is a better approximation for Gaussian $X$, but not necessarily in general? I still find the formula for $V_1$ odd, although, contrary to your answer, it is a correct approximation. – JohnA Mar 05 '20 at 15:05
  • I'm not sure what you mean when you say it is a "correct" approximation. According to your own description, it performs poorly. (Moreover, there are no distributions with $\kappa=0$.) – Ben Mar 05 '20 at 20:58
  • If you do a second-order Taylor expansion with no assumptions on $X$ (besides existence of necessary moments), then you get $V_1$. My previous comment contains references for this derivation. That's not to say you can't obtain other approximations, of course (e.g. $V_2$). – JohnA Mar 05 '20 at 21:06
  • I could assert that $42$ is a "correct approximation" if I wanted to --- anything can be asserted to be an approximation if there are no performance requirements on "correctness". The method used for $V_1$ is to take seperate Taylor estimates of $\mathbb{E}(f(X)^2)$ and $\mathbb{E}(f(X))$ and then combine these. That method yields a poor approximation that is outperformed by $V_2$, except possibly for highly playkurtic distributions (and in this case, the general formula I have shown is still better). The problem with $V_1$ is that it approximates the raw moments seperately. – Ben Mar 05 '20 at 22:07