2

In this post, the accepted answer states that we need certain conditions before a second order Taylor series approximation is robust, due to the fact that the variance does not control higher moments.

The author goes on to say that

Therefore, in general, the Taylor series method fails even for 3rd degree polynomials. Iterating this argument shows you cannot expect the Taylor series method to provide accurate results, even for polynomials, unless all moments of your random variable are well controlled.

I'm not fully convinced of this statement and I'm wondering do we really need all moments to be well-controlled for a valid second order Taylor expansion. Because if we look at the formula for the remainder of a Taylor series truncated at second order, such here or here, it only uses the third moment $$ R_2(X) = \frac{f^{(3)}(\zeta)}{3!}E[(X-\mu)^3], $$ and the derivation of this formula does not use the higher moments of the random variable.

So have I overlooked something or is it indeed sufficient to have control of the third moment to ensure the Taylor series approximation is valid? Why would we need control of all moments as the accepted answer in the other post states when the remainder is derived without using higher order moments?

Edit: The formula for the 2nd order Taylor expansion is $$ E[f(X)] = f(\mu) + \frac{1}{2}f''(\mu)E[(X-\mu)^2] + R_2(X). $$

As far as I can see the higher order moments ($E[(X-\mu)^k]$ with $k \ge 4$) have not been used to derive the remainder in that formula so I believe that the formula is accurate as long as only the third moment is well-controlled, e.g. $E[(X-\mu)^3] \ll E[(X-\mu)^2]$.

So am I incorrect or is the linked answer, where the author says all moments of the random variable need to be well controlled, incorrect? If I am incorrect please demonstrate why the higher order moments need to be well-controlled for the above formula to be accurate.

sonicboom
  • 780
  • 1
  • 6
  • 13
  • Taylor expansion is guaranteed to works because the order of the error bounds from above the error of higher order terms. If we have undefined or diverging moments (which we can have) higher than the third one, that this is not guaranteed anymore. – Three Diag Mar 01 '21 at 17:34
  • 1
    The answer depends equally on the terms of the Taylor expansion and on the moments of $X.$ Perhaps it would be more fruitful to ask "how accurate is the Taylor expansion" [for whatever purpose you are using it for] rather than to ask about "validity," because the latter is not a well-defined or even quantitative criterion. – whuber Mar 01 '21 at 17:48
  • @ThreeDiag My point is that if we consider $E[f(X)] = f(\mu) + \frac{1}{2} f''(\mu) E[(X-\mu)^2] + R_2(X)$, the formula for $R_2(X)$ is derived without using higher moments of $X$, therefore, even if the higher moments are diverging, it won't make a difference: $E[f(X)] = f(\mu) + \frac{1}{2} f''(\mu) E[(X-\mu)^2]$ will still be an accurate approximation as long as it holds that the third moment (which appears in $R_2(X)$) is sufficiently smaller than the second moment. – sonicboom Mar 02 '21 at 10:03
  • @whuber See my comment above regarding my condition on the moments of $X$ - essentially, I am saying that if we have/assume the third moment of $X$ is sufficiently smaller than the second moment, then the truncated second order Taylor expansion is a good approximation (ignoring $f(\mu)$ and $f''(\mu)$ for the sake of argument). This is because the formula $E[f(X)] = f(\mu) + \frac{1}{2} f''(\mu) E[(X-\mu)^2] + R_2(X)$ does not make us of moments higher than the third moment. – sonicboom Mar 02 '21 at 10:08
  • I don't think so, the expression for R_2 is derived in a context where, for example your addition and subtractions are well defined operation. – Three Diag Mar 02 '21 at 11:04
  • The fallacy here is in supposing the expectation of $f(X)$ is not influenced by values of $X$ far from $\mu.$ That's not necessarily the case. – whuber Mar 02 '21 at 14:46
  • @ThreeDiag The subtractions and additions used to derive $R_2$ are well defined operations. I don't see why you would suggest they are not well defined. – sonicboom Mar 09 '21 at 10:11
  • @whuber I haven't made any assumptions on how the expectation of $f(X)$ is influenced by values of $X$ far from $\mu$, I'm simply working from the definitions. The higher moments don't get used anywhere in the formula (including remainder) for $E[f(X)]$ so how can they be a problem? – sonicboom Mar 09 '21 at 10:20
  • Suppose for the sake of argument that $E[(X-\mu)^3] \ll E[(X-\mu)^2]$ and that the higher order moments are all infinite $E[(X-\mu)^k] = \infty$ for $k \ge 4$. Then my hypothesis is that the 2nd-order truncated Taylor series for $E[f(X)]$ is an accurate approximation because the moments for $k \ge 4$ are irrelevant since they do not appear in the formula $E[f(X)] = f(\mu) + \frac{1}{2}f''(\mu)E[(X-\mu)^2] + R_2(X)$, or in its derivation. – sonicboom Mar 09 '21 at 10:22
  • Kind of like how a deterministic 2nd order Taylor series approximation of a function $g$ requires only that $g$ be 3 times differentiable. It doesn't matter if $g$ isn't 4,5,6,etc. times differentiable because only $g,g',g'',g'''$ are used in $g(x) = g(a) + g'(a)(x-a) + \frac{1}{2}g''(a)(x-a)^2 + R_2(x)$, with $g'''$ being used for $R_2(x)$. – sonicboom Mar 09 '21 at 10:27
  • The higher moments aren't a "problem" *per se,* but when they are large, that indicates there is substantial probability that $f(X)$ departs far from $E[f(X)]$ and that in turn means the Taylor approximation may be poor. Where your derivations lack rigor lies (partly) in the assumption that you have control over the integral of the remainder term. – whuber Mar 09 '21 at 14:57
  • I think the key thing is the assumption $E[(X-\mu)^3] \ll E[(X-\mu)^2]$ since then $E[f(X)] - (f(\mu) + \frac{1}{2}f''(\mu)E[(X-\mu)^2]) = R_2(X)$ and the assumption ensures the error $R_2(X)$ is very small (ignoring $f'''$ which we would ignore even if it was a deterministic Taylor series when talking about orders of magnitude). – sonicboom Mar 09 '21 at 15:45
  • 1
    When there's appreciable chance that $|X-\mu|\gg 1,$ that assumption (which requires some absolute values, btw) will be violated. – whuber Mar 10 '21 at 15:16

1 Answers1

1

The short answer is that in general, yes, we do need to control all moments (and ensure the function $f$ is sufficiently "nice"). The short reason that the logic above fails is that the location $\zeta$ is a function of $x$.

Take a close look at the definition of the explicit remainder terms in Taylor's theorem. It guarantees only that $\zeta$ lies between $\mu$ and $x$, and so the formula for the expectation of the remainder should be

$$ \mathbb{E}[R_2(X)] = \mathbb{E}\left[ \frac{f^{(3)}(\zeta(X))}{3!}(X-\mu)^3\right]. $$

The term inside the expectation remains a third-degree polynomial in $X$ if and only if $f$ itself is a third-degree polynomial. (Indeed, the remainder term is, by definition, the difference between the function and it's Taylor approximation.)

A simple counterexample

Consider the random variable $X$ with pdf

$$ p_X(x) = c \cdot \frac{1}{1+ x^6} $$

where $c$ is the normalization constant. It's an exercise to check that $X$ has finite first through fourth moments. Now, taking $f(x) = x^3$, we find

$$ \mathrm{Var\left[f\bigl(\frac{1}{n}X\bigr)\right]}=\mathbb{E}\left[f\bigl(\frac{1}{n}X\bigr)^2\right] = \frac{c}{n^3} \int_{-\infty}^\infty \frac{x^6}{1+x^6}\mathrm{d}x = + \infty $$

for any $n\in \mathbb{N}$, while the second-order Taylor approximation is uniformly zero.

Mike McCoy
  • 715
  • 4
  • 6
  • Just to add, we can obtain a valid 2nd order Taylor approximation of $f(X)$ as long as $|f^3(x)| < C$ for some constant $C > 0$ for all $x$. – sonicboom Jul 14 '21 at 08:43
  • I think you mean $|f^{(3)}(x)| < C$ (that is, bounded third derivative). But that alone is not strong enough, since you'll also need $\mathbb{E}[(X-\mu)^3] \ll 1$. Anyway, yes, there are conditions under which the Taylor series approximation is valid that avoid control of any moments—trivially, the Taylor series is valid if $f(x) = 0$. But in general (and most applications), you need to control of both the moments of $X$ and the regularity of $f$. Even a slow-growth function like $log(1+X)$ for $X>0$ won't have a valid Taylor series approximation without strong control on moments. – Mike McCoy Jul 14 '21 at 15:27
  • I do think that I've answered the question. :) The key point is that we cannot pull the $f'''$ outside of the expectation in the remainder term because it depends on $X$. This is where the higher-order moments of $X$ appear in the remainder. – Mike McCoy Jul 14 '21 at 15:32