I sometimes see people use taylor approximation as follows:
$$E(e^x)\approx E(1+x)$$
I know that the taylor approximation works for $$e^x \approx 1+x$$
But it is not clear to me that we can do the approximation inside the expectation operator. Intuitively, I guess it works if "the probability that $x$ is much larger than 0 is small", but I'm not sure how rigorous this is.
Edit: I am even more confused when we have an expectation of a function:
$$E(f(e^x))\overset ?\approx E(f(1+x))$$
For your specific example, the first order Taylor approximation around $x_0=0, e^x = e^0 +e^0x+R_1 = 1+x+R_1$, so
$$E(e^x) = E(1+x) + E(R_1)$$
So the question is "what can we say about $E(R_1)$?
Well, we do not know as much as we would like to about the Taylor approximation -meaning, about the behavior of the remainder.
See this example of why the remainder is a treacherous thing, but also, I would suggest to read through the very stimulating thread, Taking the expectation of Taylor series (especially the remainder) on the matter.
An interesting result in linear regression is the following: assume we have the true non-linear model
$$y_i = m(\mathbf x_i) + e_i$$
where $m(\mathbf x_i)$ is the conditional expectation function, $E(y_i\mid \mathbf x_i) = m(\mathbf x_i)$, and so by construction $E(e_i \mid \mathbf x_i) = 0$.
Consider the first-order Taylor approximation specifically around $E(\mathbf x_i)$
$$y_i = \beta_0+\mathbf x_i'\beta + u_i, \;\;\;u_i = R_{1i} + e_i$$
where $R_{1i}$ is the Taylor remainder of the approximation, the betas are the partial derivatives of the non-linear function with respect to the $\mathbf x_i$'s evaluated at $E(\mathbf x_i)$, while the constant term collects all other fixed things of the approximation (by the way, this is the reason why a) we are told "always include a constant in the specification" but that b) the constant is beyond meaningful interpretation in most cases).
Then, if we apply Ordinary Least-Squares estimation, we obtain that the Taylor Remainder will be uncorrealted to the regressors, $E(R_{1i}\mathbf x_i) = E(R_{1i})E(\mathbf x_i)$, and also $E(R_{1i}^2) = \min$. The first result implies that the properties of the OLS estimator for the betas are not affected by the fact that we have approximated the non-linear function by its first order Taylor approximation. The second result implies that the approximation is optimal under the same criterion for which the conditional expectation is the optimal predictor (mean squared error, here mean squared remainder).
Both premises are needed for these results, namely, that we take the Taylor expansion around the expected value of the regressors, and that we use OLS.
One situation in which this is used is asymptotics.
For example, suppose $\dfrac{X_n-\mu}{\sigma/\sqrt n}\sim N(0,1)$ and $g$ is a smooth function. Then $$ \frac{g(X_n)-g(\mu)}{\left|g'(\mu)\right|\sigma/\sqrt n} \overset{\mathcal L} \longrightarrow N(0,1) \text{ as } n\to\infty, $$ where $\text{“} \overset{\mathcal L} \longrightarrow \text{''}$ means convergence in distribution (also called convergence in law). In effect we're deleting the higher-order terms in the expansion $$ g(x) = g(\mu) + g'(\mu)(x-\mu) + \frac{g''(\mu)} 2 (x-\mu)^2 + \frac{g'''(\mu)} 6 (x-\mu)^3 + \cdots $$ and treating it as $$ g(x) \approx g(\mu) + g'(\mu)(x-\mu). $$ One writes $$ g(X_n) \sim AN\left( g(\mu), \frac{g'(\mu)^2\sigma^2} n \right) $$ where $\text{“}AN\text{''}$ means "asymptotically normal."
