2

I have two datasets, one on which I have my covariates $X_1$ and my observed outcome $Y_1$, and one on which I have only my covariate $X_2$. I want to predict $\hat{Y}_2$. However, I have far better indicators ($R^2$ and $AIC$ or $BIC$ for instance), if I estimate on the log transformed model :

$$ ln(Y_1) = \beta X_1 + u_1 $$

I can then predict $$ \widehat{ln(Y_2)} = \hat{\beta}X_2 $$

But then to obtain back my $\hat{Y}_2^*$ do I simply have to get $$\exp(\widehat{ln(Y_2)})$$

Or does the fact that the mean of a lognormal variable $Z$ (that is that $W=ln(Z)$ is normal) has a mean of $\exp(\mu_W+\dfrac{\sigma_W^2}{2})$ means that I have to correct my previous equation in something like $$\exp(\widehat{ln(Y_2)})+\dfrac{\widehat{\sigma_{u_1}^2}}{2})$$

Or am I mixing thins totally unrelated ?

Related question : What transformation of $\widehat{ln(Y_1)}$ given a structure of $u_1$ (I have likely heteroskedascity) would I need from to get $$ \mathbb{E}(Y_1)=\mathbb{E}(\hat{Y_1^*}) $$

EDIT : Improved notations thanks to Taylor's observation

EDIT2 : Added my related question

Anthony Martin
  • 1,109
  • 3
  • 11
  • 26

2 Answers2

2

You can exponentiate predictions from the log transformed model. That's fine. Like you say in this case your prediction will be $$ \hat{Y}_2^* = \exp\left[ \widehat{\ln(Y_2)}\right]. $$

Notice I am putting the hat over the natural log part. This is to stress the fact that exponentiating does not just merely undo taking the log. You seem to have sort of deceptive notation, regarding this. Also, I sort of find the numbering confusing as well.

As you note there will be "bias," because $$ E \left[ \exp\{\widehat{\ln(Y_2)}\} \right] \neq EY_1, $$ and the variance is not equal to what you may suspect: $$ \text{Var} \left[ \exp\{\widehat{\ln(Y_2)}\} \right] \neq \text{Var}(Y_1). $$ You may wish to approximate the latter with something like the Delta Method, or you can compute it exactly since you know $\exp\{\widehat{\ln(Y_2)}\}$ is log-normally distributed.

Edit:

If you try to transform with $\exp(\widehat{ln(Y_2)})+\dfrac{\sigma_u^2}{2})$, well first off you don't properly "know" $\sigma$, so it's impossible, but if you did, it would still be lognormally distrbuted and you could work out it's distribution. But be careful, you might mean subtracting $\sigma_u^2/2$. So if you know the variance, $$ \hat{Y}_2^{**}=\exp(\widehat{ln(Y_2)})-\dfrac{\sigma_u^2}{2}) $$ would have the right mean.

In practice, you might use some sample statistic $\hat{\sigma}^2_u$. That's fine, too, but it would not follow a log-normal distribution. There would be more work involved deriving properties of this guy.

Taylor
  • 18,278
  • 2
  • 31
  • 66
  • But If I were using my second transformation, my estimate would be unbiased ? Then wouldn't it be a better choice ? – Anthony Martin Feb 09 '17 at 15:42
  • @Aerandal see edit – Taylor Feb 09 '17 at 15:58
  • In my idea, $\hat{\sigma_{u_1}}^2$ would have been the estimated standard error of the perturbation of my first model $ln(Y_1)=\beta X_1 + u_1$. Is it a bad idea ? – Anthony Martin Feb 09 '17 at 16:15
  • @Aerandal no it's not bad. It might have better prediction error. You might be able to approximate the mean and variance of this with the Delta Method or somethign – Taylor Feb 09 '17 at 16:41
0

Just backtransforming with

$$exp(ln(\hat{Y_2}))$$

is ok, just think of your transformed variable as

$$Z = ln(Y)$$ and everything will suddenly look very ordinary. For standard errors you need to use the delta method I guess.

Knarpie
  • 1,522
  • 9
  • 22
  • If you need mean predictions on the original scale just backtransforming will be biased low, sometimes quite severely. – Glen_b Apr 08 '17 at 05:37