3

I am trying to predict a strictly positive, continuous, right-skewed variable (purchase value) given a set of categorical attributes. The current popular options include: 1) log-transform the variable and run a (penalized) linear model. The issue is that the model is biased 2) run a poisson/negative-binomial (penalized) regression model because it deals with the bias, however it assumes an integer outcome.

Anyone who had more success with the former or the latter approach? The former is biased the latter loses information because of truncating values to integers but it is not clear to me when I should use one over the other.

user90772
  • 681
  • 1
  • 8
  • 19
  • Consider a Cox proportional hazards model (without censoring). See for example http://biostat.mc.vanderbilt.edu/wiki/pub/Main/FHHandouts/slide.pdf – Frank Harrell Jul 10 '17 at 20:06
  • Since you're presumably using generalized linear models to do your fitting in (2) why would you not simply use a continuous right-skew GLM, like say a gamma regression? To use a discrete model where there's a variety of continuous ones available seems odd. [Another alternative might be Weibull regression, which is often available in survival analysis packages, or if you don't want to specify a distribution, Frank's suggestion would work just as easily. If you're just looking at modelling the middle rather than information about the tails of the conditional distribution, it may not matter much] – Glen_b Jul 11 '17 at 00:42
  • In the case of the lognormal one, do you mean that the former is biased because of the penalizing or because of something else? (e.g. if you just exponentiate the fit that would be biased for the mean, but you can approximately adjust for that) – Glen_b Jul 11 '17 at 00:45
  • 1
    Thank you both for the detailed replies. I am trying to find some open-source implementation of GLMs with gamma distribution and also try the Cox proportional model. Regarding the bias, I was referring to that created because of the concavity of the log function (I am aware of the Smearing re-transformation but I thought that this is solved using a GLM in any case). Weirdly enough, a GLM with L1 regularization returns only the intercept (I am using 5-fold CV to pick the optimal regularization value). – user90772 Jul 11 '17 at 01:22
  • Should I still use RMSE/R^2 as my evaluation metric given the skeweness? – user90772 Jul 11 '17 at 13:41
  • It is not clear at all if I want to compare a log transformation over GLM with gamma, should I always back-transform the data in the original scale and then compare RMSE? Since RMSE is not robust to these extreme values, should I use MAE instead? My aim is predicting as close as possible both high and low values. At least some references would help, thank you in advance – user90772 Jul 14 '17 at 10:30

1 Answers1

3

To remove the bias, the solution is to use any generalized linear model. Poisson is just one of them but you can use many distributions.

A classical solution for continuous variables with the log link function is gamma regression that is a generalized linear model.

You can read more on this question : When to use gamma GLMs?

Benoit Sanchez
  • 7,377
  • 21
  • 43