2

I have monthly usage data (spanning 3 years) for a customer base of around 200K, and I need to generate 1-month ahead forecasts for each of them. There are a couple of exogenous variables that would need to be included too.

One way to go about is to build an "Arima with exogenous variables" model for each customer. In which case, I would be looking to manage 200K models. I'm wondering if there is an alternate approach to solving this --- Assuming we are able to segment the customer base into groups that are "homogeneous" in terms of usage, then would it make sense to create one model for each group? Would it be a good approximation to have the usage data time series for each group correspond to the mean values across the customers in that group?

Any pointers to literature related to this problem is appreciated. Additionally, I'm looking to use R, so pointers to specific packages will be of help as well.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
garamach
  • 43
  • 4

4 Answers4

2

The search term you need is "aggregation and disaggregation in forecasting". For instance, take a look at this paper by D'Agostino.

I think that practitioners tend to forecast aggregate then disaggregate it into parts. However, every problem is different and you may decide one way or another depending on your circumstances.

Consider forecasting house prices in US. You could build a model for nation wide price index, then use its forecast as input into forecasts of regional house prices. When you do this there's an issue of forecast consistency: you'd want the regional forecasts to aggregate back into the national, and it's very difficult. So, many practitioners allow for discrepancies in this regard.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
1

"Assuming we are able to segment the customer base into groups that are "homogenous" in terms of usage, then would it make sense to create one model for each group? " ..... I don't think so .. Managing and developing/reusing models is bread and butter stuff . No need to be afraid of what I think is routine.

IrishStat
  • 27,906
  • 5
  • 29
  • 55
1

One way to go about is to build an "Arima with exogenous variables" model for each customer. In which case, I would be looking to manage 200K models.

I always find it strange when practitioners suggest fitting a large number of individual models to individual objects --- in this case 200K models! A good statistical model should describe the overall dataset, allowing for variation between individuals. If you have covariate information on the individuals that would allow you to differentiate them and group them by characteristics then presumably it would be possible to formulate a single model for all the customers, which uses that covariate information. When dealing with time-series data for multiple individuals it is also common to see correlation in the series of a single individual, and this can be accommodated by using hierarchical models, adding "random effect" terms for each individual, or by other similar methods.

Without more information, it is not really possible to say what will be the best model, but I am always sceptical when I see practitioners segment data down into tiny parts and then apply ad hoc models to the parts. That is a method that risks loss of information (since the model excludes data from other individuals) and also risks over-fitting (since models are tailored to small parts). As a first pass with this kind of data, I would suggest fitting a time-series model that includes the covariate variables for the individuals, and also includes some kind of random effects terms to induce correlation between the actions of a single individual over time. This should give you some idea of the predictive effect of the covariates, and will highlight whether there are any residual variations in individuals that can't be adequately described by simple random effects. Regardless of the particular model you end up with, my view is that it is best to approach the problem by seeking to accommodate the entire dataset in a single model.

Ben
  • 91,027
  • 3
  • 150
  • 376
0

I think you're asking whether you should try complete pooling or no pooling. There's actually a good middle ground: hierarchical Bayes.

http://www.stat.columbia.edu/~gelman/arm/ is a great, easy-to-read source.

In R, the lmer function in the lme4 package is good for hierarchical models. I've never tried it with that much data, though. In general, OpenBUGS is more robust.

hahdawg
  • 1,142
  • 1
  • 10
  • 16
  • Thanks. This is worth exploring further. Although I need to check if lmer() allows incorporating lagged dependent variables as predictors. I was hoping to start with something that is relatively simpler, like dynlm(), which I found out after I posted my question! With the plyr package, it seems I can manage to extract the coefficients of individual models into a data frame, and subsequently do the forecasting for each customer. – garamach Sep 05 '14 at 03:42
  • Besides `lme4` check also `nlme` because it allows `AR` and `ARMA` variance-covariance matrix. Btw, check this answer: http://stats.stackexchange.com/questions/129783/is-this-logit-model-a-multilevel-model-and-what-is-the-correct-way-to-model-it/129839#129839 – Tim Dec 23 '14 at 20:38