16

I am impressed by the R forecast package, as well as e.g. the zoo package for irregular time series and interpolation of missing values.

My application is in the area of call center traffic forecasting, so data on weekends is (nearly) always missing, which can be nicely handled by zoo. Also, some discrete points may be missing, I just use R's NA for that.

The thing is: all the nice magic of the forecast package, such as eta(), auto.arima() etc, seem to expect plain ts objects, i.e. equispaced time series not containing any missing data. I think real world applications for equispaced-only time series are definitely existent, but - to my opinion - v e r y limited.

The problem of a few discrete NA values can easily be solved by using any of the offered interpolation functions in zoo as well as by forecast::interp. After that, I run the forecast.

My questions:

  1. Does anyone suggest a better solution?

  2. (my main question) At least in my application domain, call center traffic forecasting (and as far as I can imagine most other problem domains), time series are not equispaced. At least we have recurring "business days" scheme or something. What's the best way to handle that and still use all the cool magic of the forecast package?

    Should I just "compress" the time series to fill the weekends, do the forecast, and then "inflate" the data again to re-insert NA values in the weekends? (That would be a shame, I think?)

    Are there any plans to make the forecast package fully compatible with irregular time series packages like zoo or its? If yes, when and if no, why not?

I'm quite new to forecasting (and statistics in general), so I might overlook something important.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
entreprogreur
  • 161
  • 1
  • 1
  • 4
  • Welcome to the site and to forecasting! Real world applications for equispaced-only time series are definitely not *very* limited. I happen to know a little about the forecasting that goes into your supermarket having enough product on hand to deal with promotional demand, and believe me, those millions of time series (20,000 SKUs in 1,000 stores is very common) are very equispaced indeed. (Sorry, but you kind of asked for it...) But I'll try to come up with something more helpful to you in a minute. – Stephan Kolassa Jan 07 '13 at 20:53
  • 2
    Could you be more explicit as to why call center data is not equispaced? (Perhaps I am misunderstanding what you mean by "equispaced".) The call center forecasting methods I have seen usually bucket incoming calls into 15 minute intervals, which fulfills my definition of "equispaced". We then have to deal with complex seasonality (intra-daily, intra-weekly, yearly), for which topic this may help you: http://stats.stackexchange.com/questions/44704/handling-forecast-mean-with-complex-seasonality Does this answer your question? If not, just tell us what else you need. – Stephan Kolassa Jan 07 '13 at 21:02
  • 5
    `auto.arima` can handle missing values. – Rob Hyndman Jan 08 '13 at 02:43
  • 1
    Thanks for all the constructive comments! Stephan, my data is not equispaced in two ways: 1. Many call centers are just closed on Saturdays and Sundays. Some are closed on just Sundays. So the "normal" space between two adjacent data points is one day, except from Fri to Mon, which is three days. So the space is not equal, i.e. not equispaced. Second, there can be just random missing data somewhere because they just forgot to turn on their measuring device on that day or whatever. I hope that makes my point clear. – entreprogreur Jan 09 '13 at 09:09
  • Thanks, Rob, for the note that auto.arima can handle missing values. However, the weekends (that are always missing in my case) will be forcast with rising values in my sample data. I am looking for a way to safely have NA or 0 for weekends also in the forecast. Furthermore, I need ARIMA as well as EST / HoltWinters... I assume there are no plans whatsoever to extend the whole forecasting package to handle zoo and/or its as well as ts? – entreprogreur Jan 09 '13 at 09:11
  • 1
    Just a (provocative) thought. If you say call centers are closed on the week-ends. Then you have no missing data. Your time-series spans Mo-Fr. 5 days. quite equidistant. Interpolating the weekends seems to me formally wrong, since you know that no calls occur and any information you impute is false. I would argue that you can never improve an estimate by inventing data... – means-to-meaning Nov 04 '13 at 22:17

3 Answers3

2

You should be very careful when you apply interpolation before further statistical treatment. The choice you do for your interpolation introduces a bias into your data. This is something you definitely want to avoid, as it could alter the quality of your predictions. In my opinion for missing values such as those you mentioned, that are regularly spaced in time and that correspond to a stop in the activities, it might be more correct to leave these days out of your model. In the the little world of your call center (the model you are building about it), it might be better to consider that time simply stopped when it is closed instead of inventing measurements of a non-existing activity. On the other hand the ARIMA model has been statistically built on the assumption that data is equally spaced. As far as I know there is no adaptation of ARIMA to your case. If you are just missing a few measurements on actual working days, you might be forced to use interpolation.

Remi D
  • 275
  • 2
  • 9
1

I am not an R expert so maybe there is a simpler way but I have come across this before. What I did before is implement a function that measures the distance (in time units) between the actual dates and saves that in a new column in the existing time series. So we have something like:

index/date | value | distance  
01.01.2011 |  15   |   1  
02.01.2011 |  17   |   3  
05.01.2011 |  22   |   ..

This way, if your time series is not yet associated with an actual series of points in time (or wrong format or whatever), then you can still work with it.

Next, you write a function that creates a new time series for you, like so:

First, you calculate how many units of time the time series actually would have between the dates of your chosing and create that timeline in zoo or ts or whatever the choice is with empty values.

Second, you take your incomplete time series array and, using a loop, fill the values into the correct timeline, according to the limits of your choosing. When you come upon a row where the unit distance is not one (days (units) are missing), you fill in interpolated values.

Now, since this is your function, you can actually chose how to interpolate. For example you decide that if the distance is less than two units, you use a standard linear interpolation. If a week is missing, you do something else and if a certain threshold of missing dates is reached, you give out a warning about the data - really whatever you want to imagine.

If the loop reaches the end date you return your new ts.

Advantage of such a function is that you can use different interpolations or handling procedures depending on the lengths of the gap and return a cleanly creates series in the format of your choosing. Once written, it allows you to gain clean and nice ts out of any sort of tabular data. Hope this helps you somehow.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
IMA
  • 1,152
  • 5
  • 7
  • Thanks, IMA, for the helpful answer! So, what I'm doing now: for discrete missing values, I use interpolation (as well as user-provided "adjustments") to fill the missing data. IMA, your answer in further enhancing that is very helpful. For "regular" missing data such as weekend, I transform my data into a second, "pseudo" ts just for forecasting purposes, and then transform the result back to the "correct" time series, so that also the forecast will have missing values in the weekends. I would still be grateful for a more elegant suggestion on how to handle regular "gaps" in the weekends. – entreprogreur Jan 09 '13 at 09:23
  • @entreprogreur, I did not answer, IMA did. IMA get's full credit here. I just tweaked the formatting so that it would display nicely. – gung - Reinstate Monica Jan 09 '13 at 13:15
0

I would not interpolate the data before estimating the model on this data, as @Remi noted. It's a bad idea. An extreme example: imagine you have two data points Jan 2013 and Jan 2014. Now interpolate 10 monthly points in between: Feb through Dec 2013, and run regression on the monthly date. In reality it's not going to be this bad, but it's the same idea: you'll be inflating your statistics at best.

The way to go is to use time series methods which handle missing data. For instance, state space methods. Take a look at astsa R package. It comes with an excellent book on time series analysis. This will handle missing data nicely. Matlab now has a similar functionality in ssm package. You have to learn converting your models into state space form, but you have to learn this anyways if you want to step away from auto.arima "magic".

Aksakal
  • 55,939
  • 5
  • 90
  • 176