0

I was hoping I could get some guidance on a forecasting problem that I am working on. I have data with a daily frequency & I have a number of time series that follow very distinct patterns. For example:

enter image description here

This time series in fact has a value for the last Monday of every month and zero otherwise. I have used a number of models that use this information but I cannot get my forecast anywhere close to what I would expect.

Note that I have not tried any naive models where I am pretty sure they would do fine but I was interested in seeing if I could produce a forecast with modeling methods (ARIMA, Linear Regression, ETS, etc.). I have tried ARIMA with harmonic features as well as dummies indicating mondays, last week of the month & both. Here is an example of my linear regression forecast that also uses these features:

enter image description here

The timing looks right but the magnitude of the forecast is far off the mark. I am hoping for some guidance and all comments are welcome. Thank you!

[Update]: Yes I suppose there is something else on my mind that I should make more clear. I have hundreds of these time series with similar properties but this was just one example. The challenge with others are bi monthly dates weekday dates, some have some noise where omitting the zeros would not be optimal in each case. What I am trying to do is create an overall automated solution.

RDizzl3
  • 753
  • 9
  • 26
  • 1
    You have not really got a daily series but a monthly series (visually you do not even have a length of month effect). So you want to forecast that and then spread it to the appropriate day in the month. You can get "last Monday of the month" by multiplying your day of the week dummy variables by your "last week of the month" dummy as to when in the month – Henry Nov 17 '21 at 02:26
  • 1
    If you *know* your time series will be zero everywhere except for the last Monday in every month, then you should forecast a flat zero, except for the last Monday in every month - and for that, just collect all the nonzero historical data in a monthly time series and use standard methods like exponential smoothing. I hesitate to post this as an answer because it appears to be so obvious to me, so I assume I am misunderstanding something about your problem. Can you clarify why you would not try this approach? – Stephan Kolassa Nov 17 '21 at 07:41
  • Hi @StephanKolassa - Yes I suppose there is something else on my mind that I should make more clear. I have hundreds of these time series with similar properties but this was just one example. The challenge with others are bi monthly dates weekday dates, some have some noise where omitting the zeros would not be optimal in each case. What I am trying to do is create an overall automated solution. – RDizzl3 Nov 17 '21 at 14:48
  • 1
    Hm. If some of your series exhibit a "last Monday of the month" pattern, others an "every other Tuesday", yet others "the tenth day of each quarter", then your best bet is likely to figure out which patterns are prevalent in your dataset, then try to detect which pattern a given time series belongs to (even a "last Monday of the month" one may have a few data points that do not match this pattern, so include some fuzziness), and finally use a tailored time series forecasting algorithm. I don't think a "general" algorithm will have a good chance of finding your patterns. – Stephan Kolassa Nov 17 '21 at 14:59
  • 1
    [How to know that your machine learning problem is hopeless?](https://stats.stackexchange.com/q/222179/1352) – Stephan Kolassa Nov 17 '21 at 14:59
  • Thank you @StephanKolassa – RDizzl3 Nov 17 '21 at 15:46

0 Answers0