5

I have some daily sales from 2018-01-01 to 2021-10-21 and I'm trying to predict the sales a year into the future. I opted for facebook prophet. My raw data looks like this:

enter image description here

According to a DF-test, the series is stationary. However according to the documents, prophet does not need stationarity to be efficient. The forecast and fit looks like this:

enter image description here

Clearly prophet is not good at capturing the spikes of the data. Looking at the mean absolute percentage errors over a horizon it just looks horrible:

enter image description here

The mean of these MAPE's is a whopping 53%, and I'm hoping one something at around 5%. Does anyone have a clue on what I can do in order to improve on this model? Obviously if I take the logarithm of the sales the relative error will decrease, but if I inverse transform it back to original it's still quite off in predictions.

EDIT:

Here is the updated forecast with holidays inserted. Seems as if the spikes are better captured but it's several hundreds of thousands of dollars in difference between predicted and actual. I get an RMSE of 1 281 915.

enter image description here

Parseval
  • 295
  • 1
  • 7
  • 1
    A better model might predict another Black Friday spike but looking at your data, this spike was more than twice as big in 2020 compared to the other years. There is no way to predict that from the data or to predict whether the spike in 2022 will be more like 2020 or more like the other years. Maybe some out of the model information can predict that but the raw sales data cannot. This difference on its own already gives you a high MAPE. – quarague Nov 04 '21 at 09:12

1 Answers1

13

You have suspiciously regular massive spikes sometime about halfway through the third quarter of every year. You don't tell us where your data come from, but if they are US, this is presumably a Black Friday effect. Have you told your model about this very specific predictor?

In general, How to know that your machine learning problem is hopeless? may be helpful. You will need to understand your data and include any relevant predictors you have. Hoping for 5% MAPE may simply be unrealistic.

Also, note that your model will try to separate noise from signal and predict only the signal, with the result that your predictions will vary less than observations. Here is a recent thread on this.

Finally, you may want to take a look at What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? I don't know which objective function Prophet optimizes for, but I assume it is not MAPE. Thus, if your bonus depends on having a low MAPE, you may get closer by post-processing the Prophet forecasts. (I have never seen a business problem that would benefit more from MAPE-optimal forecasts rather than, e.g., MSE- or quantile loss-optimal forecasts. And as you see at that thread, optimal forecasts can depend quite heavily on the evaluation measure.)

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • 2
    That is exactly correct, those are Black Friday spikes! No I have not told my model this because I was unsure how to. Is it in the same way as they do in the prophet-docs where they specify superbowl as special event? – Parseval Nov 03 '21 at 10:54
  • 1
    I have no experience with Prophet, but I would assume that that would exactly the way to go about this. – Stephan Kolassa Nov 03 '21 at 10:56
  • 3
    +1. Just to close the loop on this: @Parseval That documentation is 100% what you would follow for adding extra variables and it will fit them like a normal regression along with the other components essentially like a GAM. Prophet is bayesian so the objective is the MAP. In terms of the raw accuracy, Prophet has a lot of pros but that isn't one of them. It is typically outperformed pretty handily by other methods like smoothers so you could give them a shot. – Tylerr Nov 03 '21 at 12:58
  • @Parseval You can add BF as a custom holiday in Prophet. – dimitriy Nov 03 '21 at 15:02
  • @Tylerr - I've updated my forecast with a figure capturing black weeks and christmas holidays. It seems to capture the spikes a little better but the mape is still 53%. I also tried an LSTM but it was very bad, I don't understand how some authors on TowardsDataScience can get nearly spot on predictions. Do you have any good source where I can see how smoothers are implemented? Never heard of these. – Parseval Nov 04 '21 at 09:58
  • @Parseval smoothers are just the standard simple, double, triple exponential smoothing methods you can see some examples using statmodels: https://www.statsmodels.org/dev/examples/notebooks/generated/ets.html#:~:text=The%20ETS%20models%20are%20a,can%20be%20used%20with%20statsmodels%20. So with seasonality you would use holt-winters. One issue with this is that you no longer can use exogenous factors directly with the methods. Is this data public or something you could share? As the answer mentioned there may be no real 'good' method just a 'best' one. – Tylerr Nov 08 '21 at 18:32