Forecasting daily data with zeros in Python

Question

I'm currently testing some forecasts on daily sales quantities. However, out of ~2000 observations I have 16 zeros.

How should I approach this? It's mainly Sundays and holidays that holds zero as value. I want to perform some transformations to the time series that doesn't allow for zeros, why I'm looking for solutions.

Example of data:

             Sales_interior
CalendarDate                
2014-01-02       1066.000000
2014-01-03       1735.000000
2014-01-04       2538.000000
2014-01-05        952.000000
2014-01-06       1417.000000
2014-01-07       2205.000000
2014-01-08       1567.000000
2014-01-09       1464.000000
2014-01-10       1636.000000
2014-01-11       1979.000000
2014-01-12          0.000000
2014-01-13       1085.000000

EDIT: I'm currently planning on using a seasonal ARIMA.

If you are using models which allow for variables then you can include dummy variables which indicate holidays (and hence lower then normal or no sales). — user2974951, Aug 02 '19 at 07:26
In that case you can include variables, at least for some implementations. — user2974951, Aug 02 '19 at 07:54

score 0 · Answer 1 · answered Aug 05 '19 at 14:07

My suggestion is simply to exclude holidays or use a dummy as suggested in the comments. In finance for example, in most cases week-ends are excluded from the time-series. why should we model the sales where we are sure that there can be no sales due to holidays and store closures? The coefficients will be estimated in such a way that they will be constant across all the samples t, so they will be influenced to some extent by the calendar effect, and if you do not adjust for this, that effect will indirectly spill to the coefficient estimates to some extent (i.e. to some extent we could imagine that the true autocorrelation will be underestimated assuming that 0s are several compared to the total number of observations). So why taking into account dates where there cannot be sales for “external, calendar, reasons” not due to the true autocorrelation in the time series? Model this as a calendar effect because it is! So that you can isolate the calendar effect from the conditional mean effect due to time-series autocorrelation and make the estimate of the latter cleaner.

If instead 0 sales are not due to holidays, then my best advice is to leave those 0s, because it is “true information”, or at most treat them as outliers.

"to exclude holidays" can work if you are bringing in fixed-effects ( 6 in number) to deal with daily data impact BUT if you are trying to use autoregressive memory things get complicated as the frequency of interval is no longer constant which would argue for interpolated estimates (somehow) . — IrishStat, Aug 05 '19 at 14:40
I agree with the fact that the length of the interval is no longer constant in theory, but in practice you can simply assume that when there is a holiday it means that there is no business day and therefore no new info on the series. For example consider the financial case, where you study the prices given by the stock exchanges. If the stock exchange is closed then there is no price provided by any exchange. The next observation of the process will be the next price given by the exchange.. in this case it is almost the same: shops are closed so there is nothing new to observe (ie no process) — Fr1, Aug 05 '19 at 14:48
So, to avoid problems, let's redefine the process to be observed as "sales for every t, where t denotes business days instead of days" if we are sure that for each non-business day the sales are 0. In this case, the frequency is still constant if expressed as frequency in business days bevause it will be one observation per business day — Fr1, Aug 05 '19 at 14:52

score 0 · Answer 2 · answered Aug 28 '20 at 12:04

One possible solution could be to intrapolate the missing observations. One way how to intrapolate data in python is to use the scipy.interpolate package. For example, some spline intrapolation might nicely fill the gaps.

Of course, intrapolating data introduces some potential issues since we dont know what the real sales at that time would have been if the store was open. Moreover, since the store was actually closed the sales were actually zero. You could still combine this approach with the idea of adding dummy for these days to see if the results are in any way being driven the sales at interpolated dates.

score -1 · Answer 3 · answered Aug 05 '19 at 13:39

-1

simply add 100 to every observation to make every observation > 0 . Please read When (and why) should you take the log of a distribution (of numbers)? . Additionally there can be different lead and lag effects around EACH holiday.

answered Aug 05 '19 at 13:39

IrishStat

27,906
5
29
55

1

What justifies such an arbitrary recipe? Where did the value of 100 come from? – whuber Aug 05 '19 at 13:42
it guarantees all #'s will be > 0 . it could be .001 . it doesn't really make a difference ..it is arbitrary as far as I know ....unless you can cite otherwise from the literature. – IrishStat Aug 05 '19 at 13:44
https://stats.stackexchange.com/questions/195293/what-is-the-most-appropriate-way-to-transform-proportions-when-they-are-an-indep might be of help here – IrishStat Aug 05 '19 at 13:51
@IrishStat and simply subtract 100 from the forecasted series afterwards? – Artem Aug 05 '19 at 13:52
that would work for me ....BUT maybe I have something to learn here ...from @whuber – IrishStat Aug 05 '19 at 13:53
1

I would like to suggest that the analysis I presented at https://stats.stackexchange.com/a/30749/919 is relevant. It makes the case that the "start value" of 100 is not arbitrary and its choice matters. – whuber Aug 05 '19 at 13:54

Forecasting daily data with zeros in Python

3 Answers3