1

I have scenario where i have time series data (1 per day) for past 365 days. And I need to make a prediction for next 365 days. Is this possible using LSTM or any other ML models. I have been through some articles where we are predicting for the next few time steps (t+1,t+2,t+3) but not sure if i can predict the next 365 time steps. Any help on the approach for this will be appreciated. Thanks Atul PS: Please also suggest if there are any other methods like ARIMA etc

Atul
  • 13
  • 4
  • any help guys ? – Atul Oct 08 '19 at 04:49
  • ARIMA just uses memory . Often daily and monthly dummies and holiday indicators should be included. Look at https://stats.stackexchange.com/questions/313810/simple-method-of-forecasting-number-of-guests-given-current-and-historical-data/313852#313852 and post your data with country info and starting date. – IrishStat Oct 14 '19 at 09:59
  • sorry about the delay in my response. I was out due to a personal emergency. i do have data for weekends and holidays. Please let me know how i can share data. Thanks – Atul Dec 22 '19 at 06:20
  • please email it to me , see my profile – IrishStat Dec 22 '19 at 14:21

1 Answers1

0

You asked "Please also suggest if there are any other methods like ARIMA etc" , I answer here .....

I took your 366 daily values starting Sept 24 , 2018 enter image description here . Your LSTM is a "crunch approach" much like arima not yielding the source of the variation where daily data often has anthropomorphic effects due to the rhythm of life such as day-of-the-week , month-of-the-year , holiday effects , monthly effects et al.

You stated that you were concerned with maximum usage being exceeded. Max usage ALTHOUGH A HUMAN INTEREST is not a precise enough statistical objective … What you need to do is to specify a critical value and then ask what is the date that value will be exceeded at a particular confidence level. For example with 95% confidence when can I expect to exceed 2150. Restated what is the probability of exceeding 2150 on October 1st , 2019 or any other day in the next x days ?

Your data suggests very strong calendar effects with numerous anomalies/pulses with this being a partial listing of the model found model ... enter image description here

An overview of the Actual/Fit and Forecast is here with forecasts for the 149 days .enter image description here

The residuals from this useful model seem to be random reflecting model sufficiency which is supported by the acf of the residuals enter image description here enter image description here

The forecasts for the next 149 days are here with monte-carlo (simulated via bootstrapping ) limits that include the possibility of FUTURE ANOMALIES .

enter image description here

the simulated forecasts for period 1 ahead are here enter image description here and period 7 (OCTOBER 1 2019) are here enter image description here

There are many detectable/assignable causes to "explain" your time series ...Partitioning the forecast is the objective ...

For each period in the future one can examine the maximum of the 1,000 simulations performed if one so desired. Day 1 ahead has a maximum of 1925.8 while day 7 has a maximum of 1569.4.

Hope this approach helps you and others as to what can be done with SARIMAX models where the X's are the suggested miscellaneous daily effects and the I's are the unknown but identifiable dummy effects that should be investigated to better understand the data.

For example there is no discernable "trend effect over time" other than a constant however there are 3 daily effects enter image description here and 5 monthly effects enter image description here effects . Once you have adjusted for these effects and holiday effects and unusual values ( 24 of them enter image description here ) there is no discernable autocorrelation.

One comment about the effect of Holidays in India ... There are about 60 plus holidays and it would be necessary to suggest which ones might be appropriate for the data being analyzed and to provide a calendar of those historical and future events what can be used for statistically significant holidays ( lead and lags identified ). If you wish I can do this but you would have to provide some specifics in a csv file format.

The OP has shared his "worrisome LSTM model daily forecasts recursively quickly converging to a constant. " enter image description here . "The issue is that the model is using predictions as input to make new predictions. And after some predictions it is flattening out." says the OP.

In my opinion , this "autoregressive/autoprojective process" is the result of simply just using memory as the predictor rather than identifiable "features" which can have non-zero expectations in the future due to day-of-the-week , day-of-the-month , week-of-the-month , month-of the year and possible level/step shifts and local time trends AND holiday effects.

Forecasts without prediction limits is like "dinner without food" . To assess the probailoity of exceeding some critical avlue at some point in the future , one need to have reliably computed prediction intervals that are not necessarily based upon distributional assumptions.

IrishStat
  • 27,906
  • 5
  • 29
  • 55