3

I'm looking to forecast (impute) missing discrete values in multiple time series in order to reach a target volume in a consolidated time serie.

The context:
I have salesmen that are selling contracts in some markets over time.
For some markets, I have gaps with missing data.

ex below: the volume of contracts over time (days) for a given market

enter image description here

I'm pretty confident to fill gaps on this market at the consolidated market level and obtain something like below. (Here I've just print a very correlated market, and there are plenty other)

enter image description here

The problem:
I'm now looking for a method able to estimate the discrete volume of contracts over days made by and for each possible salesmen on gaps period in order to reach the volume of the market I will estimate.

Another aspect to consider, is the number of salesmen on this market: roughly 5000.
As a result, many of them are "opportunist" salers with very sparse discrete activity.
Here is a tiny sample of different salers code activity for the "most productive" (yeah...). enter image description here

In my context, Is there a method (several?) in order to reach my goal?

e.g: Impute the most probable salers discrete activity according to:
- an estimated market volume over time which integrate seasonality and act as a target (the sum of all salers activity has to be equal to this target over time)
- the known activity of the salers around the gap ("past activity" before, and "future activity" after the gap)

I'm thinking and have read some things about Hidden Markov Chains, Kalman filters, ARIMA models, LSTM, Poisson regression, but I'm not an expert, and I don't really know how many methods I have to use and above all, how I have to combine them.

How would you proceed?

Thanks, Mathieu

matlabat
  • 31
  • 2

1 Answers1

1

If I had your problem and my favorite piece of software , I would develop two daily models for the two regimes of observed data. The first model would predict the missing observations using approaches described here .. Forecast customer's spending and elsewhere ( see my previous posts on daily data ). Then I would take the data from the second regime and reverse the order so that the missing past would be predicted from the future ( so to speak ! ) How to treat days when shops are closed in sales time series? and Regression on default data and backward extrapolation . I would then average these two sets of forecasts to create pseudo data for the missing/omitted period.

As a coup de gras , I would then model the entire data set and use the estimated values for the missing period as my final estimates for the omitted data.

I owe this insight to a T.W Anderson https://www.google.com/search?source=hp&ei=q43sWofxF-2b5wKbqYKgDQ&q=t.+w.+anderson&oq=t.+w.+anderson&gs_l=psy-ab.3..0j0i22i30k1l9.1358.7732.0.8947.17.16.0.0.0.0.85.1110.15.16.0..2..0...1.1.64.psy-ab..1.16.1198.6..46j35i39k1j0i131k1j0i20i264k1j0i46k1.88.N5aeGxP-0n8 a guiding light in my pursuit of time series solutions/procedures/theory.

In direct response to your question about 5,000 individual time series , I can only suggest forecasting some aggregate and then pro-rationing the estimates/fitted values in order to estimate sub-count data used to form the aggregate.

IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • Thanks IrishStat for your answer, I will dig into your materials for the 1st part. Actually, I was more interested in a solution **for the 5000 time series** in order to **discretize them with the constraints to reach the volume of the aggregate**. Do you have any recommandation? – matlabat May 07 '18 at 07:36
  • if i had 5000 series and I could logically separate them into say k groups. I would take the group1 aggregate and model it to get estimated values for each period ( the fitted values) . I would then individually examine each series and replace missing values with a simple proration using the fitted value and the relative % that each series was to the comprehensive aggregate over time. In other works say series 1 in group 1 had a total observed volume of 100 based upon the history. I would use the sum of all volumes for that group over time as the basis and compare the 100 to that . – IrishStat May 07 '18 at 09:09
  • Hi IrishStat, Thanks a lot for your suggestions and your proposition of chat. Actually and In parallel of this post, I'm also working with data engineers to possibly reload the real data (which would be the best). So, depending on the efforts both approach take, I may contact you to implement the "artificial solution". Regards, Mathieu – matlabat May 14 '18 at 09:47