6

There are 2 increases in magnitude (one between feb and march, and one at the beginning of september) enter image description here

(the chart has daily resolution btw)

Assuming weekly seasonality, is it ok to get the exact day when the increase was noticed and the day 1 week earlier, compute a scaling factor new/old and scale everything before the increase happened with that scaling factor ?

How is this operation usually called ? How do people usually do this, is it just this sort of scaling or is there a more correct(and perhaps more sophisticated) approach for this ?

What would you use for adjusting this time-series in either R or Python and statsmodels ? Is there a worked out example of this somewhere that I could read or could you describe one here ?

(The two increases have a common cause. The chart shows counts of certain posts on a website. At the times of the two increases, there was a new roll-out of a collector and as a result, more data started coming in. So the scaling I'm trying to do is trying to estimate how the time-series would have looked prior to the increases if the current collection were in place from the beginning, from January 2015)

wsdookadr
  • 181
  • 3
  • In statsmodels you can just add a indicator variable (0 before, 1 after the change date) as explanatory variable `exog` to ARMA, SARIMAX and similar models. Adding an additional linear trend could be appropriate to avoid that the indicator variable also captures some trend effects. A not very clean example of adding explanatory variables (using splines for seasonal effects) is at https://gist.github.com/josef-pkt/1ea164439b239b228557 The ARMAX version implemented in statsmodels is a linear model with ARMA errors. – Josef Sep 30 '15 at 16:39
  • (Out of sample prediction in ARMA with explanatory variables requires statsmodels master or the yet unreleased 0.7 version because it fixes a timing bug.) – Josef Sep 30 '15 at 16:41

1 Answers1

2

Detecting and incorporating Level Shifts (Step Shifts) is an integral part or both non-causal and causal modelling. The term Intervention Detection is often used. Some software implementations restrict identification to non-causal models while others do not. See http://www.unc.edu/~jbhill/tsay.pdf for a seminal article and search here for relevant posts using the string "intervention detection" . Typical output would include "what the series would have looked like without the intervention" . These are often called the "adjusted values" .

There may be a "de facto date" and a "de jure date" to the intervention, often coinciding but sometimes not due to dynamics in the series. If one knows the start date then one can simply add an indicator variable ( 0,0,0,0,1,1,1,1..) to reflect that knowledge and then use the estimated coefficient for that variable as the adjustment factor.

If you knew that an intervention had occurred at a particular point in time you could estimate a model of the form where I is the 0/1 intervention variable is used in conjunction with the ARIMA component.

For example if you had a pulses at two points in time (39 and 21) the augmented data matrix might look like .

enter image description here

enter image description here

IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • How would you add this indicator variable in practice ? Could you please write a bit more about how this is done in practice ? – wsdookadr Sep 30 '15 at 11:21
  • what does the formula you wrote above mean and where can I read more about it ? – wsdookadr Sep 30 '15 at 12:30
  • The formula includes m distinct series , each one a 0/1 series stored in the appropriate I vector. The polynomial L(B) contains the weights/coefficients for the particular j series. . If the model is a constant plus a pulse indicator plus a random error we would have m=2 , L11= 3.2 and L21=10.0 and I1=(1,1,1,1,1,.....1) and i2=(0,0,0,0,1.0.0.0...0) where the "1" represents the point in time of the pulse that was 10 units higher than the overall mean of 3.2 – IrishStat Oct 03 '15 at 17:37