Difference time series and then minus the mean of the differenced series within Arima

Question

This question is similar to the following question in the sense I am currently doing the differencing and mean removal of the time series outside the Arima function in R. And I do not know how to do these steps within Arima function in R. The reason is that I am trying to perform the following procedure (data dowj_ts can be found at the bottom):

dowj_ts_d1 <- diff(dowj_ts) # differencing at lag 1 (1-B)
drift <- mean(diff(dowj_ts))
dowj_ts_d1_demeaned <- dowj_ts_d1 - mean(dowj_ts_d1) # mean removal
# Maximum Likelihood AR(1) for the mean-corrected differences X_t
fit <- Arima(dowj_ts_d1_demeaned, order=c(1,0,0),include.mean=F, transform.pars = T)

Note that the drift is actually 0.1336364. And summary(fit) gives the table below:

Series: dowj_ts_d1_demeaned 
ARIMA(1,0,0) with zero mean     

Coefficients:
         ar1
      0.4471
s.e.  0.1051

sigma^2 estimated as 0.1455:  log likelihood=-35.16
AIC=74.32   AICc=74.48   BIC=79.01

Training set error measures:
                       ME     RMSE       MAE       MPE     MAPE      MASE
Training set -0.004721362 0.381457 0.2982851 -9.337089 209.6878 0.8477813
                    ACF1
Training set -0.04852626

Ultimately, I want to predict 2-step ahead forecast of the original series, and this starts to become ugly:

 tail(c(dowj_ts[1], dowj_ts[1] + cumsum(c(dowj_ts_d1_demeaned,forecast.Arima(fit,h=2)$mean) + drift)),2)

And currently these are all done outside the Arima function from the forecast package. I know I can do differencing within Arima like this:

 Arima(dowj_ts, order=c(1,1,0),include.drift=T,transform.pars = F)

This gives:

Series: dowj_ts 
ARIMA(1,1,0) with drift         

Coefficients:
         ar1   drift
      0.4478  0.1204
s.e.  0.1059  0.0786

sigma^2 estimated as 0.1474:  log likelihood=-34.69
AIC=75.38   AICc=75.71   BIC=82.41

But the drift term computed by R is different from the drift = 0.1336364 that I computed manually.

So my question is: how can I differenced the series and then remove the mean of the differenced series within the Arima function ?

Second question: Why is the drift term estimated by Arima different from the drift term I computed ? In fact, what does the mathematical model look like when include.drift = T ? This really confuses me.

Data can be found below:

structure(c(110.94, 110.69, 110.43, 110.56, 110.75, 110.84, 110.46, 
110.56, 110.46, 110.05, 109.6, 109.31, 109.31, 109.25, 109.02, 
108.54, 108.77, 109.02, 109.44, 109.38, 109.53, 109.89, 110.56, 
110.56, 110.72, 111.23, 111.48, 111.58, 111.9, 112.19, 112.06, 
111.96, 111.68, 111.36, 111.42, 112, 112.22, 112.7, 113.15, 114.36, 
114.65, 115.06, 115.86, 116.4, 116.44, 116.88, 118.07, 118.51, 
119.28, 119.79, 119.7, 119.28, 119.66, 120.14, 120.97, 121.13, 
121.55, 121.96, 122.26, 123.79, 124.11, 124.14, 123.37, 123.02, 
122.86, 123.02, 123.11, 123.05, 123.05, 122.83, 123.18, 122.67, 
122.73, 122.86, 122.67, 122.09, 122, 121.23), .Tsp = c(1, 78, 
1), class = "ts")

score 1 · Answer 1 · answered Jul 27 '17 at 21:36

The code

Arima(dowj_ts, order=c(1,1,0),include.drift=T,transform.pars = F)

is fine. You should be able to call forecast on this without a problem.

The reason your drift estimate is different is because Arima uses the method of maximum likelihood. Your sample mean is not the maximum likelihood estimate of this parameter. There is no closed form expression for the MLE estimates of the parameters. They have to be found using an iterative algorithm.

score 1 · Answer 2 · answered Aug 28 '17 at 20:54

Others have suggested why you are getting different answers. My focus is that the drift you are focussing on is not-significant. The arima function that you are using is a flawed approach because there are outliers (unusual values) in you series. A good model is as follows AND here

Visually we can see the three anomalies ...

The estimate of the drift is not statistically significant .0499018 . In my opinion (somwehat biased by relatively speaking long experience) is that the difference between this correct estimate of drift and the ones you posted are due to you using three errant values who should be normalized as they do not reflect the underlying structure and consequently need to me set aside or adjusted.

Difference time series and then minus the mean of the differenced series within Arima

2 Answers2