31

I use the auto.arima() function in the forecast package to fit ARMAX models with a variety of covariates. However, I often have a large number of variables to select from and usually end up with a final model that works with a subset of them. I don't like ad-hoc techniques for variable selection because I am human and subject to bias, but cross-validating time series is hard, so I haven't found a good way to automatically try different subsets of my available variables, and am stuck tuning my models using my own best judgement.

When I fit glm models, I can use the elastic net or the lasso for regularization and variable selection, via the glmnet package. Is there a existing toolkit in R for using the elastic net on ARMAX models, or am I going to have to roll my own? Is this even a good idea?

edit: Would it make sense to manually calculate the AR and MA terms (say up to AR5 and MA5) and the use glmnet to fit the model?

edit 2: It seems that the FitAR package gets me part, but not all, of the way there.

Community
  • 1
Zach
  • 22,308
  • 18
  • 114
  • 158
  • 2
    I'd be curious to know if you ever managed to do something about this problem. I had it marked for any update - so just touching base here. – Vishal Belsare Aug 16 '12 at 20:54
  • Never manged to solve this. I thought about writing my own implementation but didn't get around to it. Maybe you could manually calculate the AR and MA terms and then apply glmnet to that data? – Zach Aug 16 '12 at 23:02
  • 1
    Hi Zach, I wondered if you have taken this any further or whether it is still parked. I have a need for such an algorithm to forecast a large number of series. – Matt Weller Nov 28 '14 at 17:11
  • 3
    @MattWeller I talked a bit with Rob Hyndman, the author of the excellent `forecast` package for R. He said it would be difficult with the full ARIMA, because you'd have to wrap the lasso around the nonlinear ARIMA optimizer. One partial solution would be to fit an AR model using `glmnet` with lagged variables. As far as I know, no one's done this with a full ARIMA model yet. – Zach Nov 28 '14 at 21:28

2 Answers2

8

This is not a solution but some reflections on the possibilities and difficulties that I know of.

Whenever it is possible to specify a time-series model as $$Y_{t+1} = \mathbf{x}_t \beta + \epsilon_{t+1}$$ with $\mathbf{x}_t$ computable from covariates and time-lagged observations, it is also possible to compute the least-squares elastic net penalized estimator of $\beta$ using glmnet in R. It requires that you write code to compute $\mathbf{x}_t$ to form the model matrix that is to be specified in glmnet. This works for AR-models but not directly for ARMA-models, say. Moreover, the cross-validation procedures of glmnet are not sensible per se for time-series data.

For more general models $$Y_{t+1} = f(\mathbf{x}_t, \beta) + \epsilon_{t+1}$$ an implementation of an algorithm for computing the non-linear least-squares elastic net penalized estimator of $\beta$ is needed. To the best of my knowledge there is no such implementation in R. I am currently writing an implementation to solve the case where
$$Y_{t+1} = \mathbf{x}_t g(\beta) + \epsilon_{t+1}$$ the point being that it is paramount for model selection that the lasso penalization is on $\beta$ and not $g(\beta)$. If I recall the ARIMA-parametrization correctly it also takes this form $-$ but I cannot offer any code at the moment. It is (will be) based on A coordinate gradient descent method for nonsmooth separable minimization.

Another issue is the selection of the amount of penalization (the tuning parameters). It will generally require a form of cross-validation for time-series, but I hope to be able to work out some less computationally demanding methods for specific models.

NRH
  • 16,580
  • 56
  • 68
  • The caret package for R has a good implementation of time series cross validation that could be used to select the parameters for glmnet. Thanks for the answer. – Zach Feb 18 '13 at 15:53
3

I was challenged by a client to solve this problem in an automatic i.e. turnkey way. I implemented an approach that for each pair ( i.e. y and a candidate x ) , prewhiten , compute cross-correlations of the pre-whitened series, identify the PDL ( OR ADL AUTOREGRESSIVE DISTRIBUTED LAG MODEL including any DEAD TIME ) while incorporating Intervention Detection to yield robust estimates, develop a "measure" for this structure. After conducting this for ALL candidate regressors, rank them by the "measure" and then select the top K regressors based upon the "measure". This is sometimes referred to as Linear Filtering. We successfully incorporated this heuristic into our commercially available time series package. You should be able to "ROLL YOUR OWN" if you have sufficient time and statistical programming skills and/or available modules to implement some of these tasks.

IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • I specifically want to incorporate the lasso or elastic net into an ARIMA model. – Zach Oct 07 '11 at 15:37
  • 3
    I'm not really looking for a linear filter (though this is useful). I specifically want a way to incorporate the lasso, ridge regression, or elastic net into an ARIMAX model. – Zach Jan 12 '12 at 16:11
  • How do you propose to detect Level Shifts or Local Time trends to incorporate into your ARIMAX model ? Hpw do propose to test for the constancy of model coefficients over time ? How do you propose to deal with non-constant error variance where the error variance changed determistically at some unknown point in time. – IrishStat Feb 14 '13 at 21:43
  • 2
    At the moment I'm not interested in level shifts, local time trends, coefficient shifts, or non-constant error variance. I'm very specifically interested in applying lasso, ridge, or elastic net regularization to ARIMAX models. – Zach Feb 15 '13 at 14:37
  • :Zach OK got it ! You, at this time, are not concerned with tests of sufficiency or necessity of your coefficients. – IrishStat Feb 15 '13 at 19:31
  • 2
    If you can incorporate lasso, ridge, or elastic net regularization into the above methodology and post some example code in an open source programming language, I'd be happy to award you the bounty. – Zach Feb 16 '13 at 17:09
  • In reference to detecting level shifts and local time trends, can we just add certain intervention terms into the glmnet model, and let elastic net decide if the intervention terms are significant? – Frank Jul 05 '19 at 15:28
  • 1
    @frank you are trying to do identification via estimation (step-down) and that doesn't work because there are toooooo many combinations of possible pulses, level shifts , seasonal pulses and time trends.,. If these structures are needed but ignored , parameter estimation is biased and parametric tests of significance go out the window. – IrishStat Jul 05 '19 at 19:32