Choosing a Generative Models for time series data

Question

I'm looking to try and generate economic time series data (GDP, Inflation, Unemployment etc.) with some generative models. I am thinking of using them with LSTMs as well as more AR(p) style models so I would prefer models that generate long strings of data (ie inflation from t = 0 to t = 20 or something) but if that doesn't generate good results, which I'm guessing it might not, I would also be fine with an AR(p) style results (ie output is only t = 0 and t = 1).

I'm considering using both Generative Adversarial Networks (GANs) or Deep Belief Nets (DBNs) to generate my data, but I'm unsure which I should focus my research efforts into. One thing I don't need is deep convolutional layers for image generation that GANs are good at, and a model overfitting the time series is also a problem I want to avoid (economic time series have like maybe 5000 observations accross 50 countries, so the GANs would probably have to be pretty shallow). On the other hand, I have read that Restricted Boltzmann Machines are considered toy models now, (unsure if DBNs are also considered that). Also unsure of performance of DBN with Real Valued Output.

If you have other suggestions on other types of models I could use, definitely let me know. However, I really want as state of the art from a Machine Learning perspective as possible (considering I'm a single person with good machine learning and neural network experience, but not a professor with a lab and 30 years of experience studying generative models). I also understand that economist don't typically generate data to improve model performance, but this is a research project and I want to see if I can improve model performance (and hopefully understanding causality) better with some generated data.

I understand that this question is perhaps subjective, but perhaps if someone could explain the pros and cons of both DBNs and GANs. And how helpful each would be for my problem.

Answer 1

I think using generative model from deep learning will not lead you anywhere with the time series economic data you are considering, and GAN will probably be the worst generative model for your application.

What you are considering is to be able to sample from $p(x_{t+1}|x_{1...t})$, which should be captured by LSTM or AR models you used. The key here is that the probability is a conditional probability, while the generative models are unconditional probability $p(x)$. For example, most generative model takes as input some relatively meaningless latent variable value, so you cannot even make a model that takes $x_{1..t}$ as your input.

What I would suggest is to simply assume an AR(1) type process, and takes $x_t$ as input to a simple multilayer perceptron to predict $x_{t+1}$. This is kind of like a model with complexity in between your AR linear model and LSTM, but the multilayer perceptron can capture more nonlinear dependency than AR linear model and has less variance and is easier to train than LSTM.

You mentioned you failed in using AR and LSTM model. Using LSTM is usually hard with high variance data for economic data and can quickly overfit into some unknown region. If you still want to try it, one suggestion is to assume a normal distribution of all your variables, and uses LSTM to output the mean and variance of model prediction.