ACF and PACF are routinely used for approximate identification of a time series model, e.g. as described here. Say, one takes a look at the plots and guesses that it's something like ARMA(2,0,0)(0,0,1), or ARMA(2,0,0)(1,0,1), or ARMA(2,0,0)(0,0,2). Then he/she fits these three models and checks the updated plots and possibly the values of information criteria.
However, the model pool was clearly suggested by the data, so the amount of overfitting is a lot more than it seems. To make it more explicit, imagine that, instead of eyeballing the plots, I fitted all possible model specifications up to 2nd order and the three models above turned out to have the best AICs in that long list.
Did anyone think of a way to adjust for that or testing out of sample is the only way?
