I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. In particular, the unweighted mean of forecasts is typically better than any of the averaged forecasts. So far, in my own work, I have not encountered any exceptions to this except when the data is artificially generated from simple models.
I was, and remain, flabbergasted by this. Why should averaging models based on entirely inconsistent assumptions generate anything but nonsense? Why is the unweighted average of the best model with relatively inferior models usually better than the best? Why do we seem to get most of the benefits of sophisticated ensemble methods from the unweighted mean?
I always thought that the modeling process was intended to find the model that most nearly approximated the underlying reality, imperfectly, of course, but still assuming that there would always be the best model given specified constraints of parsimony, data availability, and the like. To me, the fact that the unweighted mean of a more-or-less arbitrary collection of model types (that experience has taught us are pretty good) does not suggest that the true model is roughly the mean of the constituent models---that would be absurd.
Instead, it suggests that there is no true data-generating process that can be approximated by any standard estimating technique, however sophisticated. The data may be generated as some complex summation or composite of many, many agents or sub-processes, each of which or who embodies a unique complex of causal forces, perhaps including multiple layers of non-linear feedback. Perhaps they are influenced or entrained by common exposure to forces that you as a modeler will never see, like the boss's mood or the ionization level in the air or irrational remnants of historical institutional structures that persist and still affect decisions.
You see this in other ways too. For example, sometimes the theory is utterly unambiguous about which models are to be preferred. It is, for example, entirely clear that most macroeconomic variables modeled by VARs or VECMs should be logged or log-differenced for multiple compelling reasons, both statistical (i.e. to avoid heteroskedasticity, to linearize any trend present), and economic. Except when you actually run such models, the opposite is true. I have no idea why.
My question is this. Has anyone found a way of formalizing the belief that processes we strive to understand have no data-generating process that we can capture in a standard mathematical model? Has anyone attempted to describe the foundations of statistics based on such a formalization -- a statistic in which all models are unavoidably misspecified? If so, does it have any known implications for hypothesis testing, and the sort of test-and-redesign process that constitutes normal workflow for a statistician or a data scientist? Should we be multiplying models earlier in the analysis process? If so, how? Should we be choosing which models to aggregate based on some principle other than the quality of fit with a complexity penalty, or some model comparison test like AIC? As things are designed ultimately to be input to ensembles, should we prioritize models that give different predictions, rather than good predictions? Is there a principled way to make such trade-offs?
And if this is the norm, why isn't it in any of the six widely-used introductory statistics texts I went through when composing this post?
