How well do Covid-19 forecasts work?

Question

I hope this question is in the scope of CrossValidated (I think so because it is in the end about statistical analyses and machine learning, I am not looking for individual opinions but for reviews or surveys, for example based on pre-registered forecasts):

Background

Modelling an epidemic like Covid-19 is difficult for well-known reasons such as behavioral changes or policy interventions (which lead to a time-dependent reproduction rate) or heterogeneity within the population (the reproduction rate differs between subpopulations).

Given these challenges and the relevance of providing accurate forecasts, numerous researchers have tried to provide well-working models to inform the public and policy-makers about the timing and the magnitude of future Covid-19 cases, hospitalizations and deaths.

Question:

After almost two years of the pandemic: is there yet a scientific consensus how well Covid-19 cases forecasts do work (e.g. based on pre-registered forecasts)? Any realizations if certain class of models work better than others?

Some further information:

There is a frequently cited article in the International Journal of Forecasting titled Forecasting for COVID-19 has failed

Epidemic forecasting has a dubious track-record, and its failures became more prominent with COVID-19. Poor data input, wrong modeling assumptions, high sensitivity of estimates, lack of incorporation of epidemiological features, poor past evidence on effects of available interventions, lack of transparency, errors, lack of determinacy, consideration of only one or a few dimensions of the problem at hand, lack of expertise in crucial disciplines, groupthink and bandwagon effects, and selective reporting are some of the causes of these failures.

However, at least one of the authors seems to be seen controversial in the community. I therefore do not know whether this article reflects the mainstream opinion in the field. Especially since there seem to be other articles available which are more positive (at least about short-term forecasts):

Conclusions Ensembles of multi-model forecasts can inform the policy response to the Covid-19 pandemic by assessing future resource needs and expected population impact of morbidity and mortality.

Answer 1

First off, it's hard to assess whether there is a consensus in a rapidly evolving field. Any publication or result from one working group can (and will, and should) be criticized by other researchers. The fact that Ioannidis has made it a hobby to step on people's toes increases this effect (and makes for better theater, too).

I personally take Ioannidis seriously. His 2005 starter toe-stepper was, statistically, not rocket science, but expressed a long-standing major problem in the application of statistics in a great, and admittedly grating, way. I don't know whether the replicability crisis in psychology would have gotten off the ground without him.

However, he is not majorly specialized in forecasting as such. Then again, neither are the authors of the other paper you cite. So little clue here. (Also, the other paper you link tests their model against a no-change null model. That is definitely a useful benchmark, but there are also other simple benchmarks that would have been a little more convincing.)

Note that there is an explicit answer to the Ioannidis paper by Taleb et al. (in press, IJF). Taleb does have more forecasting credentials, and his main point is how hard it is to evaluate forecasting performance in fat tailed situations - like pandemics. Add to that all the factors you note, like feedback loops between forecasts, political interventions, people's behavior, and it all gets very hard indeed.

You may want to take a look at recent International Symposia on Forecasting. At the 2021 conference, there were multiple presentations about forecasting COVID-19 (and others about forecasting in the context of COVID-19). The abstracts sound like this:

Laura Coroneo, "Testing the predictive accuracy of COVID-19 forecasts":

First, at short-horizon (1-week ahead) no forecasting team outperforms a simple time-series benchmark. Second, at longer horizons (3 and 4-weeks ahead) forecasters are more successful and sometimes outperform the benchmark. Third, one of the best performing forecasts is the Ensemble forecast, that combines all available forecasts using uniform weights. In view of these results, collecting a wide range of forecasts and combining them in an ensemble forecast may be a safer approach for health authorities, rather than relying on a small number of forecasts.

That ensembling improves forecasts is one of the few things forecasters agree on.

Christos Emmanouilides, "Forecasting COVID-19: A large-scale comparison of alternative models":

The study provides a substantial amount of evidence that forecasts from non-parametric regression models, estimated in about one-month-long rolling windows, are overall more accurate than point forecasts from other models.

Recall Taleb's point about the difficulty in learning anything from point forecasts in the presence of fat tails.

My personal takeaway is that there is so far little consensus about what kind of model works best, and that the more expert people are, the more they recognize that even posing and answering this question is extremely hard.