RESUMO
Near-term probabilistic forecasts for infectious diseases such as COVID-19 and influenza play an important role in public health communication and policymaking. From 2013-2019, the FluSight challenge run by the Centers for Disease Control and Prevention invited researchers to develop and submit forecasts using influenza-like illness (ILI) as a measure of influenza burden. Here we examine how several statistical models and an autoregressive neural network model perform for forecasting ILI during the COVID-19 pandemic, where historical patterns of ILI were highly disrupted. We find that the autoregressive neural network model which forecasted ILI well pre-COVID still performs well for some locations and forecast horizons, but its performance is highly variable, and performs poorly in many cases. We found that a simple exponential smoothing statistical model is in the top half of ranked models we evaluated nearly 75% of the time. Our results suggest that even simple statistical models may perform as well as or better than more complex machine learning models for forecasting ILI during the COVID-19 pandemic. We also created an ensemble model from the limited set of time series forecast models we created here. The limited ensemble model was rarely the best or the worst performing model compared to the rest of the models assessed, confirming previous observations from other infectious disease forecasting efforts on the less variable and generally favorable performance of ensemble forecasts. Our results support previous findings that no single modeling approach outperforms all other models across all locations, time points, and forecast horizons, and that ensemble forecasting consortia such as the COVID-19 Forecast Hub and FluSight continue to serve valuable roles in collecting, aggregating, and ensembling forecasts using fundamentally disparate modeling strategies.
RESUMO
Infectious disease forecasting has been a useful tool for public health planning and messaging during the COVID-19 pandemic. In partnership with the CDC, the organizers of the COVID-19 Forecast Hub have created a mechanism for forecasters from academia, industry, and government organizations to submit weekly near-term predictions of COVID-19 targets in the United States. Here we describe our efforts to participate in the COVID-19 Forecast Hub through the Forecasting COVID-19 in the United States (FOCUS) project. The effort led to more than three months of weekly submissions and development of an automated pipeline to generate forecasts. The models used in FOCUS yielded forecasts that ranked relatively well in terms of precision and accuracy.
RESUMO
Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multi-model ensemble forecast that combined predictions from dozens of different research groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naive baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-week horizon 3-5 times larger than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. Significance StatementThis paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the US. Results show high variation in accuracy between and within stand-alone models, and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public health action.
