Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multi-model ensemble forecast that combined predictions from dozens of different research groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naive baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-week horizon 3-5 times larger than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. Significance StatementThis paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the US. Results show high variation in accuracy between and within stand-alone models, and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public health action.

Optimizing Vaccine Allocation to Combat the COVID-19 Pandemic

The outbreak of COVID-19 has spurred extensive research worldwide to develop a vaccine. However, when a vaccine becomes available, limited production and distribution capabilities will likely lead to another challenge: who to prioritize for vaccination to mitigate the near-end impact of the pandemic? To tackle that question, this paper first expands a state-of-the-art epidemiological model, called DELPHI, to capture the effects of vaccinations and the variability in mortality rates across subpopulations. It then integrates this predictive model into a prescriptive model to optimize vaccine allocation, formulated as a bilinear, non-convex optimization model. To solve it, this paper proposes a coordinate descent algorithm that iterates between optimizing vaccine allocations and simulating the dynamics of the pandemic. We implement the model and algorithm using real-world data in the United States. All else equal, the optimized vaccine allocation prioritizes states with a large number of projected cases and sub-populations facing higher risks (e.g., older ones). Ultimately, the optimized vaccine allocation can reduce the death toll of the pandemic by an estimated 10-25%, or 10,000-20,000 deaths over a three-month period in the United States alone. Highlights- This paper formulates an optimization model for vaccine allocation in response to the COVID-19 pandemic. This model, referred to as DELPHI-V-OPT, integrates a predictive epidemiological model into a prescriptive model to support the allocation of vaccines across geographic regions (e.g., US states) and across risk classes (e.g., age groups). - This paper develops a scalable coordinate descent algorithm to solve the DELPHI-V-OPT model. The proposed algorithm converges effectively and in short computational times. Therefore, the proposed approach can be implemented efficiently, and allows extensive sensitivity analyses for scenario planning and policy analysis. - Computational results demonstrate that optimized vaccine allocation strategies can curb the death toll of the COVID-19 pandemic by an estimated at 10-25%, or 10,000-20,000 deaths over a three-month period in the United States alone. These results highlight the critical role of vaccine allocation to combat the COVID-19 pandemic, in addition to vaccine design and vaccine production.

From predictions to prescriptions: A data-drivenresponse to COVID-19

The COVID-19 pandemic has created unprecedented challenges worldwide. Strained healthcare providers make difficult decisions on patient triage, treatment and care management on a daily basis. Policy makers have imposed social distancing measures to slow the disease, at a steep economic price. We design analytical tools to support these decisions and combat the pandemic. Specifically, we propose a comprehensive data-driven approach to understand the clinical characteristics of COVID-19, predict its mortality, forecast its evolution, and ultimately alleviate its impact. By leveraging cohort-level clinical data, patient-level hospital data, and census-level epidemiological data, we develop an integrated four-step approach, combining descriptive, predictive and prescriptive analytics. First, we aggregate hundreds of clinical studies into the most comprehensive database on COVID-19 to paint a new macroscopic picture of the disease. Second, we build personalized calculators to predict the risk of infection and mortality as a function of demographics, symptoms, comorbidities, and lab values. Third, we develop a novel epidemiological model to project the pandemics spread and inform social distancing policies. Fourth, we propose an optimization model to reallocate ventilators and alleviate shortages. Our results have been used at the clinical level by several hospitals to triage patients, guide care management, plan ICU capacity, and re-distribute ventilators. At the policy level, they are currently supporting safe back-to-work policies at a major institution and equitable vaccine distribution planning at a major pharmaceutical company, and have been integrated into the US Center for Disease Controls pandemic forecast. Significance StatementIn the midst of the COVID-19 pandemic, healthcare providers and policy makers are wrestling with unprecedented challenges. How to treat COVID-19 patients with equipment shortages? How to allocate resources to combat the disease? How to plan for the next stages of the pandemic? We present a data-driven approach to tackle these challenges. We gather comprehensive data from various sources, including clinical studies, electronic medical records, and census reports. We develop algorithms to understand the disease, predict its mortality, forecast its spread, inform social distancing policies, and re-distribute critical equipment. These algorithms provide decision support tools that have been deployed on our publicly available website, and are actively used by hospitals, companies, and policy makers around the globe.

Forecasting COVID-19 and Analyzing the Effect of Government Interventions

One key question in the ongoing COVID-19 pandemic is understanding the impact of government interventions, and when society can return to normal. To this end, we develop DELPHI, a novel epidemiological model that captures the effect of under-detection and government intervention. We applied DELPHI across 167 geographical areas since early April, and recorded 6% and 11% two-week out-of-sample Median Absolute Percentage Error on cases and deaths respectively. Furthermore, DELPHI successfully predicted the large-scale epidemics in many areas months before, including US, UK and Russia. Using our flexible formulation of government intervention in DELPHI, we are able to understand how government interventions impacted the pandemics spread. In particular, DELPHI predicts that in absence of any interventions, over 14 million individuals would have perished by May 17th, while 280,000 current deaths could have been avoided if interventions around the world started one week earlier. Furthermore, we find mass gathering restrictions and school closings on average reduced infection rates the most, at 29.9 {+/-} 6.9% and 17.3 {+/-} 6.7%, respectively. The most stringent policy, stay-at-home, on average reduced the infection rate by 74.4 {+/-} 3.7% from baseline across countries that implemented it. We also illustrate how DELPHI can be extended to provide insights on reopening societies under different policies.