Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

BackgroundShort-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. MethodsWe used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported from a standardised source over the next one to four weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models past predictive performance. ResultsOver 52 weeks we collected and combined up to 28 forecast models for 32 countries. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 84% of participating models forecasts of incident cases (with a total N=862), and 92% of participating models forecasts of deaths (N=746). Across a one to four week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over four weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. ConclusionsOur results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than two weeks. Code and data availabilityAll data and code are publicly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.

Model-based and model-free characterization of epidemic outbreaks

Here we provide detailed background information for our work on Bayesian inference of change-points in the spread of SARS-CoV-2 and the effectiveness of non-pharmaceutical interventions (Dehning et al., Science, 2020). We outline the general background of Bayesian inference and of SIR-like models. We explain the assumptions that underlie model-based estimates of the reproduction number and compare them to the assumptions that underlie model-free estimates, such as used in the Robert-Koch Institute situation reports. We highlight effects that originate from the two estimation approaches, and how they may cause differences in the inferred reproduction number. Furthermore, we explore the challenges that originate from data availability - such as publication delays and inconsistent testing - and explain their impact on the time-course of inferred case numbers. Along with alternative data sources, this allowed us to cross-check and verify our previous results.

Inferring COVID-19 spreading rates and potential change points for case number forecasts

As COVID-19 is rapidly spreading across the globe, short-term modeling forecasts provide time-critical information for decisions on containment and mitigation strategies. A main challenge for short-term forecasts is the assessment of key epidemiological parameters and how they change when first interventions show an effect. By combining an established epidemiological model with Bayesian inference, we analyze the time dependence of the effective growth rate of new infections. Focusing on the COVID-19 spread in Germany, we detect change points in the effective growth rate that correlate well with the times of publicly announced interventions. Thereby, we can quantify the effect of interventions, and we can incorporate the corresponding change points into forecasts of future scenarios and case numbers. Our code is freely available and can be readily adapted to any country or region. IntroductionWhen faced with the outbreak of a novel epidemic like COVID-19, rapid response measures are required by individuals as well as by society as a whole to mitigate the spread of the virus. During this initial, time-critical period, neither the central epidemiological parameters, nor the effectiveness of interventions like cancellation of public events, school closings, and social distancing are known. RationaleAs one of the key epidemiological parameters, we infer the spreading rate{lambda} from confirmed COVID-19 case numbers at the example of Germany by combining Bayesian inference based on Markov-Chain Monte-Carlo sampling with a class of SIR (Susceptible-Infected-Recovered) compartmental models from epidemiology. Our analysis characterizes the temporal change of the spreading rate and, importantly, allows us to identify potential change points and to provide short-term forecast scenarios based on various degrees of social distancing. A detailed description is provided in the accompanying paper, and the models, inference, and predictions are available on github. While we apply it to Germany, our approach can be readily adapted to other countries or regions. ResultsIn Germany, interventions to contain the outbreak were implemented in three steps over three weeks: Around March 9, large public events like soccer matches were cancelled. On March 16, schools and childcare facilities as well as many non-essential stores were closed. One week later, on March 23, a far-reaching contact ban ("Kontaktsperre"), which included the prohibition of even small public gatherings as well as the further closing of restaurants and non-essential stores, was imposed by the government authorities. From the observed case numbers of COVID-19, we can quantify the impact of these measures on the disease spread (Fig. 0). Based on our analysis, which includes data until April 21, we have evidence of three change points: the first changed the spreading rate from{lambda} 0 = 0.43 (95 % credible interval (CI: [0.35, 0.51])) to{lambda} 1 = 0.25 (CI: [0.20, 0.30]), and occurred around March 6 (CI: March 2 to March 9); the second change point resulted in{lambda} 2 = 0.15 (CI: [0.12, 0.20]), and occurred around March 15 (CI: March 13 to March 17). Both changes in{lambda} slowed the spread of the virus, but still implied exponential growth (Fig. 0, red and orange traces). To contain the disease spread, and turn from exponential growth to a decline of new cases, a further decrease in{lambda} was necessary. Our analysis shows that this transition has been reached by the third change point that resulted in{lambda} 3 = 0.09 (CI: [0.06, 0.12]) around March 23 (CI: March 20 to March 25). O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=159 SRC="FIGDIR/small/20050922v3_fig0.gif" ALT="Figure 0"> View larger version (39K): org.highwire.dtl.DTLVardef@1176ccorg.highwire.dtl.DTLVardef@8e7739org.highwire.dtl.DTLVardef@13549baorg.highwire.dtl.DTLVardef@17b5d36_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOFig. 0.C_FLOATNO Bayesian analysis of the German COVID-19 data (blue diamonds) reveals three change points that match the timing of publicly announced interventions. A: The inferred effective growth rate (difference between spreading and recovery rate,{lambda} * ={lambda} - ) for an SIR model with weekly reporting modulation and reporting delay that includes scenarios with one, two or three change points (red, orange, green; fitted to case reports until March 25, April 1 and April 9, respectively). The timing of the inferred change points in growth rate is consistent with the timing of German governmental interventions (depicted as *, **, and * * *). B: Comparing inferred models with the actual new reported cases per day reveals the effectiveness of governmental interventions. After the first two interventions, the number of new cases still grew exponentially (red, orange); only after the third intervention did the number of new cases start to saturate (green) or even to decline (gray, data until April 21). This illustrates that the future development strongly depends on our distancing behavior. Note the delay between a change point and the observation of changes in the number of new cases of almost two weeks a combination of reporting delay and a minimal period of evidence accumulation. C_FIG With this third change point,{lambda} transitioned below the critical value where the spreading rate{lambda} balances the recovery rate , i.e. the effective growth rate{lambda} * ={lambda} - {approx} 0 (Fig. 0, gray traces). Importantly,{lambda} * = 0 presents the watershed between exponential growth or decay. Given the delay of approximately two weeks between an intervention and first inference of the induced changes in{lambda} *, future interventions such as lifting restrictions warrant careful consideration. Our detailed analysis shows that, in the current phase, reliable short- and long-term forecasts are very difficult as they critically hinge on how the epidemiological parameters change in response to interventions: In Fig. 0 already the three example scenarios quickly diverge from each other, and consequently span a considerable range of future case numbers. Thus, any uncertainty on the magnitude of our social distancing in the past two weeks can have a major impact on the case numbers in the next two weeks. Beyond two weeks, the case numbers depend on our future behavior, for which we have to make explicit assumptions. In the main paper we illustrate how the precise magnitude and timing of potential change points impact the forecast of case numbers (Fig. 2). O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=93 SRC="FIGDIR/small/20050922v3_fig2.gif" ALT="Figure 2"> View larger version (28K): org.highwire.dtl.DTLVardef@24b42corg.highwire.dtl.DTLVardef@1b0dcc8org.highwire.dtl.DTLVardef@6ef54aorg.highwire.dtl.DTLVardef@a9f26c_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOFig. 2.C_FLOATNO The timing and effectiveness of interventions strongly impact future COVID-19 cases. A: We assume three different scenarios for interventions starting on March 16: (I, red) no social distancing, (II, orange) mild social distancing, or (III, green) strict social distancing. B: Delaying the restrictions has a major impact on case numbers: strict restrictions starting on March 16 (green), five days later (magenta) or five days earlier (gray). C: Comparison of the time span over which interventions ramp up to full effect. For all ramps that are centered around the same day, the resulting case numbers are fairly similar. However, a sudden change of the spreading rate can cause a temporary decrease of daily new cases, although{lambda} > at all times (brown). C_FIG ConclusionsWe developed a Bayesian framework to infer central epidemiological parameters and the timing and magnitude of intervention effects. Thereby, the efficiency of political and individual intervention measures for social distancing and containment can be assessed in a timely manner. We find evidence for a successive decrease of the spreading rate in Germany around March 6 and around March 15, which significantly reduced the magnitude of exponential growth, but was not sufficient to turn growth into decay. Our analysis also shows that a further decrease of the spreading rate occurred around March 23, turning exponential growth into decay. Future interventions and lifting of restrictions can be modeled as additional change points, enabling short-term forecasts for case numbers. In general, our analysis code may help to infer the efficiency of measures taken in other countries and inform policy makers about tightening, loosening and selecting appropriate rules for containment.