Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations.

Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1-4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models' predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models' forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models' past predictive performance. Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models' forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models' forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models' forecasts of deaths (N=763 predictions from 20 models). Across a 1-4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks. Funding: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021.

BACKGROUND: During the COVID-19 pandemic there has been a strong interest in forecasts of the short-term development of epidemiological indicators to inform decision makers. In this study we evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland for the period from January through April 2021. METHODS: We evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland. These were issued by 15 different forecasting models, run by independent research teams. Moreover, we study the performance of combined ensemble forecasts. Evaluation of probabilistic forecasts is based on proper scoring rules, along with interval coverage proportions to assess calibration. The presented work is part of a pre-registered evaluation study. RESULTS: We find that many, though not all, models outperform a simple baseline model up to four weeks ahead for the considered targets. Ensemble methods show very good relative performance. The addressed time period is characterized by rather stable non-pharmaceutical interventions in both countries, making short-term predictions more straightforward than in previous periods. However, major trend changes in reported cases, like the rebound in cases due to the rise of the B.1.1.7 (Alpha) variant in March 2021, prove challenging to predict. CONCLUSIONS: Multi-model approaches can help to improve the performance of epidemiological forecasts. However, while death numbers can be predicted with some success based on current case and hospitalization data, predictability of case numbers remains low beyond quite short time horizons. Additional data sources including sequencing and mobility data, which were not extensively used in the present study, may help to improve performance.

We compare forecasts of weekly case and death numbers for COVID-19 in Germany and Poland based on 15 different modelling approaches. These cover the period from January to April 2021 and address numbers of cases and deaths one and two weeks into the future, along with the respective uncertainties. We find that combining different forecasts into one forecast can enable better predictions. However, case numbers over longer periods were challenging to predict. Additional data sources, such as information about different versions of the SARS-CoV-2 virus present in the population, might improve forecasts in the future.

Modeling the spread of COVID-19 in Germany: Early assessment and possible scenarios.

The novel coronavirus (SARS-CoV-2), identified in China at the end of December 2019 and causing the disease COVID-19, has meanwhile led to outbreaks all over the globe with about 2.2 million confirmed cases and more than 150,000 deaths as of April 17, 2020. In this work, mathematical models are used to reproduce data of the early evolution of the COVID-19 outbreak in Germany, taking into account the effect of actual and hypothetical non-pharmaceutical interventions. Systems of differential equations of SEIR type are extended to account for undetected infections, stages of infection, and age groups. The models are calibrated on data until April 5. Data from April 6 to 14 are used for model validation. We simulate different possible strategies for the mitigation of the current outbreak, slowing down the spread of the virus and thus reducing the peak in daily diagnosed cases, the demand for hospitalization or intensive care units admissions, and eventually the number of fatalities. Our results suggest that a partial (and gradual) lifting of introduced control measures could soon be possible if accompanied by further increased testing activity, strict isolation of detected cases, and reduced contact to risk groups.

Folding of Top7 in unbiased all-atom Monte Carlo simulations.

For computational studies of protein folding, proteins with both helical and ß-sheet secondary structure elements are very challenging, as they expose subtle biases of the physical models. Here, we present reproducible folding of a 92 residue α/ß protein (residues 3-94 of Top7, PDB ID: 1QYS) in computer simulations starting from random initial conformations using a transferable physical model which has been previously shown to describe the folding and thermodynamic properties of about 20 other smaller proteins of different folds. Top7 is a de novo designed protein with two α-helices and a five stranded ß-sheet. Experimentally, it is known to be unusually stable for its size, and its folding transition distinctly deviates from the two-state behavior commonly seen in natural single domain proteins. In our all-atom implicit solvent parallel tempering Monte Carlo simulations, Top7 shows a rapid transition to a group of states with high native-like secondary structure, and a much slower subsequent transition to the native state with a root mean square deviation of about 3.5 Å from the experimentally determined structure. Consistent with experiments, we find Top7 to be thermally extremely stable, although the simulations also find a large number of very stable non-native states with high native-like secondary structure.

Free-energy-driven folding and thermodynamics of the 67-residue protein GS-alpha3W--a large-scale Monte Carlo study.

Utilizing the computational power of a few thousand processors on a BlueGene/P, we have explored the folding mechanism of the 67-residue protein GS-alpha(3)W. Results from our large-scale simulation indicate a diffusion-collision mechanism for folding. However, the lower-than-expected frequency of native-like configurations at physiological temperatures indicates shortcomings of our energy function. Our results suggest that computational studies of large proteins call for redevelopment and reparametrization of force fields that in turn require extensive simulations only possible with the newly available supercomputers with computing powers reaching the petaflop range.

Simulation of Top7-CFr: a transient helix extension guides folding.

Protein structures often feature beta-sheets in which adjacent beta-strands have large sequence separation. How the folding process orchestrates the formation and correct arrangement of these strands is not comprehensively understood. Particularly challenging are proteins in which beta-strands at the N and C termini are neighbors in a beta-sheet. The N-terminal beta-strand is synthesized early on, but it can not bind to the C terminus before the chain is fully synthesized. During this time, there is a danger that the beta-strand at the N terminus interacts with nearby molecules, leading to potentially harmful aggregates of incompletely folded proteins. Simulations of the C-terminal fragment of Top7 show that this risk of misfolding and aggregation can be avoided by a "caching" mechanism that relies on the "chameleon" behavior of certain segments.

Folding proteins by first-passage-times-optimized replica exchange.

Replica exchange simulations have become the method of choice in computational protein science, but they still often do not allow an efficient sampling of low-energy protein configurations. Here, we reconstruct replica flow in the temperature ladder from first passage times and use it for temperature optimization, thereby maximizing sampling. The method is applied in simulations of folding thermodynamics for a number of proteins starting from the pentapeptide Met-enkephalin, through the 36-residue HP-36, up to the 67-residue protein GS-alpha3W.

Aggregation of beta-amyloid fragments.

The authors study the folding and aggregation of six chains of the beta-amyloid fragment 16-22 using Monte Carlo simulations. While the isolated fragment prefers a helical form at room temperature, in the system of six interacting fragments one observes both parallel and antiparallel beta sheets below a crossover temperature T(x) approximately equal to 420 K. The antiparallel sheets have lower energy and are therefore more stable. Above the nucleation temperature the aggregate quickly dissolves into widely separated, weakly interacting chains.

Direct visualization of dislocation dynamics in grain-boundary scars.

Mesoscale objects with unusual structural features may serve as the analogues of atoms in the design of larger-scale materials with novel optical, electronic or mechanical behaviour. In this paper we investigate the structural features and the equilibrium dynamics of micrometre-scale spherical crystals formed by polystyrene particles adsorbed on the surface of a spherical water droplet. The ground state of sufficiently large crystals possesses finite-length grain boundaries (scars). We determine the elastic response of the crystal by measuring single-particle diffusion, and quantify the fluctuations of individual dislocations about their equilibrium positions within a scar by determining the dislocation spring constants. We observe rapid dislocation glide with fluctuations over the barriers separating one local Peierls minimum from the next and rather weak binding of dislocations to their associated scars. The long-distance (renormalized) dislocation diffusion glide constant is extracted directly from the experimental data and is found to be moderately faster than single-particle diffusion. We are also able to determine the parameters of the Peierls potential induced by the underlying crystalline lattice.