Challenges of COVID-19 Case Forecasting in the US, 2020-2021.

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

A data-driven semi-parametric model of SARS-CoV-2 transmission in the United States.

To support decision-making and policy for managing epidemics of emerging pathogens, we present a model for inference and scenario analysis of SARS-CoV-2 transmission in the USA. The stochastic SEIR-type model includes compartments for latent, asymptomatic, detected and undetected symptomatic individuals, and hospitalized cases, and features realistic interval distributions for presymptomatic and symptomatic periods, time varying rates of case detection, diagnosis, and mortality. The model accounts for the effects on transmission of human mobility using anonymized mobility data collected from cellular devices, and of difficult to quantify environmental and behavioral factors using a latent process. The baseline transmission rate is the product of a human mobility metric obtained from data and this fitted latent process. We fit the model to incident case and death reports for each state in the USA and Washington D.C., using likelihood Maximization by Iterated particle Filtering (MIF). Observations (daily case and death reports) are modeled as arising from a negative binomial reporting process. We estimate time-varying transmission rate, parameters of a sigmoidal time-varying fraction of hospitalized cases that result in death, extra-demographic process noise, two dispersion parameters of the observation process, and the initial sizes of the latent, asymptomatic, and symptomatic classes. In a retrospective analysis covering March-December 2020, we show how mobility and transmission strength became decoupled across two distinct phases of the pandemic. The decoupling demonstrates the need for flexible, semi-parametric approaches for modeling infectious disease dynamics in real-time.

The statistics of epidemic transitions.

Emerging and re-emerging pathogens exhibit very complex dynamics, are hard to model and difficult to predict. Their dynamics might appear intractable. However, new statistical approaches-rooted in dynamical systems and the theory of stochastic processes-have yielded insight into the dynamics of emerging and re-emerging pathogens. We argue that these approaches may lead to new methods for predicting epidemics. This perspective views pathogen emergence and re-emergence as a "critical transition," and uses the concept of noisy dynamic bifurcation to understand the relationship between the system observables and the distance to this transition. Because the system dynamics exhibit characteristic fluctuations in response to perturbations for a system in the vicinity of a critical point, we propose this information may be harnessed to develop early warning signals. Specifically, the motion of perturbations slows as the system approaches the transition.

Anticipating epidemic transitions with imperfect data.

Epidemic transitions are an important feature of infectious disease systems. As the transmissibility of a pathogen increases, the dynamics of disease spread shifts from limited stuttering chains of transmission to potentially large scale outbreaks. One proposed method to anticipate this transition are early-warning signals (EWS), summary statistics which undergo characteristic changes as the transition is approached. Although theoretically predicted, their mathematical basis does not take into account the nature of epidemiological data, which are typically aggregated into periodic case reports and subject to reporting error. The viability of EWS for epidemic transitions therefore remains uncertain. Here we demonstrate that most EWS can predict emergence even when calculated from imperfect data. We quantify performance using the area under the curve (AUC) statistic, a measure of how well an EWS distinguishes between numerical simulations of an emerging disease and one which is stationary. Values of the AUC statistic are compared across a range of different reporting scenarios. We find that different EWS respond to imperfect data differently. The mean, variance and first differenced variance all perform well unless reporting error is highly overdispersed. The autocorrelation, autocovariance and decay time perform well provided that the aggregation period of the data is larger than the serial interval and reporting error is not highly overdispersed. The coefficient of variation, skewness and kurtosis are found to be unreliable indicators of emergence. Overall, we find that seven of ten EWS considered perform well for most realistic reporting scenarios. We conclude that imperfect epidemiological data is not a barrier to using EWS for many potentially emerging diseases.

Forecasting infectious disease emergence subject to seasonal forcing.

BACKGROUND: Despite high vaccination coverage, many childhood infections pose a growing threat to human populations. Accurate disease forecasting would be of tremendous value to public health. Forecasting disease emergence using early warning signals (EWS) is possible in non-seasonal models of infectious diseases. Here, we assessed whether EWS also anticipate disease emergence in seasonal models. METHODS: We simulated the dynamics of an immunizing infectious pathogen approaching the tipping point to disease endemicity. To explore the effect of seasonality on the reliability of early warning statistics, we varied the amplitude of fluctuations around the average transmission. We proposed and analyzed two new early warning signals based on the wavelet spectrum. We measured the reliability of the early warning signals depending on the strength of their trend preceding the tipping point and then calculated the Area Under the Curve (AUC) statistic. RESULTS: Early warning signals were reliable when disease transmission was subject to seasonal forcing. Wavelet-based early warning signals were as reliable as other conventional early warning signals. We found that removing seasonal trends, prior to analysis, did not improve early warning statistics uniformly. CONCLUSIONS: Early warning signals anticipate the onset of critical transitions for infectious diseases which are subject to seasonal forcing. Wavelet-based early warning statistics can also be used to forecast infectious disease.

A semi-parametric, state-space compartmental model with time-dependent parameters for forecasting COVID-19 cases, hospitalizations and deaths.

Short-term forecasts of the dynamics of coronavirus disease 2019 (COVID-19) in the period up to its decline following mass vaccination was a task that received much attention but proved difficult to do with high accuracy. However, the availability of standardized forecasts and versioned datasets from this period allows for continued work in this area. Here, we introduce the Gaussian infection state space with time dependence (GISST) forecasting model. We evaluate its performance in one to four weeks ahead forecasts of COVID-19 cases, hospital admissions and deaths in the state of California made with official reports of COVID-19, Google's mobility reports and vaccination data available each week. Evaluation of these forecasts with a weighted interval score shows them to consistently outperform a naive baseline forecast and often score closer to or better than a high-performing ensemble forecaster. The GISST model also provides parameter estimates for a compartmental model of COVID-19 dynamics, includes a regression submodel for the transmission rate and allows for parameters to vary over time according to a random walk. GISST provides a novel, balanced combination of computational efficiency, model interpretability and applicability to large multivariate datasets that may prove useful in improving the accuracy of infectious disease forecasts.

Anticipating infectious disease re-emergence and elimination: a test of early warning signals using empirically based models.

Timely forecasts of the emergence, re-emergence and elimination of human infectious diseases allow for proactive, rather than reactive, decisions that save lives. Recent theory suggests that a generic feature of dynamical systems approaching a tipping point-early warning signals (EWS) due to critical slowing down (CSD)-can anticipate disease emergence and elimination. Empirical studies documenting CSD in observed disease dynamics are scarce, but such demonstration of concept is essential to the further development of model-independent outbreak detection systems. Here, we use fitted, mechanistic models of measles transmission in four cities in Niger to detect CSD through statistical EWS. We find that several EWS accurately anticipate measles re-emergence and elimination, suggesting that CSD should be detectable before disease transmission systems cross key tipping points. These findings support the idea that statistical signals based on CSD, coupled with decision-support algorithms and expert judgement, could provide the basis for early warning systems of disease outbreaks.

Does mutational robustness inhibit extinction by lethal mutagenesis in viral populations?

Lethal mutagenesis is a promising new antiviral therapy that kills a virus by raising its mutation rate. One potential shortcoming of lethal mutagenesis is that viruses may resist the treatment by evolving genomes with increased robustness to mutations. Here, we investigate to what extent mutational robustness can inhibit extinction by lethal mutagenesis in viruses, using both simple toy models and more biophysically realistic models based on RNA secondary-structure folding. We show that although the evolution of greater robustness may be promoted by increasing the mutation rate of a viral population, such evolution is unlikely to greatly increase the mutation rate required for certain extinction. Using an analytic multi-type branching process model, we investigate whether the evolution of robustness can be relevant on the time scales on which extinction takes place. We find that the evolution of robustness matters only when initial viral population sizes are small and deleterious mutation rates are only slightly above the level at which extinction can occur. The stochastic calculations are in good agreement with simulations of self-replicating RNA sequences that have to fold into a specific secondary structure to reproduce. We conclude that the evolution of mutational robustness is in most cases unlikely to prevent the extinction of viruses by lethal mutagenesis.

Transient indicators of tipping points in infectious diseases.

The majority of known early warning indicators of critical transitions rely on asymptotic resilience and critical slowing down. In continuous systems, critical slowing down is mathematically described by a decrease in magnitude of the dominant eigenvalue of the Jacobian matrix on the approach to a critical transition. Here, we show that measures of transient dynamics, specifically, reactivity and the maximum of the amplification envelope, also change systematically as a bifurcation is approached in an important class of models for epidemics of infectious diseases. Furthermore, we introduce indicators designed to detect trends in these measures and find that they reliably classify time series of case notifications simulated from stochastic models according to levels of vaccine uptake. Greater attention should be focused on the potential for systems to exhibit transient amplification of perturbations as a critical threshold is approached, and should be considered when searching for generic leading indicators of tipping points. Awareness of this phenomenon will enrich understanding of the dynamics of complex systems on the verge of a critical transition.

Disentangling reporting and disease transmission.

Second-order statistics such as the variance and autocorrelation can be useful indicators of the stability of randomly perturbed systems, in some cases providing early warning of an impending, dramatic change in the system's dynamics. One specific application area of interest is the surveillance of infectious diseases. In the context of disease (re-)emergence, a goal could be to have an indicator that is informative of whether the system is approaching the epidemic threshold, a point beyond which a major outbreak becomes possible. Prior work in this area has provided some proof of this principle but has not analytically treated the effect of imperfect observation on the behavior of indicators. This work provides expected values for several moments of the number of reported cases, where reported cases follow a binomial or negative binomial distribution with a mean based on the number of deaths in a birth-death-immigration process over some reporting interval. The normalized second factorial moment and the decay time of the number of reported cases are two indicators that are insensitive to the reporting probability. Simulation is used to show how this insensitivity could be used to distinguish a trend of increased reporting from a trend of increased transmission. The simulation study also illustrates both the high variance of estimates and the possibility of reducing the variance by averaging over an ensemble of estimates from multiple time series.

Eigenvalues of the covariance matrix as early warning signals for critical transitions in ecological systems.

Many ecological systems are subject critical transitions, which are abrupt changes to contrasting states triggered by small changes in some key component of the system. Temporal early warning signals such as the variance of a time series, and spatial early warning signals such as the spatial correlation in a snapshot of the system's state, have been proposed to forecast critical transitions. However, temporal early warning signals do not take the spatial pattern into account, and past spatial indicators only examine one snapshot at a time. In this study, we propose the use of eigenvalues of the covariance matrix of multiple time series as early warning signals. We first show theoretically why these indicators may increase as the system moves closer to the critical transition. Then, we apply the method to simulated data from several spatial ecological models to demonstrate the method's applicability. This method has the advantage that it takes into account only the fluctuations of the system about its equilibrium, thus eliminating the effects of any change in equilibrium values. The eigenvector associated with the largest eigenvalue of the covariance matrix is helpful for identifying the regions that are most vulnerable to the critical transition.

Estimating the distance to an epidemic threshold.

The epidemic threshold of the susceptible-infected-recovered model is a boundary separating parameters that permit epidemics from those that do not. This threshold corresponds to parameters where the system's equilibrium becomes unstable. Consequently, we use the average rate at which deviations from the equilibrium shrink to define a distance to this threshold. However, the vital dynamics of the host population may occur slowly even when transmission is far from threshold levels. Here, we show analytically how such slow dynamics can prevent estimation of the distance to the threshold from fluctuations in the susceptible population. Although these results are exact only in the limit of long-term observation of a large system, simulations show that they still provide useful insight into systems with a range of population sizes, environmental noise and observation schemes. Having established some guidelines about when estimates are accurate, we then illustrate how multiple distance estimates can be used to estimate the rate of approach to the threshold. The estimation approach is general and may be applicable to zoonotic pathogens such as Middle East respiratory syndrome-related coronavirus (MERS-CoV) as well as vaccine-preventable diseases like measles.

Using heterogeneity in the population structure of U.S. swine farms to compare transmission models for porcine epidemic diarrhoea.

In 2013, U.S. swine producers were confronted with the disruptive emergence of porcine epidemic diarrhoea (PED). Movement of animals among farms is hypothesised to have played a role in the spread of PED among farms. Via this or other mechanisms, the rate of spread may also depend on the geographic density of farms and climate. To evaluate such effects on a large scale, we analyse state-level counts of outbreaks with variables describing the distribution of farm sizes and types, aggregate flows of animals among farms, and an index of climate. Our first main finding is that it is possible for a correlation analysis to be sensitive to transmission model parameters. This finding is based on a global sensitivity analysis of correlations on simulated data that included a biased and noisy observation model based on the available PED data. Our second main finding is that flows are significantly associated with the reports of PED outbreaks. This finding is based on correlations of pairwise relationships and regression modeling of total and weekly outbreak counts. These findings illustrate how variation in population structure may be employed along with observational data to improve understanding of disease spread.

Waiting time to infectious disease emergence.

Emerging diseases must make a transition from stuttering chains of transmission to sustained chains of transmission, but this critical transition need not coincide with the system becoming supercritical. That is, the introduction of infection to a supercritical system results in a significant fraction of the population becoming infected only with a certain probability. Understanding the waiting time to the first major outbreak of an emerging disease is then more complicated than determining when the system becomes supercritical. We treat emergence as a dynamic bifurcation, and use the concept of bifurcation delay to understand the time to emergence after a system becomes supercritical. Specifically, we consider an SIR model with a time-varying transmission term and random infections originating from outside the population. We derive an analytic density function for the delay times and find it to be, in general, in agreement with stochastic simulations. We find the key parameters to be the rate of introduction of infection and the rate of change of the basic reproductive ratio. These findings aid our understanding of real emergence events, and can be incorporated into early-warning systems aimed at forecasting disease risk.

Fitting outbreak models to data from many small norovirus outbreaks.

Infectious disease often occurs in small, independent outbreaks in populations with varying characteristics. Each outbreak by itself may provide too little information for accurate estimation of epidemic model parameters. Here we show that using standard stochastic epidemic models for each outbreak and allowing parameters to vary between outbreaks according to a linear predictor leads to a generalized linear model that accurately estimates parameters from many small and diverse outbreaks. By estimating initial growth rates in addition to transmission rates, we are able to characterize variation in numbers of initially susceptible individuals or contact patterns between outbreaks. With simulation, we find that the estimates are fairly robust to the data being collected at discrete intervals and imputation of about half of all infectious periods. We apply the method by fitting data from 75 norovirus outbreaks in health-care settings. Our baseline regression estimates are 0.0037 transmissions per infective-susceptible day, an initial growth rate of 0.27 transmissions per infective day, and a symptomatic period of 3.35 days. Outbreaks in long-term-care facilities had significantly higher transmission and initial growth rates than outbreaks in hospitals.

Contact heterogeneity and phylodynamics: how contact networks shape parasite evolutionary trees.

The inference of population dynamics from molecular sequence data is becoming an important new method for the surveillance of infectious diseases. Here, we examine how heterogeneity in contact shapes the genealogies of parasitic agents. Using extensive simulations, we find that contact heterogeneity can have a strong effect on how the structure of genealogies reflects epidemiologically relevant quantities such as the proportion of a population that is infected. Comparing the simulations to BEAST reconstructions, we also find that contact heterogeneity can increase the number of sequence isolates required to estimate these quantities over the course of an epidemic. Our results suggest that data about contact-network structure will be required in addition to sequence data for accurate estimation of a parasitic agent's genealogy. We conclude that network models will be important for progress in this area.