Correction: Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines.

[This corrects the article DOI: 10.1371/journal.pmed.1003793.].

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines.

BACKGROUND: The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research. METHODS AND FINDINGS: We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies. CONCLUSIONS: These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

An extensible framework and database of infectious disease for biosurveillance.

Biosurveillance, a relatively young field, has recently increased in importance because of increasing emphasis on global health. Databases and tools describing particular subsets of disease are becoming increasingly common in the field. Here, we present an infectious disease database that includes diseases of biosurveillance relevance and an extensible framework for the easy expansion of the database.

Forecasting the 2013-2014 influenza season using Wikipedia.

Infectious diseases are one of the leading causes of morbidity and mortality around the world; thus, forecasting their impact is crucial for planning an effective response strategy. According to the Centers for Disease Control and Prevention (CDC), seasonal influenza affects 5% to 20% of the U.S. population and causes major economic impacts resulting from hospitalization and absenteeism. Understanding influenza dynamics and forecasting its impact is fundamental for developing prevention and mitigation strategies. We combine modern data assimilation methods with Wikipedia access logs and CDC influenza-like illness (ILI) reports to create a weekly forecast for seasonal influenza. The methods are applied to the 2013-2014 influenza season but are sufficiently general to forecast any disease outbreak, given incidence or case count data. We adjust the initialization and parametrization of a disease model and show that this allows us to determine systematic model bias. In addition, we provide a way to determine where the model diverges from observation and evaluate forecast accuracy. Wikipedia article access logs are shown to be highly correlated with historical ILI records and allow for accurate prediction of ILI data several weeks before it becomes available. The results show that prior to the peak of the flu season, our forecasting method produced 50% and 95% credible intervals for the 2013-2014 ILI observations that contained the actual observations for most weeks in the forecast. However, since our model does not account for re-infection or multiple strains of influenza, the tail of the epidemic is not predicted well after the peak of flu season has passed.

Global disease monitoring and forecasting with Wikipedia.

Infectious disease is a leading threat to public health, economic stability, and other key social structures. Efforts to mitigate these impacts depend on accurate and timely monitoring to measure the risk and progress of disease. Traditional, biologically-focused monitoring techniques are accurate but costly and slow; in response, new techniques based on social internet data, such as social media and search queries, are emerging. These efforts are promising, but important challenges in the areas of scientific peer review, breadth of diseases and countries, and forecasting hamper their operational usefulness. We examine a freely available, open data source for this use: access logs from the online encyclopedia Wikipedia. Using linear models, language as a proxy for location, and a systematic yet simple article selection procedure, we tested 14 location-disease combinations and demonstrate that these data feasibly support an approach that overcomes these challenges. Specifically, our proof-of-concept yields models with r2 up to 0.92, forecasting value up to the 28 days tested, and several pairs of models similar enough to suggest that transferring models from one location to another without re-training is feasible. Based on these preliminary results, we close with a research agenda designed to overcome these challenges and produce a disease monitoring and forecasting system that is significantly more effective, robust, and globally comprehensive than the current state of the art.

Warning Signs of Potential Black Swan Outbreaks in Infectious Disease.

Black swan events in infectious disease describe rare but devastatingly large outbreaks. While experts are skeptical that such events are predictable, it might be possible to identify the warning signs of a black swan event. Specifically, following the initiation of an outbreak, key differentiating features could serve as alerts. Such features could be derived from meta-analyses of large outbreaks for multiple infectious diseases. We hypothesized there may be common features among the pathogen, environment, and host epidemiological triad that characterize an infectious disease black swan event. Using Los Alamos National Laboratory's tool, Analytics for Investigation of Disease Outbreaks, we investigated historical disease outbreak information and anomalous events for several infectious diseases. By studying 32 different infectious diseases and global outbreaks, we observed that in the past 20-30 years, there have been potential black swan events in the majority of infectious diseases analyzed. Importantly, these potential black swan events cannot be attributed to the first introduction of the disease to a susceptible host population. This paper describes our observations and perspectives and illustrates the value of broad analysis of data across the infectious disease realm, providing insights that may not be possible when we focus on singular infectious agents or diseases. Data analytics could be developed to warn health authorities at the beginning of an outbreak of an impending black swan event. Such tools could complement traditional epidemiological modeling to help forecast future large outbreaks and facilitate timely warning and effective, targeted resource allocation for mitigation efforts.

Fast Evaluation of Viral Emerging Risks (FEVER): A computational tool for biosurveillance, diagnostics, and mutation typing of emerging viral pathogens.

Viral pathogens can rapidly evolve, adapt to novel hosts, and evade human immunity. The early detection of emerging viral pathogens through biosurveillance coupled with rapid and accurate diagnostics are required to mitigate global pandemics. However, RNA viruses can mutate rapidly, hampering biosurveillance and diagnostic efforts. Here, we present a novel computational approach called FEVER (Fast Evaluation of Viral Emerging Risks) to design assays that simultaneously accomplish: 1) broad-coverage biosurveillance of an entire group of viruses, 2) accurate diagnosis of an outbreak strain, and 3) mutation typing to detect variants of public health importance. We demonstrate the application of FEVER to generate assays to simultaneously 1) detect sarbecoviruses for biosurveillance; 2) diagnose infections specifically caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2); and 3) perform rapid mutation typing of the D614G SARS-CoV-2 spike variant associated with increased pathogen transmissibility. These FEVER assays had a high in silico recall (predicted positive) up to 99.7% of 525,708 SARS-CoV-2 sequences analyzed and displayed sensitivities and specificities as high as 92.4% and 100% respectively when validated in 100 clinical samples. The D614G SARS-CoV-2 spike mutation PCR test was able to identify the single nucleotide identity at position 23,403 in the viral genome of 96.6% SARS-CoV-2 positive samples without the need for sequencing. This study demonstrates the utility of FEVER to design assays for biosurveillance, diagnostics, and mutation typing to rapidly detect, track, and mitigate future outbreaks and pandemics caused by emerging viruses.

Persistence of Bacillus thuringiensis subsp. kurstaki in Urban Environments following Spraying.

Bacillus thuringiensis subsp. kurstaki is applied extensively in North America to control the gypsy moth, Lymantria dispar. Since B. thuringiensis subsp. kurstaki shares many physical and biological properties with Bacillus anthracis, it is a reasonable surrogate for biodefense studies. A key question in biodefense is how long a biothreat agent will persist in the environment. There is some information in the literature on the persistence of Bacillus anthracis in laboratories and historical testing areas and for Bacillus thuringiensis in agricultural settings, but there is no information on the persistence of Bacillus spp. in the type of environment that would be encountered in a city or on a military installation. Since it is not feasible to release B. anthracis in a developed area, the controlled release of B. thuringiensis subsp. kurstaki for pest control was used to gain insight into the potential persistence of Bacillus spp. in outdoor urban environments. Persistence was evaluated in two locations: Fairfax County, VA, and Seattle, WA. Environmental samples were collected from multiple matrices and evaluated for the presence of viable B. thuringiensis subsp. kurstaki at times ranging from less than 1 day to 4 years after spraying. Real-time PCR and culture were used for analysis. B. thuringiensis subsp. kurstaki was found to persist in urban environments for at least 4 years. It was most frequently detected in soils and less frequently detected in wipes, grass, foliage, and water. The collective results indicate that certain species of Bacillus may persist for years following their dispersal in urban environments.

Improving Detection of Disease Re-emergence Using a Web-Based Tool (RED Alert): Design and Case Analysis Study.

BACKGROUND: Currently, the identification of infectious disease re-emergence is performed without describing specific quantitative criteria that can be used to identify re-emergence events consistently. This practice may lead to ineffective mitigation. In addition, identification of factors contributing to local disease re-emergence and assessment of global disease re-emergence require access to data about disease incidence and a large number of factors at the local level for the entire world. This paper presents Re-emerging Disease Alert (RED Alert), a web-based tool designed to help public health officials detect and understand infectious disease re-emergence. OBJECTIVE: Our objective is to bring together a variety of disease-related data and analytics needed to help public health analysts answer the following 3 primary questions for detecting and understanding disease re-emergence: Is there a potential disease re-emergence at the local (country) level? What are the potential contributing factors for this re-emergence? Is there a potential for global re-emergence? METHODS: We collected and cleaned disease-related data (eg, case counts, vaccination rates, and indicators related to disease transmission) from several data sources including the World Health Organization (WHO), Pan American Health Organization (PAHO), World Bank, and Gideon. We combined these data with machine learning and visual analytics into a tool called RED Alert to detect re-emergence for the following 4 diseases: measles, cholera, dengue, and yellow fever. We evaluated the performance of the machine learning models for re-emergence detection and reviewed the output of the tool through a number of case studies. RESULTS: Our supervised learning models were able to identify 82%-90% of the local re-emergence events, although with 18%-31% (except 46% for dengue) false positives. This is consistent with our goal of identifying all possible re-emergences while allowing some false positives. The review of the web-based tool through case studies showed that local re-emergence detection was possible and that the tool provided actionable information about potential factors contributing to the local disease re-emergence and trends in global disease re-emergence. CONCLUSIONS: To the best of our knowledge, this is the first tool that focuses specifically on disease re-emergence and addresses the important challenges mentioned above.

Simultaneous pathogen detection and antibiotic resistance characterization using SNP-based multiplexed oligonucleotide ligation-PCR (MOL-PCR).

Extensive use of antibiotics in both public health and animal husbandry has resulted in rapid emergence of antibiotic resistance in almost all human pathogens, including biothreat pathogens. Antibiotic resistance has thus become a major concern for both public health and national security. We developed multiplexed assays for rapid, simultaneous pathogen detection and characterization of ciprofloxacin and doxycycline resistance in Bacillus anthracis, Yersinia pestis, and Francisella tularensis. These assays are SNP-based and use Multiplexed Oligonucleotide Ligation-PCR (MOL-PCR). The MOL-PCR assay chemistry and MOLigo probe design process are presented. A web-based tool - MOLigoDesigner (http://MOLigoDesigner.lanl.gov) was developed to facilitate the probe design. All probes were experimentally validated individually and in multiplexed assays, and minimal sets of multiplexed MOLigo probes were identified for simultaneous pathogen detection and antibiotic resistance characterization.

Development of a Supervised Learning Algorithm for Detection of Potential Disease Reemergence: A Proof of Concept.

Infectious disease reemergence is an important yet ambiguous concept that lacks a quantitative definition. Currently, reemergence is identified without specific criteria describing what constitutes a reemergent event. This practice affects reproducible assessments of high-consequence public health events and disease response prioritization. This in turn can lead to misallocation of resources. More important, early recognition of reemergence facilitates effective mitigation. We used a supervised machine learning approach to detect potential disease reemergence. We demonstrate the feasibility of applying a machine learning classifier to identify reemergence events in a systematic way for 4 different infectious diseases. The algorithm is applicable to temporal trends of disease incidence and includes disease-specific features to identify potential reemergence. Through this study, we offer a structured means of identifying potential reemergence using a data-driven approach.

Analytics for Investigation of Disease Outbreaks: Web-Based Analytics Facilitating Situational Awareness in Unfolding Disease Outbreaks.

BACKGROUND: Information from historical infectious disease outbreaks provides real-world data about outbreaks and their impacts on affected populations. These data can be used to develop a picture of an unfolding outbreak in its early stages, when incoming information is sparse and isolated, to identify effective control measures and guide their implementation. OBJECTIVE: This study aimed to develop a publicly accessible Web-based visual analytic called Analytics for the Investigation of Disease Outbreaks (AIDO) that uses historical disease outbreak information for decision support and situational awareness of an unfolding outbreak. METHODS: We developed an algorithm to allow the matching of unfolding outbreak data to a representative library of historical outbreaks. This process provides epidemiological clues that facilitate a user's understanding of an unfolding outbreak and facilitates informed decisions about mitigation actions. Disease-specific properties to build a complete picture of the unfolding event were identified through a data-driven approach. A method of analogs approach was used to develop a short-term forecasting feature in the analytic. The 4 major steps involved in developing this tool were (1) collection of historic outbreak data and preparation of the representative library, (2) development of AIDO algorithms, (3) development of user interface and associated visuals, and (4) verification and validation. RESULTS: The tool currently includes representative historical outbreaks for 39 infectious diseases with over 600 diverse outbreaks. We identified 27 different properties categorized into 3 broad domains (population, location, and disease) that were used to evaluate outbreaks across all diseases for their effect on case count and duration of an outbreak. Statistical analyses revealed disease-specific properties from this set that were included in the disease-specific similarity algorithm. Although there were some similarities across diseases, we found that statistically important properties tend to vary, even between similar diseases. This may be because of our emphasis on including diverse representative outbreak presentations in our libraries. AIDO algorithm evaluations (similarity algorithm and short-term forecasting) were conducted using 4 case studies and we have shown details for the Q fever outbreak in Bilbao, Spain (2014), using data from the early stages of the outbreak. Using data from only the initial 2 weeks, AIDO identified historical outbreaks that were very similar in terms of their epidemiological picture (case count, duration, source of exposure, and urban setting). The short-term forecasting algorithm accurately predicted case count and duration for the unfolding outbreak. CONCLUSIONS: AIDO is a decision support tool that facilitates increased situational awareness during an unfolding outbreak and enables informed decisions on mitigation strategies. AIDO analytics are available to epidemiologists across the globe with access to internet, at no cost. In this study, we presented a new approach to applying historical outbreak data to provide actionable information during the early stages of an unfolding infectious disease outbreak.

Epidemiological Data Challenges: Planning for a More Robust Future Through Data Standards.

Accessible epidemiological data are of great value for emergency preparedness and response, understanding disease progression through a population, and building statistical and mechanistic disease models that enable forecasting. The status quo, however, renders acquiring and using such data difficult in practice. In many cases, a primary way of obtaining epidemiological data is through the internet, but the methods by which the data are presented to the public often differ drastically among institutions. As a result, there is a strong need for better data sharing practices. This paper identifies, in detail and with examples, the three key challenges one encounters when attempting to acquire and use epidemiological data: (1) interfaces, (2) data formatting, and (3) reporting. These challenges are used to provide suggestions and guidance for improvement as these systems evolve in the future. If these suggested data and interface recommendations were adhered to, epidemiological and public health analysis, modeling, and informatics work would be significantly streamlined, which can in turn yield better public health decision-making capabilities.

Corrigendum: An approach to and web-based tool for infectious disease outbreak intervention analysis.

This corrects the article DOI: 10.1038/srep46076.

An approach to and web-based tool for infectious disease outbreak intervention analysis.

Infectious diseases are a leading cause of death globally. Decisions surrounding how to control an infectious disease outbreak currently rely on a subjective process involving surveillance and expert opinion. However, there are many situations where neither may be available. Modeling can fill gaps in the decision making process by using available data to provide quantitative estimates of outbreak trajectories. Effective reduction of the spread of infectious diseases can be achieved through collaboration between the modeling community and public health policy community. However, such collaboration is rare, resulting in a lack of models that meet the needs of the public health community. Here we show a Susceptible-Infectious-Recovered (SIR) model modified to include control measures that allows parameter ranges, rather than parameter point estimates, and includes a web user interface for broad adoption. We apply the model to three diseases, measles, norovirus and influenza, to show the feasibility of its use and describe a research agenda to further promote interactions between decision makers and the modeling community.

Measuring Global Disease with Wikipedia: Success, Failure, and a Research Agenda.

Effective disease monitoring provides a foundation for effective public health systems. This has historically been accomplished with patient contact and bureaucratic aggregation, which tends to be slow and expensive. Recent internet-based approaches promise to be real-time and cheap, with few parameters. However, the question of when and how these approaches work remains open. We addressed this question using Wikipedia access logs and category links. Our experiments, replicable and extensible using our open source code and data, test the effect of semantic article filtering, amount of training data, forecast horizon, and model staleness by comparing across 6 diseases and 4 countries using thousands of individual models. We found that our minimal-configuration, language-agnostic article selection process based on semantic relatedness is effective for improving predictions, and that our approach is relatively insensitive to the amount and age of training data. We also found, in contrast to prior work, very little forecasting value, and we argue that this is consistent with theoretical considerations about the nature of forecasting. These mixed results lead us to propose that the currently observational field of internet-based disease surveillance must pivot to include theoretical models of information flow as well as controlled experiments based on simulations of disease.

Quantitative analysis of the effect of cell type and cellular differentiation on protective antigen binding to human target cells.

We quantitatively measured protective antigen (PA) binding to human cells targeted by anthrax lethal toxin (LT). Affinities were less than 50 nM for all cells, but differentiated cells (macrophages and neutrophils) had significantly increased PA binding and endothelial cells demonstrated the most binding. Combined with the function of such cells, this suggests that PA receptors interact with the extracellular matrix and that differentiation increases the number of PA-specific receptors, which supports previously observed differentiation-induced LT susceptibility. Our results quantifiably confirm that the generality of PA binding will complicate its use as a tumor targeting agent.

Primer-design for multiplexed genotyping.

Single-nucleotide polymorphism (SNP) analysis is a powerful tool for mapping and diagnosing disease-related alleles. Mutation analysis by polymerase-mediated single-base primer extension (minisequencing) can be massively parallelized using DNA microchips or flow cytometry with microspheres as solid support. By adding a unique oligonucleotide tag to the 5' end of the minisequencing primer and attaching the complementary antitag to the array or bead surface, the assay can be 'demultiplexed'. Such high-throughput scoring of SNPs requires a high level of primer multiplexing in order to analyze multiple loci in one assay, thus enabling inexpensive and fast polymorphism scoring. We present a computer program to automate the design process for the assay. Oligonucleotide primers for the reaction are automatically selected by the software, a unique DNA tag/antitag system is generated, and the pairing of primers and DNA tags is automatically done in a way to avoid any crossreactivity. We report results on a 45-plex genotyping assay, indicating that minisequencing can be adapted to be a powerful tool for high-throughput, massively parallel genotyping. The software is available to academic users on request.

Novel Use of Flu Surveillance Data: Evaluating Potential of Sentinel Populations for Early Detection of Influenza Outbreaks.

Influenza causes significant morbidity and mortality each year, with 2-8% of weekly outpatient visits around the United States for influenza-like-illness (ILI) during the peak of the season. Effective use of existing flu surveillance data allows officials to understand and predict current flu outbreaks and can contribute to reductions in influenza morbidity and mortality. Previous work used the 2009-2010 influenza season to investigate the possibility of using existing military and civilian surveillance systems to improve early detection of flu outbreaks. Results suggested that civilian surveillance could help predict outbreak trajectory in local military installations. To further test that hypothesis, we compare pairs of civilian and military outbreaks in seven locations between 2000 and 2013. We find no predictive relationship between outbreak peaks or time series of paired outbreaks. This larger study does not find evidence to support the hypothesis that civilian data can be used as sentinel surveillance for military installations. We additionally investigate the effect of modifying the ILI case definition between the standard Department of Defense definition, a more specific definition proposed in literature, and confirmed Influenza A. We find that case definition heavily impacts results. This study thus highlights the importance of careful selection of case definition, and appropriate consideration of case definition in the interpretation of results.