Correction: Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines.

[This corrects the article DOI: 10.1371/journal.pmed.1003793.].

Warning Signs of Potential Black Swan Outbreaks in Infectious Disease.

Black swan events in infectious disease describe rare but devastatingly large outbreaks. While experts are skeptical that such events are predictable, it might be possible to identify the warning signs of a black swan event. Specifically, following the initiation of an outbreak, key differentiating features could serve as alerts. Such features could be derived from meta-analyses of large outbreaks for multiple infectious diseases. We hypothesized there may be common features among the pathogen, environment, and host epidemiological triad that characterize an infectious disease black swan event. Using Los Alamos National Laboratory's tool, Analytics for Investigation of Disease Outbreaks, we investigated historical disease outbreak information and anomalous events for several infectious diseases. By studying 32 different infectious diseases and global outbreaks, we observed that in the past 20-30 years, there have been potential black swan events in the majority of infectious diseases analyzed. Importantly, these potential black swan events cannot be attributed to the first introduction of the disease to a susceptible host population. This paper describes our observations and perspectives and illustrates the value of broad analysis of data across the infectious disease realm, providing insights that may not be possible when we focus on singular infectious agents or diseases. Data analytics could be developed to warn health authorities at the beginning of an outbreak of an impending black swan event. Such tools could complement traditional epidemiological modeling to help forecast future large outbreaks and facilitate timely warning and effective, targeted resource allocation for mitigation efforts.

Fast Evaluation of Viral Emerging Risks (FEVER): A computational tool for biosurveillance, diagnostics, and mutation typing of emerging viral pathogens.

Viral pathogens can rapidly evolve, adapt to novel hosts, and evade human immunity. The early detection of emerging viral pathogens through biosurveillance coupled with rapid and accurate diagnostics are required to mitigate global pandemics. However, RNA viruses can mutate rapidly, hampering biosurveillance and diagnostic efforts. Here, we present a novel computational approach called FEVER (Fast Evaluation of Viral Emerging Risks) to design assays that simultaneously accomplish: 1) broad-coverage biosurveillance of an entire group of viruses, 2) accurate diagnosis of an outbreak strain, and 3) mutation typing to detect variants of public health importance. We demonstrate the application of FEVER to generate assays to simultaneously 1) detect sarbecoviruses for biosurveillance; 2) diagnose infections specifically caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2); and 3) perform rapid mutation typing of the D614G SARS-CoV-2 spike variant associated with increased pathogen transmissibility. These FEVER assays had a high in silico recall (predicted positive) up to 99.7% of 525,708 SARS-CoV-2 sequences analyzed and displayed sensitivities and specificities as high as 92.4% and 100% respectively when validated in 100 clinical samples. The D614G SARS-CoV-2 spike mutation PCR test was able to identify the single nucleotide identity at position 23,403 in the viral genome of 96.6% SARS-CoV-2 positive samples without the need for sequencing. This study demonstrates the utility of FEVER to design assays for biosurveillance, diagnostics, and mutation typing to rapidly detect, track, and mitigate future outbreaks and pandemics caused by emerging viruses.

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines.

BACKGROUND: The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research. METHODS AND FINDINGS: We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies. CONCLUSIONS: These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

Improving Detection of Disease Re-emergence Using a Web-Based Tool (RED Alert): Design and Case Analysis Study.

BACKGROUND: Currently, the identification of infectious disease re-emergence is performed without describing specific quantitative criteria that can be used to identify re-emergence events consistently. This practice may lead to ineffective mitigation. In addition, identification of factors contributing to local disease re-emergence and assessment of global disease re-emergence require access to data about disease incidence and a large number of factors at the local level for the entire world. This paper presents Re-emerging Disease Alert (RED Alert), a web-based tool designed to help public health officials detect and understand infectious disease re-emergence. OBJECTIVE: Our objective is to bring together a variety of disease-related data and analytics needed to help public health analysts answer the following 3 primary questions for detecting and understanding disease re-emergence: Is there a potential disease re-emergence at the local (country) level? What are the potential contributing factors for this re-emergence? Is there a potential for global re-emergence? METHODS: We collected and cleaned disease-related data (eg, case counts, vaccination rates, and indicators related to disease transmission) from several data sources including the World Health Organization (WHO), Pan American Health Organization (PAHO), World Bank, and Gideon. We combined these data with machine learning and visual analytics into a tool called RED Alert to detect re-emergence for the following 4 diseases: measles, cholera, dengue, and yellow fever. We evaluated the performance of the machine learning models for re-emergence detection and reviewed the output of the tool through a number of case studies. RESULTS: Our supervised learning models were able to identify 82%-90% of the local re-emergence events, although with 18%-31% (except 46% for dengue) false positives. This is consistent with our goal of identifying all possible re-emergences while allowing some false positives. The review of the web-based tool through case studies showed that local re-emergence detection was possible and that the tool provided actionable information about potential factors contributing to the local disease re-emergence and trends in global disease re-emergence. CONCLUSIONS: To the best of our knowledge, this is the first tool that focuses specifically on disease re-emergence and addresses the important challenges mentioned above.

Development of a Supervised Learning Algorithm for Detection of Potential Disease Reemergence: A Proof of Concept.

Infectious disease reemergence is an important yet ambiguous concept that lacks a quantitative definition. Currently, reemergence is identified without specific criteria describing what constitutes a reemergent event. This practice affects reproducible assessments of high-consequence public health events and disease response prioritization. This in turn can lead to misallocation of resources. More important, early recognition of reemergence facilitates effective mitigation. We used a supervised machine learning approach to detect potential disease reemergence. We demonstrate the feasibility of applying a machine learning classifier to identify reemergence events in a systematic way for 4 different infectious diseases. The algorithm is applicable to temporal trends of disease incidence and includes disease-specific features to identify potential reemergence. Through this study, we offer a structured means of identifying potential reemergence using a data-driven approach.

Analytics for Investigation of Disease Outbreaks: Web-Based Analytics Facilitating Situational Awareness in Unfolding Disease Outbreaks.

BACKGROUND: Information from historical infectious disease outbreaks provides real-world data about outbreaks and their impacts on affected populations. These data can be used to develop a picture of an unfolding outbreak in its early stages, when incoming information is sparse and isolated, to identify effective control measures and guide their implementation. OBJECTIVE: This study aimed to develop a publicly accessible Web-based visual analytic called Analytics for the Investigation of Disease Outbreaks (AIDO) that uses historical disease outbreak information for decision support and situational awareness of an unfolding outbreak. METHODS: We developed an algorithm to allow the matching of unfolding outbreak data to a representative library of historical outbreaks. This process provides epidemiological clues that facilitate a user's understanding of an unfolding outbreak and facilitates informed decisions about mitigation actions. Disease-specific properties to build a complete picture of the unfolding event were identified through a data-driven approach. A method of analogs approach was used to develop a short-term forecasting feature in the analytic. The 4 major steps involved in developing this tool were (1) collection of historic outbreak data and preparation of the representative library, (2) development of AIDO algorithms, (3) development of user interface and associated visuals, and (4) verification and validation. RESULTS: The tool currently includes representative historical outbreaks for 39 infectious diseases with over 600 diverse outbreaks. We identified 27 different properties categorized into 3 broad domains (population, location, and disease) that were used to evaluate outbreaks across all diseases for their effect on case count and duration of an outbreak. Statistical analyses revealed disease-specific properties from this set that were included in the disease-specific similarity algorithm. Although there were some similarities across diseases, we found that statistically important properties tend to vary, even between similar diseases. This may be because of our emphasis on including diverse representative outbreak presentations in our libraries. AIDO algorithm evaluations (similarity algorithm and short-term forecasting) were conducted using 4 case studies and we have shown details for the Q fever outbreak in Bilbao, Spain (2014), using data from the early stages of the outbreak. Using data from only the initial 2 weeks, AIDO identified historical outbreaks that were very similar in terms of their epidemiological picture (case count, duration, source of exposure, and urban setting). The short-term forecasting algorithm accurately predicted case count and duration for the unfolding outbreak. CONCLUSIONS: AIDO is a decision support tool that facilitates increased situational awareness during an unfolding outbreak and enables informed decisions on mitigation strategies. AIDO analytics are available to epidemiologists across the globe with access to internet, at no cost. In this study, we presented a new approach to applying historical outbreak data to provide actionable information during the early stages of an unfolding infectious disease outbreak.

Epidemiological Data Challenges: Planning for a More Robust Future Through Data Standards.

Accessible epidemiological data are of great value for emergency preparedness and response, understanding disease progression through a population, and building statistical and mechanistic disease models that enable forecasting. The status quo, however, renders acquiring and using such data difficult in practice. In many cases, a primary way of obtaining epidemiological data is through the internet, but the methods by which the data are presented to the public often differ drastically among institutions. As a result, there is a strong need for better data sharing practices. This paper identifies, in detail and with examples, the three key challenges one encounters when attempting to acquire and use epidemiological data: (1) interfaces, (2) data formatting, and (3) reporting. These challenges are used to provide suggestions and guidance for improvement as these systems evolve in the future. If these suggested data and interface recommendations were adhered to, epidemiological and public health analysis, modeling, and informatics work would be significantly streamlined, which can in turn yield better public health decision-making capabilities.

Measuring Global Disease with Wikipedia: Success, Failure, and a Research Agenda.

Effective disease monitoring provides a foundation for effective public health systems. This has historically been accomplished with patient contact and bureaucratic aggregation, which tends to be slow and expensive. Recent internet-based approaches promise to be real-time and cheap, with few parameters. However, the question of when and how these approaches work remains open. We addressed this question using Wikipedia access logs and category links. Our experiments, replicable and extensible using our open source code and data, test the effect of semantic article filtering, amount of training data, forecast horizon, and model staleness by comparing across 6 diseases and 4 countries using thousands of individual models. We found that our minimal-configuration, language-agnostic article selection process based on semantic relatedness is effective for improving predictions, and that our approach is relatively insensitive to the amount and age of training data. We also found, in contrast to prior work, very little forecasting value, and we argue that this is consistent with theoretical considerations about the nature of forecasting. These mixed results lead us to propose that the currently observational field of internet-based disease surveillance must pivot to include theoretical models of information flow as well as controlled experiments based on simulations of disease.

An extensible framework and database of infectious disease for biosurveillance.

Biosurveillance, a relatively young field, has recently increased in importance because of increasing emphasis on global health. Databases and tools describing particular subsets of disease are becoming increasingly common in the field. Here, we present an infectious disease database that includes diseases of biosurveillance relevance and an extensible framework for the easy expansion of the database.

Corrigendum: An approach to and web-based tool for infectious disease outbreak intervention analysis.

This corrects the article DOI: 10.1038/srep46076.

An approach to and web-based tool for infectious disease outbreak intervention analysis.

Infectious diseases are a leading cause of death globally. Decisions surrounding how to control an infectious disease outbreak currently rely on a subjective process involving surveillance and expert opinion. However, there are many situations where neither may be available. Modeling can fill gaps in the decision making process by using available data to provide quantitative estimates of outbreak trajectories. Effective reduction of the spread of infectious diseases can be achieved through collaboration between the modeling community and public health policy community. However, such collaboration is rare, resulting in a lack of models that meet the needs of the public health community. Here we show a Susceptible-Infectious-Recovered (SIR) model modified to include control measures that allows parameter ranges, rather than parameter point estimates, and includes a web user interface for broad adoption. We apply the model to three diseases, measles, norovirus and influenza, to show the feasibility of its use and describe a research agenda to further promote interactions between decision makers and the modeling community.

Development of 11-Plex MOL-PCR Assay for the Rapid Screening of Samples for Shiga Toxin-Producing Escherichia coli.

Strains of Shiga toxin-producing Escherichia coli (STEC) are a serious threat to the health, with approximately half of the STEC related food-borne illnesses attributable to contaminated beef. We developed an assay that was able to screen samples for several important STEC associated serogroups (O26, O45, O103, O104, O111, O121, O145, O157) and three major virulence factors (eae, stx 1 , stx 2) in a rapid and multiplexed format using the Multiplex oligonucleotide ligation-PCR (MOL-PCR) assay chemistry. This assay detected unique STEC DNA signatures and is meant to be used on samples from various sources related to beef production, providing a multiplex and high-throughput complement to the multiplex PCR assays currently in use. Multiplex oligonucleotide ligation-PCR (MOL-PCR) is a nucleic acid-based assay chemistry that relies on flow cytometry/image cytometry and multiplex microsphere arrays for the detection of nucleic acid-based signatures present in target agents. The STEC MOL-PCR assay provided greater than 90% analytical specificity across all sequence markers designed when tested against panels of DNA samples that represent different STEC serogroups and toxin gene profiles. This paper describes the development of the 11-plex assay and the results of its validation. This highly multiplexed, but more importantly dynamic and adaptable screening assay allows inclusion of additional signatures as they are identified in relation to public health. As the impact of STEC associated illness on public health is explored additional information on classification will be needed on single samples; thus, this assay can serve as the backbone for a complex screening system.

Novel Use of Flu Surveillance Data: Evaluating Potential of Sentinel Populations for Early Detection of Influenza Outbreaks.

Influenza causes significant morbidity and mortality each year, with 2-8% of weekly outpatient visits around the United States for influenza-like-illness (ILI) during the peak of the season. Effective use of existing flu surveillance data allows officials to understand and predict current flu outbreaks and can contribute to reductions in influenza morbidity and mortality. Previous work used the 2009-2010 influenza season to investigate the possibility of using existing military and civilian surveillance systems to improve early detection of flu outbreaks. Results suggested that civilian surveillance could help predict outbreak trajectory in local military installations. To further test that hypothesis, we compare pairs of civilian and military outbreaks in seven locations between 2000 and 2013. We find no predictive relationship between outbreak peaks or time series of paired outbreaks. This larger study does not find evidence to support the hypothesis that civilian data can be used as sentinel surveillance for military installations. We additionally investigate the effect of modifying the ILI case definition between the standard Department of Defense definition, a more specific definition proposed in literature, and confirmed Influenza A. We find that case definition heavily impacts results. This study thus highlights the importance of careful selection of case definition, and appropriate consideration of case definition in the interpretation of results.

Surveillance for Emerging Diseases with Multiplexed Point-of-Care Diagnostics.

We present an analysis of the diagnostic technologies that were used to identify historical outbreaks of Ebola virus disease and consider systematic surveillance strategies that may greatly reduce the peak size of future epidemics. We observe that clinical signs and symptoms alone are often insufficient to recognize index cases of diseases of global concern against the considerable background infectious disease burden that is present throughout the developing world. We propose a simple sampling strategy to enrich in especially dangerous pathogens with a low background for molecular diagnostics by targeting blood-borne pathogens in the healthiest age groups. With existing multiplexed diagnostic technologies, such a system could be combined with existing public health screening and reference laboratory systems for malaria, dengue, and common bacteremia or be used to develop such an infrastructure in less-developed locations. Because the needs for valid samples and accurate recording of patient attributes are aligned with needs for global biosurveillance, local public health needs, and improving patient care, co-development of these capabilities appears to be quite natural, flexible, and extensible as capabilities, technologies, and needs evolve over time. Moreover, implementation of multiplexed diagnostic technologies to enhance fundamental clinical lab capacity will increase public health monitoring and biosurveillance as a natural extension.

The Biosurveillance Analytics Resource Directory (BARD): Facilitating the Use of Epidemiological Models for Infectious Disease Surveillance.

Epidemiological modeling for infectious disease is important for disease management and its routine implementation needs to be facilitated through better description of models in an operational context. A standardized model characterization process that allows selection or making manual comparisons of available models and their results is currently lacking. A key need is a universal framework to facilitate model description and understanding of its features. Los Alamos National Laboratory (LANL) has developed a comprehensive framework that can be used to characterize an infectious disease model in an operational context. The framework was developed through a consensus among a panel of subject matter experts. In this paper, we describe the framework, its application to model characterization, and the development of the Biosurveillance Analytics Resource Directory (BARD; http://brd.bsvgateway.org/brd/), to facilitate the rapid selection of operational models for specific infectious/communicable diseases. We offer this framework and associated database to stakeholders of the infectious disease modeling field as a tool for standardizing model description and facilitating the use of epidemiological models.

Forecasting the 2013-2014 influenza season using Wikipedia.

Infectious diseases are one of the leading causes of morbidity and mortality around the world; thus, forecasting their impact is crucial for planning an effective response strategy. According to the Centers for Disease Control and Prevention (CDC), seasonal influenza affects 5% to 20% of the U.S. population and causes major economic impacts resulting from hospitalization and absenteeism. Understanding influenza dynamics and forecasting its impact is fundamental for developing prevention and mitigation strategies. We combine modern data assimilation methods with Wikipedia access logs and CDC influenza-like illness (ILI) reports to create a weekly forecast for seasonal influenza. The methods are applied to the 2013-2014 influenza season but are sufficiently general to forecast any disease outbreak, given incidence or case count data. We adjust the initialization and parametrization of a disease model and show that this allows us to determine systematic model bias. In addition, we provide a way to determine where the model diverges from observation and evaluate forecast accuracy. Wikipedia article access logs are shown to be highly correlated with historical ILI records and allow for accurate prediction of ILI data several weeks before it becomes available. The results show that prior to the peak of the flu season, our forecasting method produced 50% and 95% credible intervals for the 2013-2014 ILI observations that contained the actual observations for most weeks in the forecast. However, since our model does not account for re-infection or multiple strains of influenza, the tail of the epidemic is not predicted well after the peak of flu season has passed.

Global disease monitoring and forecasting with Wikipedia.

Infectious disease is a leading threat to public health, economic stability, and other key social structures. Efforts to mitigate these impacts depend on accurate and timely monitoring to measure the risk and progress of disease. Traditional, biologically-focused monitoring techniques are accurate but costly and slow; in response, new techniques based on social internet data, such as social media and search queries, are emerging. These efforts are promising, but important challenges in the areas of scientific peer review, breadth of diseases and countries, and forecasting hamper their operational usefulness. We examine a freely available, open data source for this use: access logs from the online encyclopedia Wikipedia. Using linear models, language as a proxy for location, and a systematic yet simple article selection procedure, we tested 14 location-disease combinations and demonstrate that these data feasibly support an approach that overcomes these challenges. Specifically, our proof-of-concept yields models with r2 up to 0.92, forecasting value up to the 28 days tested, and several pairs of models similar enough to suggest that transferring models from one location to another without re-training is feasible. Based on these preliminary results, we close with a research agenda designed to overcome these challenges and produce a disease monitoring and forecasting system that is significantly more effective, robust, and globally comprehensive than the current state of the art.

Selecting essential information for biosurveillance--a multi-criteria decision analysis.

The National Strategy for Biosurveillance defines biosurveillance as "the process of gathering, integrating, interpreting, and communicating essential information related to all-hazards threats or disease activity affecting human, animal, or plant health to achieve early detection and warning, contribute to overall situational awareness of the health aspects of an incident, and to enable better decision-making at all levels." However, the strategy does not specify how "essential information" is to be identified and integrated into the current biosurveillance enterprise, or what the metrics qualify information as being "essential". The question of data stream identification and selection requires a structured methodology that can systematically evaluate the tradeoffs between the many criteria that need to be taken in account. Multi-Attribute Utility Theory, a type of multi-criteria decision analysis, can provide a well-defined, structured approach that can offer solutions to this problem. While the use of Multi-Attribute Utility Theoryas a practical method to apply formal scientific decision theoretical approaches to complex, multi-criteria problems has been demonstrated in a variety of fields, this method has never been applied to decision support in biosurveillance.We have developed a formalized decision support analytic framework that can facilitate identification of "essential information" for use in biosurveillance systems or processes and we offer this framework to the global BSV community as a tool for optimizing the BSV enterprise. To demonstrate utility, we applied the framework to the problem of evaluating data streams for use in an integrated global infectious disease surveillance system.