Quantifying the determinants of outbreak detection performance through simulation and machine learning.

OBJECTIVE: To develop a probabilistic model for discovering and quantifying determinants of outbreak detection and to use the model to predict detection performance for new outbreaks. MATERIALS AND METHODS: We used an existing software platform to simulate waterborne disease outbreaks of varying duration and magnitude. The simulated data were overlaid on real data from visits to emergency department in Montreal for gastroenteritis. We analyzed the combined data using biosurveillance algorithms, varying their parameters over a wide range. We then applied structure and parameter learning algorithms to the resulting data set to build a Bayesian network model for predicting detection performance as a function of outbreak characteristics and surveillance system parameters. We evaluated the predictions of this model through 5-fold cross-validation. RESULTS: The model predicted performance metrics of commonly used outbreak detection methods with an accuracy greater than 0.80. The model also quantified the influence of different outbreak characteristics and parameters of biosurveillance algorithms on detection performance in practically relevant surveillance scenarios. In addition to identifying characteristics expected a priori to have a strong influence on detection performance, such as the alerting threshold and the peak size of the outbreak, the model suggested an important role for other algorithm features, such as adjustment for weekly patterns. CONCLUSION: We developed a model that accurately predicts how characteristics of disease outbreaks and detection methods will influence on detection. This model can be used to compare the performance of detection methods under different surveillance scenarios, to gain insight into which characteristics of outbreaks and biosurveillance algorithms drive detection performance, and to guide the configuration of surveillance systems.

Optimizing the response to surveillance alerts in automated surveillance systems.

Although much research effort has been directed toward refining algorithms for disease outbreak alerting, considerably less attention has been given to the response to alerts generated from statistical detection algorithms. Given the inherent inaccuracy in alerting, it is imperative to develop methods that help public health personnel identify optimal policies in response to alerts. This study evaluates the application of dynamic decision making models to the problem of responding to outbreak detection methods, using anthrax surveillance as an example. Adaptive optimization through approximate dynamic programming is used to generate a policy for decision making following outbreak detection. We investigate the degree to which the model can tolerate noise theoretically, in order to keep near optimal behavior. We also evaluate the policy from our model empirically and compare it with current approaches in routine public health practice for investigating alerts. Timeliness of outbreak confirmation and total costs associated with the decisions made are used as performance measures. Using our approach, on average, 80 per cent of outbreaks were confirmed prior to the fifth day of post-attack with considerably less cost compared to response strategies currently in use. Experimental results are also provided to illustrate the robustness of the adaptive optimization approach and to show the realization of the derived error bounds in practice.

Incorporating educational institutions into a unified model of health human resources.

This commentary discusses Tzountzouris and Gilbert's article on issues related to emerging health human resources (HHR) requirements and the role of educational institutions in anticipating and meeting them. It is apparent that educational institutions are influenced by and, in turn, influence a number of elements of HHR. They therefore have a unique role in responding to emerging HHR needs, from assessing the needs to evaluating responses. While this commentary shares the general idea presented by the authors of the lead paper, it outlines some of the issues that should be taken into consideration in formulating these roles.

Using hierarchical mixture of experts model for fusion of outbreak detection methods.

A wide variety of disease outbreak detection methods has been developed in automated public health surveillance systems. The choice of outbreak detection method results in large changes in performance under different circumstances. In this paper, we investigate how outbreak detection methods can be combined in order to improve the overall detection performance. We used Hierarchical Mixture of Experts, which is a probabilistic model for combining classification methods, for fusion of detection methods. Simulated surveillance data for waterborne disease outbreaks are used in this research to train and evaluate a Hierarchical Mixture of Experts model. Performance evaluation of our approach with respect to sensitivity-specificity trade-off and detection timeliness is provided in comparison with several other detection methods.

PHIO: a knowledge base for interpretation and calculation of public health indicators.

Existing population health indicators tend to be out-of-date, not fully available at local levels of geography, and not developed in a coherent/consistent manner, which hinders their use in public health. The PopHR platform aims to deliver an electronic repository that contains multiple aggregated clinical, administrative, and environmental data sources to provide a coherent view of the health status of populations in the province of Quebec, Canada. This platform is designed to provide representative information in near-real time with high geographical resolution, thereby assisting public health professionals, analysts, clinicians and the public in decision-making. This paper presents our ongoing efforts to develop an integrated population health indicator ontology (PHIO) that captures the knowledge required for calculation and interpretation of health indicators within a PopHR semantic framework.

A Bayesian network model for analysis of detection performance in surveillance systems.

Worldwide developments concerning infectious diseases and bioterrorism are driving forces for improving aberrancy detection in public health surveillance. The performance of an aberrancy detection algorithm can be measured in terms of sensitivity, specificity and timeliness. However, these metrics are probabilistically dependent variables and there is always a trade-off between them. This situation raises the question of how to quantify this tradeoff. The answer to this question depends on the characteristics of the specific disease under surveillance, the characteristics of data used for surveillance, and the algorithmic properties of detection methods. In practice, the evidence describing the relative performance of different algorithms remains fragmented and mainly qualitative. In this paper, we consider the development and evaluation of a Bayesian network framework for analysis of performance measures of aberrancy detection algorithms. This framework enables principled comparison of algorithms and identification of suitable algorithms for use in specific public health surveillance settings.

Decision theoretic analysis of improving epidemic detection.

The potentially catastrophic impact of an epidemic specially these due to bioterrorist attack, makes developing effective detection methods essential for public health. Current detection methods trade off reliability of alarms for early detection of outbreaks. The performance of these methods can be improved by disease-specific modeling techniques that take into account the potential costs and effects of an attack to provide optimal warnings and the cost and effectiveness of interventions. We study this optimization problem in the framework of sequential decision making under uncertainty. Our approach relies on estimating the future benefit of true alarms and the costs of false alarms. Using these quantities it identifies optimal decisions regarding the credibility of outputs from a traditional detection method at each point in time. The key contribution of this paper is to apply Partially Observable Markov Decision Processes (POMDPs) on outbreak detection methods for improving alarm function in the case of anthrax. We present empirical evidence illustrating that at a fixed specificity, the performance of detection methods with respect to sensitivity and timeliness is improved significantly by utilizing POMDPs in detection of anthrax attacks.