When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting.

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design.

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Participatory Mathematical Modeling Approach for Policymaking during the First Year of the COVID-19 Crisis, Jordan.

We engaged in a participatory modeling approach with health sector stakeholders in Jordan to support government decision-making regarding implementing public health measures to mitigate COVID-19 disease burden. We considered the effect of 4 physical distancing strategies on reducing COVID-19 transmission and mortality in Jordan during March 2020-January 2021: no physical distancing; intermittent physical distancing where all but essential services are closed once a week; intermittent physical distancing where all but essential services are closed twice a week; and a permanent physical distancing intervention. Modeling showed that the fourth strategy would be most effective in reducing cases and deaths; however, this approach was only marginally beneficial to reducing COVID-19 disease compared with an intermittently enforced physical distancing intervention. Scenario-based model influenced policy-making and the evolution of the pandemic in Jordan confirmed the forecasting provided by the modeling exercise and helped confirm the effectiveness of the policy adopted by the government of Jordan.

Multiple models for outbreak decision support in the face of uncertainty.

Policymakers must make management decisions despite incomplete knowledge and conflicting model projections. Little guidance exists for the rapid, representative, and unbiased collection of policy-relevant scientific input from independent modeling teams. Integrating approaches from decision analysis, expert judgment, and model aggregation, we convened multiple modeling teams to evaluate COVID-19 reopening strategies for a mid-sized United States county early in the pandemic. Projections from seventeen distinct models were inconsistent in magnitude but highly consistent in ranking interventions. The 6-mo-ahead aggregate projections were well in line with observed outbreaks in mid-sized US counties. The aggregate results showed that up to half the population could be infected with full workplace reopening, while workplace restrictions reduced median cumulative infections by 82%. Rankings of interventions were consistent across public health objectives, but there was a strong trade-off between public health outcomes and duration of workplace closures, and no win-win intermediate reopening strategies were identified. Between-model variation was high; the aggregate results thus provide valuable risk quantification for decision making. This approach can be applied to the evaluation of management interventions in any setting where models are used to inform decision making. This case study demonstrated the utility of our approach and was one of several multimodel efforts that laid the groundwork for the COVID-19 Scenario Modeling Hub, which has provided multiple rounds of real-time scenario projections for situational awareness and decision making to the Centers for Disease Control and Prevention since December 2020.

Context-dependent representation of within- and between-model uncertainty: aggregating probabilistic predictions in infectious disease epidemiology.

Probabilistic predictions support public health planning and decision making, especially in infectious disease emergencies. Aggregating outputs from multiple models yields more robust predictions of outcomes and associated uncertainty. While the selection of an aggregation method can be guided by retrospective performance evaluations, this is not always possible. For example, if predictions are conditional on assumptions about how the future will unfold (e.g. possible interventions), these assumptions may never materialize, precluding any direct comparison between predictions and observations. Here, we summarize literature on aggregating probabilistic predictions, illustrate various methods for infectious disease predictions via simulation, and present a strategy for choosing an aggregation method when empirical validation cannot be used. We focus on the linear opinion pool (LOP) and Vincent average, common methods that make different assumptions about between-prediction uncertainty. We contend that assumptions of the aggregation method should align with a hypothesis about how uncertainty is expressed within and between predictions from different sources. The LOP assumes that between-prediction uncertainty is meaningful and should be retained, while the Vincent average assumes that between-prediction uncertainty is akin to sampling error and should not be preserved. We provide an R package for implementation. Given the rising importance of multi-model infectious disease hubs, our work provides useful guidance on aggregation and a deeper understanding of the benefits and risks of different approaches.

Projected outcomes of universal testing and treatment in a generalised HIV epidemic in Zambia and South Africa (the HPTN 071 [PopART] trial): a modelling study.

BACKGROUND: The long-term impact of universal home-based testing and treatment as part of universal testing and treatment (UTT) on HIV incidence is unknown. We made projections using a detailed individual-based model of the effect of the intervention delivered in the HPTN 071 (PopART) cluster-randomised trial. METHODS: In this modelling study, we fitted an individual-based model to the HIV epidemic and HIV care cascade in 21 high prevalence communities in Zambia and South Africa that were part of the PopART cluster-randomised trial (intervention period Nov 1, 2013, to Dec 31, 2017). The model represents coverage of home-based testing and counselling by age and sex, delivered as part of the trial, antiretroviral therapy (ART) uptake, and any changes in national guidelines on ART eligibility. In PopART, communities were randomly assigned to one of three arms: arm A received the full PopART intervention for all individuals who tested positive for HIV, arm B received the intervention with ART provided in accordance with national guidelines, and arm C received standard of care. We fitted the model to trial data twice using Approximate Bayesian Computation, once before data unblinding and then again after data unblinding. We compared projections of intervention impact with observed effects, and for four different scenarios of UTT up to Jan 1, 2030 in the study communities. FINDINGS: Compared with standard of care, a 51% (95% credible interval 40-60) reduction in HIV incidence is projected if the trial intervention (arms A and B combined) is continued from 2020 to 2030, over and above a declining trend in HIV incidence under standard of care. INTERPRETATION: A widespread and continued commitment to UTT via home-based testing and counselling can have a substantial effect on HIV incidence in high prevalence communities. FUNDING: National Institute of Allergy and Infectious Diseases, US President's Emergency Plan for AIDS Relief, International Initiative for Impact Evaluation, Bill & Melinda Gates Foundation, National Institute on Drug Abuse, and National Institute of Mental Health.

Vote-processing rules for combining control recommendations from multiple models.

Mathematical modelling is used during disease outbreaks to compare control interventions. Using multiple models, the best method to combine model recommendations is unclear. Existing methods weight model projections, then rank control interventions using the combined projections, presuming model outputs are directly comparable. However, the way each model represents the epidemiological system will vary. We apply electoral vote-processing rules to combine model-generated rankings of interventions. Combining rankings of interventions, instead of combining model projections, avoids assuming that projections are comparable as all comparisons of projections are made within each model. We investigate four rules: First-past-the-post, Alternative Vote (AV), Coombs Method and Borda Count. We investigate rule sensitivity by including models that favour only one action or including those that rank interventions randomly. We investigate two case studies: the 2014 Ebola outbreak in West Africa (37 compartmental models) and a hypothetical foot-and-mouth disease outbreak in UK (four individual-based models). The Coombs Method was least susceptible to adding models that favoured a single action, Borda Count and AV were most susceptible to adding models that ranked interventions randomly. Each rule chose the same intervention as when ranking interventions by mean projections, suggesting that combining rankings provides similar recommendations with fewer assumptions about model comparability. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.

Estimating SARS-CoV-2 variant fitness and the impact of interventions in England using statistical and geo-spatial agent-based models.

The SARS-CoV-2 epidemic has been extended by the evolution of more transmissible viral variants. In autumn 2020, the B.1.177 lineage became the dominant variant in England, before being replaced by the B.1.1.7 (Alpha) lineage in late 2020, with the sweep occurring at different times in each region. This period coincided with a large number of non-pharmaceutical interventions (e.g. lockdowns) to control the epidemic, making it difficult to estimate the relative transmissibility of variants. In this paper, we model the spatial spread of these variants in England using a meta-population agent-based model which correctly characterizes the regional variation in cases and distribution of variants. As a test of robustness, we additionally estimated the relative transmissibility of multiple variants using a statistical model based on the renewal equation, which simultaneously estimates the effective reproduction number R. Relative to earlier variants, the transmissibility of B.1.177 is estimated to have increased by 1.14 (1.12-1.16) and that of Alpha by 1.71 (1.65-1.77). The vaccination programme starting in December 2020 is also modelled. Counterfactual simulations demonstrate that the vaccination programme was essential for reopening in March 2021, and that if the January lockdown had started one month earlier, up to 30 k (24 k-38 k) deaths could have been prevented. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.

Synergistic interventions to control COVID-19: Mass testing and isolation mitigates reliance on distancing.

Stay-at-home orders and shutdowns of non-essential businesses are powerful, but socially costly, tools to control the pandemic spread of SARS-CoV-2. Mass testing strategies, which rely on widely administered frequent and rapid diagnostics to identify and isolate infected individuals, could be a potentially less disruptive management strategy, particularly where vaccine access is limited. In this paper, we assess the extent to which mass testing and isolation strategies can reduce reliance on socially costly non-pharmaceutical interventions, such as distancing and shutdowns. We develop a multi-compartmental model of SARS-CoV-2 transmission incorporating both preventative non-pharmaceutical interventions (NPIs) and testing and isolation to evaluate their combined effect on public health outcomes. Our model is designed to be a policy-guiding tool that captures important realities of the testing system, including constraints on test administration and non-random testing allocation. We show how strategic changes in the characteristics of the testing system, including test administration, test delays, and test sensitivity, can reduce reliance on preventative NPIs without compromising public health outcomes in the future. The lowest NPI levels are possible only when many tests are administered and test delays are short, given limited immunity in the population. Reducing reliance on NPIs is highly dependent on the ability of a testing program to identify and isolate unreported, asymptomatic infections. Changes in NPIs, including the intensity of lockdowns and stay at home orders, should be coordinated with increases in testing to ensure epidemic control; otherwise small additional lifting of these NPIs can lead to dramatic increases in infections, hospitalizations and deaths. Importantly, our results can be used to guide ramp-up of testing capacity in outbreak settings, allow for the flexible design of combined interventions based on social context, and inform future cost-benefit analyses to identify efficient pandemic management strategies.

PopART-IBM, a highly efficient stochastic individual-based simulation model of generalised HIV epidemics developed in the context of the HPTN 071 (PopART) trial.

Mathematical models are powerful tools in HIV epidemiology, producing quantitative projections of key indicators such as HIV incidence and prevalence. In order to improve the accuracy of predictions, such models need to incorporate a number of behavioural and biological heterogeneities, especially those related to the sexual network within which HIV transmission occurs. An individual-based model, which explicitly models sexual partnerships, is thus often the most natural type of model to choose. In this paper we present PopART-IBM, a computationally efficient individual-based model capable of simulating 50 years of an HIV epidemic in a large, high-prevalence community in under a minute. We show how the model calibrates within a Bayesian inference framework to detailed age- and sex-stratified data from multiple sources on HIV prevalence, awareness of HIV status, ART status, and viral suppression for an HPTN 071 (PopART) study community in Zambia, and present future projections of HIV prevalence and incidence for this community in the absence of trial intervention.

OpenABM-Covid19-An agent-based model for non-pharmaceutical interventions against COVID-19 including contact tracing.

SARS-CoV-2 has spread across the world, causing high mortality and unprecedented restrictions on social and economic activity. Policymakers are assessing how best to navigate through the ongoing epidemic, with computational models being used to predict the spread of infection and assess the impact of public health measures. Here, we present OpenABM-Covid19: an agent-based simulation of the epidemic including detailed age-stratification and realistic social networks. By default the model is parameterised to UK demographics and calibrated to the UK epidemic, however, it can easily be re-parameterised for other countries. OpenABM-Covid19 can evaluate non-pharmaceutical interventions, including both manual and digital contact tracing, and vaccination programmes. It can simulate a population of 1 million people in seconds per day, allowing parameter sweeps and formal statistical model-based inference. The code is open-source and has been developed by teams both inside and outside academia, with an emphasis on formal testing, documentation, modularity and transparency. A key feature of OpenABM-Covid19 are its Python and R interfaces, which has allowed scientists and policymakers to simulate dynamic packages of interventions and help compare options to suppress the COVID-19 epidemic.

Strategic testing approaches for targeted disease monitoring can be used to inform pandemic decision-making.

More than 1.6 million Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) tests were administered daily in the United States at the peak of the epidemic, with a significant focus on individual treatment. Here, we show that objective-driven, strategic sampling designs and analyses can maximize information gain at the population level, which is necessary to increase situational awareness and predict, prepare for, and respond to a pandemic, while also continuing to inform individual treatment. By focusing on specific objectives such as individual treatment or disease prediction and control (e.g., via the collection of population-level statistics to inform lockdown measures or vaccine rollout) and drawing from the literature on capture-recapture methods to deal with nonrandom sampling and testing errors, we illustrate how public health objectives can be achieved even with limited test availability when testing programs are designed a priori to meet those objectives.

Causes of delayed outbreak responses and their impacts on epidemic spread.

Livestock diseases have devastating consequences economically, socially and politically across the globe. In certain systems, pathogens remain viable after host death, which enables residual transmissions from infected carcasses. Rapid culling and carcass disposal are well-established strategies for stamping out an outbreak and limiting its impact; however, wait-times for these procedures, i.e. response delays, are typically farm-specific and time-varying due to logistical constraints. Failing to incorporate variable response delays in epidemiological models may understate outbreak projections and mislead management decisions. We revisited the 2001 foot-and-mouth epidemic in the United Kingdom and sought to understand how misrepresented response delays can influence model predictions. Survival analysis identified farm size and control demand as key factors that impeded timely culling and disposal activities on individual farms. Using these factors in the context of an existing policy to predict local variation in response times significantly affected predictions at the national scale. Models that assumed fixed, timely responses grossly underestimated epidemic severity and its long-term consequences. As a result, this study demonstrates how general inclusion of response dynamics and recognition of partial controllability of interventions can help inform management priorities during epidemics of livestock diseases.

Cost and cost-effectiveness of a universal HIV testing and treatment intervention in Zambia and South Africa: evidence and projections from the HPTN 071 (PopART) trial.

BACKGROUND: The HPTN 071 (PopART) trial showed that a combination HIV prevention package including universal HIV testing and treatment (UTT) reduced population-level incidence of HIV compared with standard care. However, evidence is scarce on the costs and cost-effectiveness of such an intervention. METHODS: Using an individual-based model, we simulated the PopART intervention and standard care with antiretroviral therapy (ART) provided according to national guidelines for the 21 trial communities in Zambia and South Africa (for all individuals aged >14 years), with model parameters and primary cost data collected during the PopART trial and from published sources. Two intervention scenarios were modelled: annual rounds of PopART from 2014 to 2030 (PopART 2014-30; as the UNAIDS Fast-Track target year) and three rounds of PopART throughout the trial intervention period (PopART 2014-17). For each country, we calculated incremental cost-effectiveness ratios (ICERs) as the cost per disability-adjusted life-year (DALY) and cost per HIV infection averted. Cost-effectiveness acceptability curves were used to indicate the probability of PopART being cost-effective compared with standard care at different thresholds of cost per DALY averted. We also assessed budget impact by projecting undiscounted costs of the intervention compared with standard care up to 2030. FINDINGS: During 2014-17, the mean cost per person per year of delivering home-based HIV counselling and testing, linkage to care, promotion of ART adherence, and voluntary medical male circumcision via community HIV care providers for the simulated population was US$6·53 (SD 0·29) in Zambia and US$7·93 (0·16) in South Africa. In the PopART 2014-30 scenario, median ICERs for PopART delivered annually until 2030 were $2111 (95% credible interval [CrI] 1827-2462) per HIV infection averted in Zambia and $3248 (2472-3963) per HIV infection averted in South Africa; and $593 (95% CrI 526-674) per DALY averted in Zambia and $645 (538-757) per DALY averted in South Africa. In the PopART 2014-17 scenario, PopART averted one infection at a cost of $1318 (1098-1591) in Zambia and $2236 (1601-2916) in South Africa, and averted one DALY at $258 (225-298) in Zambia and $326 (266-391) in South Africa, when outcomes were projected until 2030. The intervention had almost 100% probability of being cost-effective at thresholds greater than $700 per DALY averted in Zambia, and greater than $800 per DALY averted in South Africa, in the PopART 2014-30 scenario. Incremental programme costs for annual rounds until 2030 were $46·12 million (for a mean of 341 323 people) in Zambia and $30·24 million (for a mean of 165 852 people) in South Africa. INTERPRETATION: Combination prevention with universal home-based testing can be delivered at low annual cost per person but accumulates to a considerable amount when scaled for a growing population. Combination prevention including UTT is cost-effective at thresholds greater than $800 per DALY averted and can be an efficient strategy to reduce HIV incidence in high-prevalence settings. FUNDING: US National Institutes of Health, President's Emergency Plan for AIDS Relief, International Initiative for Impact Evaluation, Bill & Melinda Gates Foundation.

COVID-19 reopening strategies at the county level in the face of uncertainty: Multiple Models for Outbreak Decision Support.

Policymakers make decisions about COVID-19 management in the face of considerable uncertainty. We convened multiple modeling teams to evaluate reopening strategies for a mid-sized county in the United States, in a novel process designed to fully express scientific uncertainty while reducing linguistic uncertainty and cognitive biases. For the scenarios considered, the consensus from 17 distinct models was that a second outbreak will occur within 6 months of reopening, unless schools and non-essential workplaces remain closed. Up to half the population could be infected with full workplace reopening; non-essential business closures reduced median cumulative infections by 82%. Intermediate reopening interventions identified no win-win situations; there was a trade-off between public health outcomes and duration of workplace closures. Aggregate results captured twice the uncertainty of individual models, providing a more complete expression of risk for decision-making purposes.

Anticipating future learning affects current control decisions: A comparison between passive and active adaptive management in an epidemiological setting.

Infectious disease epidemics present a difficult task for policymakers, requiring the implementation of control strategies under significant time constraints and uncertainty. Mathematical models can be used to predict the outcome of control interventions, providing useful information to policymakers in the event of such an epidemic. However, these models suffer in the early stages of an outbreak from a lack of accurate, relevant information regarding the dynamics and spread of the disease and the efficacy of control. As such, recommendations provided by these models are often incorporated in an ad hoc fashion, as and when more reliable information becomes available. In this work, we show that such trial-and-error-type approaches to management, which do not formally take into account the resolution of uncertainty and how control actions affect this, can lead to sub-optimal management outcomes. We compare three approaches to managing a theoretical epidemic: a non-adaptive management (AM) approach that does not use real-time outbreak information to adapt control, a passive AM approach that incorporates real-time information if and when it becomes available, and an active AM approach that explicitly incorporates the future resolution of uncertainty through gathering real-time information into its initial recommendations. The structured framework of active AM encourages the specification of quantifiable objectives, models of system behaviour and possible control and monitoring actions, followed by an iterative learning and control phase that is able to employ complex control optimisations and resolve system uncertainty. The result is a management framework that is able to provide dynamic, long-term projections to help policymakers meet the objectives of management. We investigate in detail the effect of different methods of incorporating up-to-date outbreak information. We find that, even in a highly simplified system, the method of incorporating new data can lead to different results that may influence initial policy decisions, with an active AM approach to management providing better information that can lead to more desirable outcomes from an epidemic.

A large-scale application of project prioritization to threatened species investment by a government agency.

In a global environment of increasing species extinctions and decreasing availability of funds with which to combat the causes of biodiversity loss, maximising the efficiency of conservation efforts is crucial. The only way to ensure maximum return on conservation investment is to incorporate the cost, benefit and likelihood of success of conservation actions into decision-making in a systematic and objective way. Here we report on the application of a Project Prioritization Protocol (PPP), first implemented by the New Zealand Government, to target and prioritize investment in threatened species in New South Wales, Australia, under the state's new Saving our Species program. Detailed management prescriptions for 368 threatened species were developed via an expert elicitation process, and were then prioritized using quantitative data on benefit, likelihood of success and implementation cost, and a simple cost-efficiency equation. We discuss the outcomes that have been realized even in the early stages of the program; including the efficient development of planning resources made available to all potential threatened species investors and the demonstration of a transparent and objective approach to threatened species management that will significantly increase the probability of meeting an objective to secure the greatest number of threatened species from extinction.

Real-time decision-making during emergency disease outbreaks.

In the event of a new infectious disease outbreak, mathematical and simulation models are commonly used to inform policy by evaluating which control strategies will minimize the impact of the epidemic. In the early stages of such outbreaks, substantial parameter uncertainty may limit the ability of models to provide accurate predictions, and policymakers do not have the luxury of waiting for data to alleviate this state of uncertainty. For policymakers, however, it is the selection of the optimal control intervention in the face of uncertainty, rather than accuracy of model predictions, that is the measure of success that counts. We simulate the process of real-time decision-making by fitting an epidemic model to observed, spatially-explicit, infection data at weekly intervals throughout two historical outbreaks of foot-and-mouth disease, UK in 2001 and Miyazaki, Japan in 2010, and compare forward simulations of the impact of switching to an alternative control intervention at the time point in question. These are compared to policy recommendations generated in hindsight using data from the entire outbreak, thereby comparing the best we could have done at the time with the best we could have done in retrospect. Our results show that the control policy that would have been chosen using all the data is also identified from an early stage in an outbreak using only the available data, despite high variability in projections of epidemic size. Critically, we find that it is an improved understanding of the locations of infected farms, rather than improved estimates of transmission parameters, that drives improved prediction of the relative performance of control interventions. However, the ability to estimate undetected infectious premises is a function of uncertainty in the transmission parameters. Here, we demonstrate the need for both real-time model fitting and generating projections to evaluate alternative control interventions throughout an outbreak. Our results highlight the use of using models at outbreak onset to inform policy and the importance of state-dependent interventions that adapt in response to additional information throughout an outbreak.

Essential information: Uncertainty and optimal control of Ebola outbreaks.

Early resolution of uncertainty during an epidemic outbreak can lead to rapid and efficient decision making, provided that the uncertainty affects prioritization of actions. The wide range in caseload projections for the 2014 Ebola outbreak caused great concern and debate about the utility of models. By coding and running 37 published Ebola models with five candidate interventions, we found that, despite this large variation in caseload projection, the ranking of management options was relatively consistent. Reducing funeral transmission and reducing community transmission were generally ranked as the two best options. Value of information (VoI) analyses show that caseloads could be reduced by 11% by resolving all model-specific uncertainties, with information about model structure accounting for 82% of this reduction and uncertainty about caseload only accounting for 12%. Our study shows that the uncertainty that is of most interest epidemiologically may not be the same as the uncertainty that is most relevant for management. If the goal is to improve management outcomes, then the focus of study should be to identify and resolve those uncertainties that most hinder the choice of an optimal intervention. Our study further shows that simplifying multiple alternative models into a smaller number of relevant groups (here, with shared structure) could streamline the decision-making process and may allow for a better integration of epidemiological modeling and decision making for policy.