A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E).

BACKGROUND: Observational epidemiologic studies provide critical data for the evaluation of the potential effects of environmental, occupational and behavioural exposures on human health. Systematic reviews of these studies play a key role in informing policy and practice. Systematic reviews should incorporate assessments of the risk of bias in results of the included studies. OBJECTIVE: To develop a new tool, Risk Of Bias In Non-randomized Studies - of Exposures (ROBINS-E) to assess risk of bias in estimates from cohort studies of the causal effect of an exposure on an outcome. METHODS AND RESULTS: ROBINS-E was developed by a large group of researchers from diverse research and public health disciplines through a series of working groups, in-person meetings and pilot testing phases. The tool aims to assess the risk of bias in a specific result (exposure effect estimate) from an individual observational study that examines the effect of an exposure on an outcome. A series of preliminary considerations informs the core ROBINS-E assessment, including details of the result being assessed and the causal effect being estimated. The assessment addresses bias within seven domains, through a series of 'signalling questions'. Domain-level judgements about risk of bias are derived from the answers to these questions, then combined to produce an overall risk of bias judgement for the result, together with judgements about the direction of bias. CONCLUSION: ROBINS-E provides a standardized framework for examining potential biases in results from cohort studies. Future work will produce variants of the tool for other epidemiologic study designs (e.g. case-control studies). We believe that ROBINS-E represents an important development in the integration of exposure assessment, evidence synthesis and causal inference.

I-REFF diagrams: enhancing transparency in systematic review through interactive reference flow diagrams.

Systematic review methods are recognized for their rigor and transparency and are widely adapted to frameworks that cover review types such as systematic reviews, scoping reviews, and systematic evidence maps. Reporting guidelines help promote better systematic review practices and detailed documentation of the review process for different types of health research (e.g., PRISMA-Preferred Reporting Items for Systematic Reviews and Meta-Analyses; CONSORT-Consolidated Standards of Reporting Trials; and STROBE-Strengthening the Reporting of Observational Studies in Epidemiology). Transparency in the systematic review process and reporting of results is one of the key advantages of the methods and particularly important for hazard and risk assessments due to the high level of scrutiny these reviews face from scientific, political, and public communities. Data visualizations are important to clearly convey information from a review by helping readers perceive, understand, and assess the displayed information easily and quickly. The study flow diagram is a required element of a systematic review and maps out the number of included and excluded records identified, and the reasons for exclusion. Static literature flow diagrams help viewers readily understand the general review methodology and summarize the number of records included or excluded at each stage of the review. However, such diagrams can be time-consuming to develop and maintain during a systematic review or scoping review, and they provide limited summary-level information. We explored how the use of online systematic review tools such as DistillerSR coupled with visualization software such as Tableau can efficiently generate an Interactive REFerence Flow (I-REFF) diagram that is linked to the literature screening data, thus requiring minimal preparation, and resulting in a simplified process for updating the diagram. Furthermore, I-REFF diagrams enhance transparency and traceability by not only summarizing the records in the review but also allowing viewers to follow specific records throughout the review process. We present an example I-REFF diagram and discuss recommendations for key interactive elements to include in these diagrams and how this workflow can improve efficiency and result in an accessible and transparent interactive literature flow diagram without advanced programming.

Systematic evidence mapping informs a class-based approach to assessing personal care products and pubertal timing.

BACKGROUND: Personal care products (PCPs) contain many different compounds and are a source of exposure to endocrine disrupting chemicals (EDCs), including phthalates and phenols. Early-life exposure to EDCs commonly found in PCPs has been linked to earlier onset of puberty. OBJECTIVE: To characterize the human and animal evidence on the association between puberty-related outcomes and exposure to PCPs and their chemical constituents and, if there is sufficient evidence, identify groups of chemicals and outcomes to support a systematic review for a class-based hazard or risk assessment. METHODS: We followed the OHAT systematic review framework to characterize the human and animal evidence on the association between puberty-related health outcomes and exposure to PCPs and their chemical constituents. RESULTS: Ninety-eight human and 299 animal studies that evaluated a total of 96 different chemicals were identified and mapped by key concepts including chemical class, data stream, and puberty-related health outcome. Among these studies, phthalates and phenols were the most well-studied chemical classes. Most of the phthalate and phenol studies examined secondary sex characteristics and changes in estradiol and testosterone levels. Studies evaluating PCP use and other chemical classes (e.g., parabens) had less data. CONCLUSIONS: This systematic evidence map identified and mapped the published research evaluating the association between exposure to PCPs and their chemical constituents and puberty-related health outcomes. The resulting interactive visualization allows researchers to make evidence-based decisions on the available research by enabling them to search, sort, and filter the literature base of puberty-related studies by key concepts. This map can be used by researchers and regulators to prioritize and target future research and funding to reduce uncertainties and address data gaps. It also provides information to inform a class-based hazard or risk assessment on the association between phthalate and phenol exposures and puberty-related health outcomes.

Becoming aWARE: The Development of a Web-Based Tool for Autism Research and the Environment.

A sharp rise in autism spectrum disorder (ASD) prevalence estimates, beginning in the 1990s, suggested factors additional to genetics were at play. This stimulated increased research investment in nongenetic factors, including the study of environmental chemical exposures, diet, nutrition, lifestyle, social factors, and maternal medical conditions. Consequently, both peer- and non-peer-reviewed bodies of evidence investigating environmental contributors to ASD etiology have grown significantly. The heterogeneity in the design and conduct of this research results in an inconclusive and unwieldy 'virtual stack' of publications. We propose to develop a Web-based tool for Autism Research and the Environment (aWARE) to comprehensively aggregate and assess these highly variable and often conflicting data. The interactive aWARE tool will use an approach for the development of systematic evidence maps (SEMs) to identify and display all available relevant published evidence, enabling users to explore multiple research questions within the scope of the SEM. Throughout tool development, listening sessions and workshops will be used to seek perspectives from the broader autism community. New evidence will be indexed in the tool annually, which will serve as a living resource to investigate the association between environmental factors and ASD.

Evaluating endocrine disrupting chemicals: A perspective on the novel assessments in CLARITY-BPA.

BACKGROUND: The Consortium Linking Academic and Regulatory Insights on Bisphenol A Toxicity (CLARITY-BPA) was a collaborative research effort to better link academic research with governmental guideline studies. This review explores the secondary goal of CLARITY-BPA: to identify endpoints or technologies from CLARITY-BPA and prior/concurrent literature from these laboratories that may enhance the capacity of rodent toxicity studies to detect endocrine disrupting chemicals (EDCs). METHODS: A systematic literature search was conducted with search terms for BPA and the CLARITY-BPA participants. Relevant studies employed a laboratory rodent model and reported results on 1 of the 10 organs/organ systems evaluated in CLARITY-BPA (brain and behavior, cardiac, immune, mammary gland, ovary, penile function, prostate gland and urethra, testis and epididymis, thyroid hormone and metabolism, and uterus). Study design and findings were summarized, and a risk-of-bias assessment was conducted. RESULTS: Several endpoints and methods were identified as potentially helpful to detect effects of EDCs. For example, molecular and quantitative morphological approaches were sensitive in detecting alterations in early postnatal development of the brain, ovary, and mammary glands. Hormone challenge studies mimicking human aging reported increased susceptibility of the prostate to disease following developmental BPA exposure. Statistical analyses for nonmonotonic dose responses, and computational approaches assessing multiple treatment-related outcomes concurrently in linked hormone-sensitive organ systems, reported effects at low BPA doses. CONCLUSIONS: This review provided an opportunity to evaluate the unique insights provided by nontraditional assessments in CLARITY-BPA to identify technologies and endpoints to enhance detection of EDCs in future studies.

Protocol for designing INVITES-IN, a tool for assessing the internal validity of in vitro studies.

This protocol describes the design and development of a tool for evaluation of the internal validity of in vitro studies, which is needed to include the data as evidence in systematic reviews and chemical risk assessments. The tool will be designed specifically to be applied to cell culture studies, including, but not restricted to, studies meeting the new approach methodology (NAM) definition. The tool is called INVITES-IN (IN VITro Experimental Studies INternal validity). In this protocol, three of the four studies that will be performed to create the release version of INVITES-IN are described. In the first study, evaluation of existing assessment tools will be combined with focus group discussions to identify how characteristics of the design or conduct of an in vitro study can affect its internal validity. Bias domains and items considered to be of relevance for in vitro studies will be identified. In the second study, group agreement on internal validity domains and items of importance for in vitro studies will be identified via a modified Delphi methodology. In the third study, the draft version of the tool will be created, based on the data on relevance and importance of bias domains and items collected in Studies 1 and 2. A separate protocol will be prepared for the fourth study, which includes the user testing and validation of the tool, and collection of users' experience.

Consensus on the Key Characteristics of Immunotoxic Agents as a Basis for Hazard Identification.

BACKGROUND: Key characteristics (KCs), properties of agents or exposures that confer potential hazard, have been developed for carcinogens and other toxicant classes. KCs have been used in the systematic assessment of hazards and to identify assay and data gaps that limit screening and risk assessment. Many of the mechanisms through which pharmaceuticals and occupational or environmental agents modulate immune function are well recognized. Thus KCs could be identified for immunoactive substances and applied to improve hazard assessment of immunodulatory agents. OBJECTIVES: The goal was to generate a consensus-based synthesis of scientific evidence describing the KCs of agents known to cause immunotoxicity and potential applications, such as assays to measure the KCs. METHODS: A committee of 18 experts with diverse specialties identified 10 KCs of immunotoxic agents, namely, 1) covalently binds to proteins to form novel antigens, 2) affects antigen processing and presentation, 3) alters immune cell signaling, 4) alters immune cell proliferation, 5) modifies cellular differentiation, 6) alters immune cell-cell communication, 7) alters effector function of specific cell types, 8) alters immune cell trafficking, 9) alters cell death processes, and 10) breaks down immune tolerance. The group considered how these KCs could influence immune processes and contribute to hypersensitivity, inappropriate enhancement, immunosuppression, or autoimmunity. DISCUSSION: KCs can be used to improve efforts to identify agents that cause immunotoxicity via one or more mechanisms, to develop better testing and biomarker approaches to evaluate immunotoxicity, and to enable a more comprehensive and mechanistic understanding of adverse effects of exposures on the immune system. https://doi.org/10.1289/EHP10800.

Development of an Evidence-Based Risk Assessment Framework.

Assessment of potential human health risks associated with environmental and other agents requires careful evaluation of all available and relevant evidence for the agent of interest, including both data-rich and data-poor agents. With the advent of new approach methodologies in toxicological risk assessment, guidance on integrating evidence from mul-tiple evidence streams is needed to ensure that all available data is given due consideration in both qualitative and quantitative risk assessment. The present report summarizes the discussions among academic, government, and private sector participants from North America and Europe in an international workshop convened to explore the development of an evidence-based risk assessment framework, taking into account all available evidence in an appropriate manner in order to arrive at the best possible characterization of potential human health risks and associated uncertainty. Although consensus among workshop participants was not a specific goal, there was general agreement on the key consider-ations involved in evidence-based risk assessment incorporating 21st century science into human health risk assessment. These considerations have been embodied into an overarching prototype framework for evidence integration that will be explored in more depth in a follow-up meeting.

Assessment of key characteristics, methodology, and effect size measures used in meta-analysis of human-health-related animal studies.

Since the early 1990s the number of systematic reviews (SR) of animal studies has steadily increased. There is, however, little guidance on when and how to conduct a meta-analysis of human-health-related animal studies. To gain insight about the methods that are currently used we created an overview of the key characteristics of published meta-analyses of animal studies, with a focus on the choice of effect size measures. An additional goal was to learn about the rationale behind the meta-analysis methods used by the review authors. We show that important details of the meta-analyses are not fully described, only a fraction of all human-health-related meta-analyses provided rationales for their decision to use specific effect size measures. In addition, our data may suggest that authors make post-hoc decisions to switch to another effect size measure during the course of their meta-analysis, and possibly search for significant effects. Based on analyses in this paper we recommend that review teams: 1) publish a review protocol before starting the conduct of a SR, prespecifying all methodological details (providing special attention to the planned meta-analysis including the effect size measure and the rational behind choosing a specific effect size, prespecifying subgroups and restricting the number of subgroup analyses), 2) always use the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) checklist to report your SR of animal studies, and 3) use the random effects model (REM) in human-health-related meta-analysis of animal studies, unless the assumptions for using the fixed effect model (FEM) are all met.

Identifying environmental factors that influence immune response to SARS-CoV-2: Systematic evidence map protocol.

BACKGROUND: Widespread environmental contamination can directly interact with human immune system functions. Environmental effects on the immune system may influence human susceptibility to respiratory infections as well as the severity of infectious diseases, such as the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Furthermore, the efficacy of vaccines to respiratory diseases may be impacted by environmental exposures through immune perturbations. Given the quick pace of research about COVID-19 and associated risk factors, it is critical to identify and curate the streams of evidence quickly and effectively. OBJECTIVE: We developed this systematic evidence map protocol to identify and organize existing human and animal literature on high-priority environmental chemical classes (Per- and polyfluoroalkyl substances, pesticides, phthalates, quaternary ammonium compounds, and air pollutants) and their potential to influence three key outcomes: (1) susceptibility to respiratory infection, including SARS-CoV-2 (2) severity of the resultant disease progression, and (3) impact on vaccine efficacy. The result of this project will be an online, interactive database which will show what evidence is currently available between involuntary exposures to select environmental chemicals and immune health effects, data gaps that require further research, and data rich areas that may support further analysis. SEARCH AND STUDY ELIGIBILITY: We will search PubMed for epidemiological or toxicological literature on select toxicants from each of the chemical classes and each of the three outcomes listed above. STUDY APPRAISAL AND SYNTHESIS OF METHODS: For each study, two independent reviewers will conduct title and abstract screening as well as full text review for data extraction of study characteristics. Study quality will not be evaluated in this evidence mapping. The main findings from the systematic evidence map will be visualized using a publicly available and interactive database hosted on Tableau Public.

Application of evidence-based methods to construct mechanism-driven chemical assessment frameworks.

The workshop titled "Application of evidence-based methods to construct mechanism-driven chemical assessment frameworks" was co-organized by the Evidence-based Toxicology Collaboration and the European Food Safety Authority (EFSA) and hosted by EFSA at its headquarters in Parma, Italy on October 2 and 3, 2019. The goal was to explore integration of systematic review with mechanistic evidence evaluation. Participants were invited to work on concrete products to advance the exploration of how evidence-based approaches can support the development and application of adverse outcome pathways (AOP) in chemical risk assessment. The workshop discussions were centered around three related themes: 1) assessing certainty in AOPs, 2) literature-based AOP development, and 3) integrating certainty in AOPs and non-animal evidence into decision frameworks. Several challenges, mostly related to methodology, were identified and largely determined the workshop recommendations. The workshop recommendations included the comparison and potential alignment of processes used to develop AOP and systematic review methodology, including the translation of vocabulary of evidence-based methods to AOP and vice versa, the development and improvement of evidence mapping and text mining methods and tools, as well as a call for a fundamental change in chemical risk and uncertainty assessment methodology if to be conducted based on AOPs and new approach methodologies (NAM). The usefulness of evidence-based approaches for mechanism-based chemical risk assessments was stressed, particularly the potential contribution of the rigor and transparency inherent to such approaches in building stakeholders' trust for implementation of NAM evidence and AOPs into chemical risk assessment.

Evaluation of a semi-automated data extraction tool for public health literature-based reviews: Dextr.

INTRODUCTION: There has been limited development and uptake of machine-learning methods to automate data extraction for literature-based assessments. Although advanced extraction approaches have been applied to some clinical research reviews, existing methods are not well suited for addressing toxicology or environmental health questions due to unique data needs to support reviews in these fields. OBJECTIVES: To develop and evaluate a flexible, web-based tool for semi-automated data extraction that: 1) makes data extraction predictions with user verification, 2) integrates token-level annotations, and 3) connects extracted entities to support hierarchical data extraction. METHODS: Dextr was developed with Agile software methodology using a two-team approach. The development team outlined proposed features and coded the software. The advisory team guided developers and evaluated Dextr's performance on precision, recall, and extraction time by comparing a manual extraction workflow to a semi-automated extraction workflow using a dataset of 51 environmental health animal studies. RESULTS: The semi-automated workflow did not appear to affect precision rate (96.0% vs. 95.4% manual, p = 0.38), resulted in a small reduction in recall rate (91.8% vs. 97.0% manual, p < 0.01), and substantially reduced the median extraction time (436 s vs. 933 s per study manual, p < 0.01) compared to a manual workflow. DISCUSSION: Dextr provides similar performance to manual extraction in terms of recall and precision and greatly reduces data extraction time. Unlike other tools, Dextr provides the ability to extract complex concepts (e.g., multiple experiments with various exposures and doses within a single study), properly connect the extracted elements within a study, and effectively limit the work required by researchers to generate machine-readable, annotated exports. The Dextr tool addresses data-extraction challenges associated with environmental health sciences literature with a simple user interface, incorporates the key capabilities of user verification and entity connecting, provides a platform for further automation developments, and has the potential to improve data extraction for literature reviews in this and other fields.

An approach to quantifying the potential importance of residual confounding in systematic reviews of observational studies: A GRADE concept paper.

Small relative effect sizes are common in observational studies of exposure in environmental and public health. However, such effects can still have considerable policy importance when the baseline rate of the health outcome is high, and many persons are exposed. Assessing the certainty of the evidence based on these effect sizes is challenging because they can be prone to residual confounding due to the non-randomized nature of the evidence. When applying GRADE, a precise relative risk >2.0 increases the certainty in an existing effect because residual confounding is unlikely to explain the association. GRADE also suggests rating up when opposing plausible residual confounding exists for other effect sizes. In this concept paper, we propose using the E-value, defined as the smallest effect size of a confounder that still can reduce an observed RR to the null value, and a reference confounder to assess the likelihood of residual confounding. We propose a 4-step approach. 1. Assess the association of interest for relevant exposure levels. 2. Calculate the E-value for this observed association. 3. Choose a reference confounder with sufficient strength and information and assess its effect on the observed association using the E-value. 4. Assess how likely it is that residual confounding will still bias the observed RR. We present three case studies and discuss the feasibility of the approach.

Integrating Personal Air Sensor and GPS to Determine Microenvironment-Specific Exposures to Volatile Organic Compounds.

Personal exposure to volatile organic compounds (VOCs) from indoor sources including consumer products is an understudied public health concern. To develop and evaluate methods for monitoring personal VOC exposures, we performed a pilot study and examined time-resolved sensor-based measurements of geocoded total VOC (TVOC) exposures across individuals and microenvironments (MEs). We integrated continuous (1 min) data from a personal TVOC sensor and a global positioning system (GPS) logger, with a GPS-based ME classification model, to determine TVOC exposures in four MEs, including indoors at home (Home-In), indoors at other buildings (Other-In), inside vehicles (In-Vehicle), and outdoors (Out), across 45 participant-days for five participants. To help identify places with large emission sources, we identified high-exposure events (HEEs; TVOC > 500 ppb) using geocoded TVOC time-course data overlaid on Google Earth maps. Across the 45 participant-days, the MEs ranked from highest to lowest median TVOC were: Home-In (165 ppb), Other-In (86 ppb), In-Vehicle (52 ppb), and Out (46 ppb). For the two participants living in single-family houses with attached garages, the median exposures for Home-In were substantially higher (209, 416 ppb) than the three participant homes without attached garages: one living in a single-family house (129 ppb), and two living in apartments (38, 60 ppb). The daily average Home-In exposures exceeded the estimated Leadership in Energy and Environmental Design (LEED) building guideline of 108 ppb for 60% of the participant-days. We identified 94 HEEs across all participant-days, and 67% of the corresponding peak levels exceeded 1000 ppb. The MEs ranked from the highest to the lowest number of HEEs were: Home-In (60), Other-In (13), In-Vehicle (12), and Out (9). For Other-In and Out, most HEEs occurred indoors at fast food restaurants and retail stores, and outdoors in parking lots, respectively. For Home-In HEEs, the median TVOC emission and removal rates were 5.4 g h-1 and 1.1 h-1, respectively. Our study demonstrates the ability to determine individual sensor-based time-resolved TVOC exposures in different MEs, in support of identifying potential sources and exposure factors that can inform exposure mitigation strategies.

Improving the quality of toxicology and environmental health systematic reviews: What journal editors can do.

Systematic reviews are fast increasing in prevalence in the toxicology and environmental health literature. However, how well these complex research projects are being conducted and reported is unclear. Since editors have an essential role in ensuring the scientific quality of manuscripts being published in their journals, a workshop was convened where editors, systematic review practitioners, and research quality control experts could discuss what editors can do to ensure the systematic reviews they publish are of sufficient scientific quality. Interventions were explored along four themes: setting standards; reviewing protocols; optimizing editorial workflows; and measuring the effectiveness of editorial interventions. In total, 58 editorial interventions were proposed. Of these, 26 were shortlisted for being potentially effective, and 5 were prioritized as short-term actions that editors could relatively easily take to improve the quality of published systematic reviews. Recent progress in improving systematic reviews is summarized, and outstanding challenges to further progress are highlighted.

Developing a database of systematic reviews of animal studies.

Systematic reviews (SRs) are common practice in clinical and public health research, but less common in non-human animal research. Systematic reviews of animal studies can be valuable to inform clinical research, to evaluate the need for further animal experiments on a given topic, and to assess the hazard of an environmental exposure in the evaluation of toxicological studies. In the last 10 years, there has been an increase in the number of SRs of animal research, as well as several publications with detailed guidance on how to perform high-quality systematic reviews of experimental animal studies. In order to evaluate current analytical approaches used in SRs of animal studies, easily identify all systematic reviews on a specific topic, and subsequently the original animal studies and their results and promote awareness and understanding of these emerging approaches, we compiled a database of SRs of animal studies. The database was developed using a rigorous, systematic approach and covers a broad range of research fields: preclinical research, toxicology, environmental health, and veterinary medicine. The database currently includes 3113 SRs of animal studies (search date June 2019). In addition to bibliographical information, data on whether or not a risk of bias assessment and meta-analysis were conducted were extracted. For future users, the search features of the database provide users with a platform to identify and select SRs with a particular characteristic for export to Microsoft Word or Microsoft Excel. From there, users may perform additional data extraction to meet their research needs. The database is freely available at www.Mendeley.com (link). The database provides methodologists a comprehensive source that can be used to explore and advance the current methodology applied to SRs of animal studies, and can help researchers to easily identify all systematic reviews on a specific topic, and subsequently the original animal studies and their results and avoid duplication and unnecessary animal research.

Applying evidence-based methods to the development and use of adverse outcome pathways

The workshop "Application of evidence-based methods to construct mechanistic frameworks for the development and use of non-animal toxicity tests" was organized by the Evidence-based Toxicology Collaboration and hosted by the Grading of Recommendations Assessment, Development and Evaluation Working Group on June 12, 2019. The purpose of the workshop was to bring together international regulatory bodies, risk assessors, academic scientists, and industry to explore how systematic review methods and the adverse outcome pathway framework could be combined to develop and use mechanistic test methods for predicting the toxicity of chemical substances in an evidence-based manner. The meeting covered the history of biological frameworks, the way adverse outcome pathways are currently developed, the basic principles of systematic methodology, including systematic reviews and evidence maps, and assessment of cer­tainty in models, and adverse outcome pathways in particular. Specific topics were discussed via case studies in small break-out groups. The group concluded that adverse outcome pathways provide an important framework to support mechanism-based assessment in environmental health. The process of their development has a few challenges that could be addressed with systematic methods and automation tools. Addressing these challenges will increase the transparency of the evidence behind adverse outcome pathways and the consistency with which they are defined; this in turn will increase their value for supporting public health decisions. It was suggested to explore the details of applying systematic methods to adverse outcome pathway development in a series of case studies and workshops.

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making.

OBJECTIVES: The objective of the study is to present the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty of evidence from modeling studies (i.e., certainty associated with model outputs). STUDY DESIGN AND SETTING: Expert consultations and an international multidisciplinary workshop informed development of a conceptual approach to assessing the certainty of evidence from models within the context of systematic reviews, health technology assessments, and health care decisions. The discussions also clarified selected concepts and terminology used in the GRADE approach and by the modeling community. Feedback from experts in a broad range of modeling and health care disciplines addressed the content validity of the approach. RESULTS: Workshop participants agreed that the domains determining the certainty of evidence previously identified in the GRADE approach (risk of bias, indirectness, inconsistency, imprecision, reporting bias, magnitude of an effect, dose-response relation, and the direction of residual confounding) also apply when assessing the certainty of evidence from models. The assessment depends on the nature of model inputs and the model itself and on whether one is evaluating evidence from a single model or multiple models. We propose a framework for selecting the best available evidence from models: 1) developing de novo, a model specific to the situation of interest, 2) identifying an existing model, the outputs of which provide the highest certainty evidence for the situation of interest, either "off-the-shelf" or after adaptation, and 3) using outputs from multiple models. We also present a summary of preferred terminology to facilitate communication among modeling and health care disciplines. CONCLUSION: This conceptual GRADE approach provides a framework for using evidence from models in health decision-making and the assessment of certainty of evidence from a model or models. The GRADE Working Group and the modeling community are currently developing the detailed methods and related guidance for assessing specific domains determining the certainty of evidence from models across health care-related disciplines (e.g., therapeutic decision-making, toxicology, environmental health, and health economics).

Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper.

OBJECTIVES: The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses (1-2 week) in the clinical, public health, and health systems fields. STUDY DESIGN AND SETTING: We conducted a review of the literature, outlined a draft approach, refined the concept through iterative discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group, and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group. RESULTS: A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2) assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the adequacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommendations, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new recommendations using expert panel input; and (8) consider an updating plan. CONCLUSION: An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidisciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the principles of transparency; and properly managing conflicts of interest.