Case studies to explore the optimal use of randomized and nonrandomized studies in evidence syntheses that use GRADE.

OBJECTIVES: Randomized controlled trials (RCTs) are the preferred source of evidence for the relative effect of healthcare interventions summarized in knowledge syntheses. Nonrandomized studies of interventions (NRSI) may provide replacement, sequential, or complementary evidence to RCTs. The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach can provide different options for properly using RCTs and NRSI integrated in health syntheses. In this article, we discuss different implications on the certainty of evidence when authors consider the use of NRSI and RCTs in systematic reviews using GRADE. Although this is a GRADE-related article, it is not an official GRADE guidance or concept article. STUDY DESIGN AND SETTING: We present case studies used during GRADE working group meetings for discussion of the effects of using NRSI and RCTs on GRADE domains and on the certainty of evidence. Several concepts were discussed through iterative feedback with experts in GRADE methods and Cochrane authors. We compared suggested solutions for possible scenarios that can be met in evidence syntheses informing decisions and future guidance. RESULTS: Different scenarios for the use of RCTs and NRSI in evidence syntheses are presented, focusing on how different GRADE ratings between RCTs and NRSI affect the overall assessment of the evidence and possible health recommendations. CONCLUSIONS: Considering differences and similarities grounded in the GRADE approach between NRSI and RCTs may help complement one another and maximize the value of knowledge syntheses and health recommendations.

Application of evidence-based methods to construct mechanism-driven chemical assessment frameworks.

The workshop titled "Application of evidence-based methods to construct mechanism-driven chemical assessment frameworks" was co-organized by the Evidence-based Toxicology Collaboration and the European Food Safety Authority (EFSA) and hosted by EFSA at its headquarters in Parma, Italy on October 2 and 3, 2019. The goal was to explore integration of systematic review with mechanistic evidence evaluation. Participants were invited to work on concrete products to advance the exploration of how evidence-based approaches can support the development and application of adverse outcome pathways (AOP) in chemical risk assessment. The workshop discussions were centered around three related themes: 1) assessing certainty in AOPs, 2) literature-based AOP development, and 3) integrating certainty in AOPs and non-animal evidence into decision frameworks. Several challenges, mostly related to methodology, were identified and largely determined the workshop recommendations. The workshop recommendations included the comparison and potential alignment of processes used to develop AOP and systematic review methodology, including the translation of vocabulary of evidence-based methods to AOP and vice versa, the development and improvement of evidence mapping and text mining methods and tools, as well as a call for a fundamental change in chemical risk and uncertainty assessment methodology if to be conducted based on AOPs and new approach methodologies (NAM). The usefulness of evidence-based approaches for mechanism-based chemical risk assessments was stressed, particularly the potential contribution of the rigor and transparency inherent to such approaches in building stakeholders' trust for implementation of NAM evidence and AOPs into chemical risk assessment.

GRADE guidance 24 optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelines.

BACKGROUND AND OBJECTIVE: This is the 24th in the ongoing series of articles describing the GRADE approach for assessing the certainty of a body of evidence in systematic reviews and health technology assessments and how to move from evidence to recommendations in guidelines. METHODS: Guideline developers and authors of systematic reviews and other evidence syntheses use randomized controlled studies (RCTs) and non-randomized studies of interventions (NRSI) as sources of evidence for questions about health interventions. RCTs with low risk of bias are the most trustworthy source of evidence for estimating relative effects of interventions because of protection against confounding and other biases. However, in several instances, NRSI can still provide valuable information as complementary, sequential, or replacement evidence for RCTs. RESULTS: In this article we offer guidance on the decision regarding when to search for and include either or both types of studies in systematic reviews to inform health recommendations. CONCLUSION: This work aims to help methodologists in review teams, technology assessors, guideline panelists, and anyone conducting evidence syntheses using GRADE.

Sourcing data on chemical properties and hazard data from the US-EPA CompTox Chemicals Dashboard: A practical guide for human risk assessment.

For the past six decades, human health risk assessment of chemicals has relied on in vivo data from human epidemiological and experimental animal toxicological studies to inform the derivation of non-cancer toxicity values. The ongoing evolution of this risk assessment paradigm in an environmental landscape of data-poor chemicals has highlighted the need to develop and implement non-testing methods, so-called New Approach Methodologies (NAMs). NAMs include a growing number of in silico and in vitro data streams designed to inform hazard properties of chemicals, including kinetics and dynamics at different levels of biological organization, environmental fate and transport, and exposure. NAMs provide a fit-for-purpose science-basis for human hazard and risk characterization of chemicals ranging from data-gap filling applications to broad evidence-based decision-making. Systematic assembly and delivery of empirical and predicted data for chemicals are paramount to advancing chemical evaluation, and software tools serve an essential role in delivering these data to the scientific community. The CompTox Chemicals Dashboard (from here on referred to as the "Dashboard") is one such tool and is a publicly available web-based application developed by the US Environmental Protection Agency to provide access to chemistry, toxicity and exposure information for ~900,000 chemicals. The Dashboard is increasingly becoming a valuable resource for assessors tasked with the evaluation of potential human health risks associated with chemical exposures. In this context, the significant amount of information present in the Dashboard facilitates: 1) assembly of information on physicochemical properties and environmental fate and transport and exposure parameters and metrics; 2) identification of cancer and non-cancer health effects from extant human and experimental animal studies in the public domain and/or information not available in the public domain (i.e., "grey literature"); 3) systematic literature searching and review for developing cancer and non-cancer hazard evidence bases; and 4) access to mechanistic information that can aid or augment the analysis of traditional toxicology evidence bases, or potentially, serve as the primary basis for informing hazard identification and dose-response when traditional bioassay data are lacking. Finally, in silico predictive tools developed to conduct structure-activity or read-across analyses are also available within the Dashboard. This practical tutorial is intended to address key questions from the human health risk assessment community dealing with chemicals in both food and in the environment. Perspectives for future development or refinement of the Dashboard highlight foreseen activities to further support the research and risk assessment community in cancer and non-cancer chemical evaluations.

Human exposure pathways to poly- and perfluoroalkyl substances (PFAS) from indoor media: A systematic review protocol.

BACKGROUND: Human exposure to per- and polyfluoroalkyl substances (PFAS) has been primarily attributed to contaminated food and drinking water. However, additional PFAS exposure pathways have been raised by a limited number of studies reporting correlations between commercial and industrial products and PFAS levels in human media and biomonitoring. Systematic review (SR) methodologies have been widely used to evaluate similar questions using an unbiased approach in the fields of clinical medicine, epidemiology, and toxicology, but the deployment in exposure science is ongoing. Here we present a systematic review protocol that adapts existing systematic review methodologies and study evaluation tools to exposure science studies in order to investigate evidence for important PFAS exposure pathways from indoor media including consumer products, household articles, cleaning products, personal care products, plus indoor air and dust. OBJECTIVES: We will systematically review exposure science studies that present both PFAS concentrations from indoor exposure media and PFAS concentrations in blood serum or plasma. Exposure estimates will be synthesized from the evidence to answer the question, "For the general population, what effect does exposure from PFAS chemicals via indoor media have on blood, serum or plasma concentrations of PFAS?" We adapt existing systematic review methodologies and study evaluation tools from the U.S. EPA's Systematic Review Protocol for the PFBA, PFHxA, PFHxS, PFNA, and PFDA IRIS Assessments and the Navigation Guide for exposure science studies, as well as present innovative developments of exposure pathway-specific search strings for use in artificial intelligence screening software. DATA SOURCES: We will search electronic databases for potentially relevant literature, including Web of Science, PubMed, and ProQuest. Literature search results will be stored in EPA's Health and Environmental Research Online (HERO) database. STUDY ELIGIBILITY AND CRITERIA: Included studies will present exposure measures from indoor media including consumer products, household articles, cleaning products, personal care products, plus indoor air and dust, paired with PFAS concentrations in blood, serum or plasma from adults and/or children in the general population. We focus on a subset of PFAS chemicals including perfluorooctanoic acid (PFOA), perfluorooctanesulfonate (PFOS), perfluorobutanoic acid (PFBA), perfluorobutane sulfonate (PFBS), perfluorodecanoic acid (PFDA), perfluorohexanoic acid (PFHxA), perfluorohexanesulfonate (PFHxS), and perfluorononanoic acid (PFNA). STUDY APPRAISAL AND SYNTHESIS METHODS: Studies will be prefiltered at the title and abstract level using computationally intelligent search strings to expedite the screening process for reviewers. Two independent reviewers will screen the prefiltered studies against inclusion criteria at the title/abstract level and then full-text level, after which the reviewers will assess the studies' risk of bias using an approach modified from established systematic review tools for exposure studies. Exposure estimates will be calculated to investigate the proportion of blood, serum or plasma) PFAS concentrations that can be explained by exposure to PFAS in indoor media.

Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper.

OBJECTIVES: The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses (1-2 week) in the clinical, public health, and health systems fields. STUDY DESIGN AND SETTING: We conducted a review of the literature, outlined a draft approach, refined the concept through iterative discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group, and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group. RESULTS: A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2) assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the adequacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommendations, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new recommendations using expert panel input; and (8) consider an updating plan. CONCLUSION: An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidisciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the principles of transparency; and properly managing conflicts of interest.

A novel study evaluation strategy in the systematic review of animal toxicology studies for human health assessments of environmental chemicals.

A key aspect of the systematic review process is study evaluation to understand the strengths and weaknesses of individual studies included in the review. The present manuscript describes the process currently being used by the Environmental Protection Agency's (EPA) Integrated Risk Information System (IRIS) Program to evaluate animal toxicity studies, illustrated by application to the recent systematic reviews of two phthalates: diisobutyl phthalate (DIBP) and diethyl phthalate (DEP). The IRIS Program uses a domain-based approach that was developed after careful consideration of tools used by others to evaluate experimental animal studies in toxicology and pre-clinical research. Standard practice is to have studies evaluated by at least two independent reviewers for aspects related to reporting quality, risk of bias/internal validity (e.g., randomization, blinding at outcome assessment, methods used to expose animals and assess outcomes, etc.), and sensitivity to identify factors that may limit the ability of a study to detect a true effect. To promote consistency across raters, prompting considerations and example responses are provided to reviewers, and a pilot phase is conducted. The evaluation process is performed separately for each outcome reported in a study, as the utility of a study may vary for different outcomes. Input from subject matter experts is used to identify chemical- and outcome-specific considerations (e.g., lifestage of exposure and outcome assessment when considering reproductive effects) to guide judgments within particular evaluation domains. For each evaluation domain, reviewers reach a consensus on a rating of Good, Adequate, Deficient, or Critically Deficient. These individual domain ratings are then used to determine the overall confidence in the study (High Confidence, Medium Confidence, Low Confidence, or Deficient). Study evaluation results, including the justifications for reviewer judgements, are documented and made publicly available in EPA's version of Health Assessment Workspace Collaborative (HAWC), a free and open source web-based software application. (The views expressed are those of the authors and do not necessarily represent the views or policies of the US EPA).

GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.

OBJECTIVE: To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process. STUDY DESIGN AND SETTING: The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group. RESULTS: We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I. CONCLUSIONS: The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

A scoping review and survey provides the rationale, perceptions, and preferences for the integration of randomized and nonrandomized studies in evidence syntheses and GRADE assessments.

OBJECTIVES: To review the literature and obtain preferences and perceptions from experts regarding the role of randomized studies (RSs) and nonrandomized studies (NRSs) in systematic reviews of intervention effects. STUDY DESIGN AND SETTING: Scoping review and survey of experts. Using levels of certainty developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group, experts expressed their preferences about the use of RS and NRS in health syntheses. RESULTS: Of 189 respondents, 123 had the expertise required to answer the questionnaire; 116 provided their extent of agreement with approaches to use NRS with RS. Most respondents would include NRS when RS was unfeasible (83.6%) or unethical (71.5%) and a majority to maximize the body of evidence (66.3%), compare results in NRS and RS (53.5%) and to identify subgroups (51.7%). Sizable minorities would include NRS and RS to address the effect of randomization (29.5%) or because the question being addressed was a public-health intervention (36.5%). In summary of findings tables, most respondents would include both bodies of evidence-in two rows in the same table-when RS provided moderate, low, or very-low certainty evidence; even when RS provided high certainty evidence, a sizable minority (25%) would still present results from both bodies of evidence. Very few (3.6%) would, under realistic circumstances, pool RS and NRS results. CONCLUSIONS: Most experts would include both RS and NRS in the same review under a wide variety of circumstances, but almost all would present results of two bodies of evidence separately.