Assessing the Certainty of the Evidence in Systematic Reviews: Importance, Process, and Use.

When interpreting results and drawing conclusions, authors of systematic reviews should consider the limitations of the evidence included in their review. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach provides a framework for the explicit consideration of the limitations of the evidence included in a systematic review, and for incorporating this assessment into the conclusions. Assessments of certainty of evidence are a methodological expectation of systematic reviews. The certainty of the evidence is specific to each outcome in a systematic review, and can be rated as high, moderate, low, or very low. Because it will have an important impact, before conducting certainty of evidence, reviewers must clarify the intent of their question: are they interested in causation or association. Serious concerns regarding limitations in the study design, inconsistency, imprecision, indirectness, and publication bias can decrease the certainty of the evidence. Using an example, this article describes and illustrates the importance and the steps for assessing the certainty of evidence and drawing accurate conclusions in a systematic review.

Clinical systematic reviews - a brief overview.

OBJECTIVE: Systematic reviews answer research questions through a defined methodology. It is a complex task and multiple articles need to be referred to acquire wide range of required knowledge to conduct a systematic review. The aim of this article is to bring the process into a single paper. METHOD: The statistical concepts and sequence of steps to conduct a systematic review or a meta-analysis are examined by authors. RESULTS: The process of conducting a clinical systematic review is described in seven manageable steps in this article. Each step is explained with examples to understand the method evidently. CONCLUSION: A complex process of conducting a systematic review is presented simply in a single article.

Guidance to best tools and practices for systematic reviews.

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to best tools and practices for systematic reviews.

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

One-Third of Systematic Reviews in Rehabilitation Applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) System to Evaluate Certainty of Evidence: A Meta-Research Study.

OBJECTIVE: To determine how many systematic reviews (SRs) of the literature in rehabilitation assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to do this. DATA SOURCES: For this meta-research study, we searched PubMed and Cochrane Database of Systematic Reviews databases for SRs on rehabilitation published in 2020. STUDY SELECTION AND DATA EXTRACTION: Two reviewers independently selected the SRs and extracted the data. Reporting characteristics and appropriate use of the GRADE system were assessed. DATA SYNTHESIS: The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of which applied the GRADE system. GRADE was used by SRs of randomized controlled trials (RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according to the study design was not provided in 75% (12/16). The reasons for GRADE judgment were reported in 82% (134/163) of SRs. CONCLUSIONS: One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment was presented transparently by most SRs. Journal editors and funders should encourage the uptake of the GRADE system when considering SRs in rehabilitation for publication. The authors should pre-define GRADE assessment in a registered and/or published protocol.

Editor's Choice - European Society for Vascular Surgery Clinical Practice Guideline Development Scheme: An Overview of Evidence Quality Assessment Methods, Evidence to Decision Frameworks, and Reporting Standards in Guideline Development.

OBJECTIVE: A structured and transparent approach is instrumental in translating research evidence to health recommendations and evidence informed clinical decisions. The aim was to conduct an overview and analysis of principles and methodologies for health guideline development. METHODS: A literature review on methodologies, strategies, and fundamental steps in the process of guideline development was performed. The clinical practice guideline development process and methodology adopted by the European Society for Vascular Surgery are also presented. RESULTS: Sophisticated methodologies for health guideline development are being applied increasingly by national and international organisations. Their overarching principle is a systematic, structured, transparent, and iterative process that is aimed at making well informed healthcare choices. Critical steps in guideline development include the assessment of the certainty of the body of evidence; evidence to decision frameworks; and guideline reporting. The goal of strength of evidence assessments is to provide well reasoned judgements about the guideline developers' confidence in study findings, and several evidence hierarchy schemes and evidence rating systems have been described for this purpose. Evidence to decision frameworks help guideline developers and users conceptualise and interpret the construct of the quality of the body of evidence. The most widely used evidence to decision frameworks are those developed by the GRADE Working Group and the WHO-INTEGRATE, and are structured into three distinct components: background; assessment; and conclusions. Health guideline reporting tools are employed to ensure methodological rigour and transparency in guideline development. Such reporting instruments include the AGREE II and RIGHT, with the former being used for guideline development and appraisal, as well as reporting. CONCLUSION: This guide will help guideline developers/expert panels enhance their methodology, and patients/clinicians/policymakers interpret guideline recommendations and put them in context. This document may be a useful methodological summary for health guideline development by other societies and organisations.

An updated systematic review and meta-analysis on adherence to mediterranean diet and risk of cancer.

PURPOSE: The aim of current systematic review was to update the body of evidence on associations between adherence to the Mediterranean diet (MedDiet) and risk of cancer mortality, site-specific cancer in the general population; all-cause, and cancer mortality as well as cancer reoccurrence among cancer survivors. METHODS: A literature search for randomized controlled trials (RCTs), case-control and cohort studies published up to April 2020 was performed using PubMed and Scopus. Study-specific risk estimates for the highest versus lowest adherence to the MedDiet category were pooled using random-effects meta-analyses. Certainty of evidence from cohort studies and RCTs was evaluated using the NutriGrade scoring system. RESULTS: The updated search revealed 44 studies not identified in the previous review. Altogether, 117 studies including 3,202,496 participants were enclosed for meta-analysis. The highest adherence to MedDiet was inversely associated with cancer mortality (RRcohort: 0.87, 95% CI 0.82, 0.92; N = 18 studies), all-cause mortality among cancer survivors (RRcohort: 0.75, 95% CI 0.66, 0.86; N = 8), breast (RRobservational: 0.94, 95% CI 0.90, 0.97; N = 23), colorectal (RRobservational: 0.83, 95% CI 0.76, 0.90; N = 17), head and neck (RRobservational: 0.56, 95% CI 0.44, 0.72; N = 9), respiratory (RRcohort: 0.84, 95% CI 0.76, 0.94; N = 5), gastric (RRobservational: 0.70, 95% CI 0.61, 0.80; N = 7), bladder (RRobservational: 0.87, 95% CI 0.76, 0.98; N = 4), and liver cancer (RRobservational: 0.64, 95% CI 0.54, 0.75; N = 4). Adhering to MedDiet did not modify risk of blood, esophageal, pancreatic and prostate cancer risk. CONCLUSION: In conclusion, our results suggest that highest adherence to the MedDiet was related to lower risk of cancer mortality in the general population, and all-cause mortality among cancer survivors as well as colorectal, head and neck, respiratory, gastric, liver and bladder cancer risks. Moderate certainty of evidence from cohort studies suggest an inverse association for cancer mortality and colorectal cancer, but most of the comparisons were rated as low or very low certainty of evidence.

Network meta-analyses in reproductive medicine: challenges and opportunities.

Network meta-analysis allows researchers to synthesise both direct and indirect evidence, thus enabling simultaneous comparisons of multiple treatments. A relatively recent addition to evidence synthesis in reproductive medicine, this approach has become increasingly popular. Yet, the underlying assumptions of network meta-analyses, which drive the validity of their findings, have been frequently ignored. In this article, we discuss the strengths and limitations of network meta-analyses. In addition, we present an overview of published network meta-analyses in reproductive medicine, summarize their challenges and provide insights into future research opportunities.

The GRADE Working Group and CINeMA approaches provided inconsistent certainty of evidence ratings for a network meta-analysis of opioids for chronic noncancer pain.

OBJECTIVES: Assessment of the certainty of evidence (CoE) from network meta-analysis is critical to convey the strength of inferences for clinical decision-making. Both the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group (GWG) and the Confidence in Network Meta-Analysis (CINeMA) framework have been designed to assess the CoE of treatment effects informed by network meta-analysis; however, the concordance of results is uncertain. STUDY DESIGN AND SETTING: We assessed the CoE for treatment effects of individual opioids on pain relief and physical functioning from a network meta-analysis for chronic noncancer pain using the GWG approach and the CINeMA framework. Both approaches evaluate the CoE as high, moderate, low or very low. We quantified the number of discrepant CoE ratings between approaches and the magnitude of the difference (ie, one level, two levels, or three levels). RESULTS: Across 105 comparisons among individual opioids for pain relief, the GWG and CINeMA approaches provided different CoE ratings in 34% of cases (36 of 105). Across 66 comparisons for physical functioning, there was discordance in 17% of cases (11 of 66). All discrepancies were separated by one level. The CINeMA framework typically provided lower CoE ratings compared to the GWG approach, predominantly because of differences in the assessment of transitivity and heterogeneity. CONCLUSION: Our findings suggest there are differences between the CoE ratings provided by the GWG and CINeMA approaches when applied to network meta-analyses. Further research is needed to replicate or refute our findings in other network meta-analyses and assess the implications for clinical decision-making.

Uncovering gaps in GRADE reporting of Eastern Association for the Surgery of Trauma (EAST) guidelines.

OBJECTIVES: To formally evaluate the uptake and reporting of the Grading of Recommendation Assessments, Development and Evaluation (GRADE) approach in clinical practice guidelines (CPGs) developed by the Eastern Association for the Surgery of Trauma (EAST). STUDY DESIGN AND SETTING: Based on an a priori, written protocol, we developed a dedicated data abstraction form that included the six suggested criteria for using and applying GRADE. By searching the EAST website, we identified all EAST guidelines that referenced the use of GRADE. All steps of the data abstraction process were completed independently and in duplicate by two members of the research team. RESULTS: We identified a total of 48 CPGs that used GRADE. Trauma and violence prevention (n = 11; 23.9%) was the most common topic. The median number of patient/population, intervention, comparison, and outcomes (PICO) questions addressed was 3 (interquartile range: 2; 4) with a median of 2.5 (interquartile range: 1; 4) critical outcomes. A conditional/weak recommendation was provided for n = 79 (51.4%) PICOs, whereas a strong recommendation was provided for 33 PICOs (23.9%). For 22 PICOs (15.9%), no recommendation was made. Nearly all guideline documents provided search dates (n = 44; 95.7%) and a Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram (n = 44; 95.7%). Most described categories for rating down (n = 35; 76.1%). GRADE decision-making domains related to the ratio of benefits to harms, values and preferences, and resource utilization were referenced by 43.5% (n = 20), 43.5% (n = 20), and 30.4% (n = 14) of CPGs, respectively. For nearly half of PICO questions (n = 59; 44.2%) authors did not provide an evidence profile or summary of findings table. Comparing time periods from 2014-2018 to 2019-2022, the proportion of recommendations with an overall certainty of evidence increased (52.4% vs 83.9%; P < 0.001). CONCLUSION: EAST has successfully adopted GRADE to develop many trauma-related guidelines, each addressing a finite number of focused clinical questions based on systematic reviews conducted in-house. Overall reporting improved over time. There is for improvement when it comes to consistent provision of an overall certainty of evidence, the reporting of the evidence to decision-making process, and the justification of strong recommendations based on low/very low certainty evidence.

Guidance to best tools and practices for systematic reviews.

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

The impact of an online course on agreement rates of the certainty of evidence assessment using Grading of Recommendations, Assessment, Development, and Evaluation Approach: a before-and-after study.

BACKGROUND AND OBJECTIVE: The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach is a systematic method for assessing the certainty of evidence (CoE) and strength of recommendations in health care. We aimed to verify the effects of an online-based GRADE course on multirater consistency in the evaluation of the CoE in systematic reviews (SRs) analysis. STUDY DESIGN AND SETTINGS: Sixty-five Brazilian methodologists and researchers participated in an online course over 8 weeks. Asynchronous lessons and weekly synchronous meetings addressed the GRADE system in the context of CoE assessment. We asked participants to evaluate the CoE of random SRs (two before and another two after the course). Analyzes focused on the multirater agreement with a standard response, in the interrater agreement, and before-after changes in the proportion of participants that rated down the domains. RESULTS: 48 individuals completed the course. Participants presented improvements in the raters' assessment of the CoE using the GRADE approach after the course. The multirater consistency of indirectness, imprecision, and the overall CoE increased after the course, as well as the agreement between raters and the standard response. Furthermore, interrater reliability increased for risk of bias, inconsistency, indirectness, publication bias, and overall CoE, indicating progress in between-raters consistency. After the course, approximately 78% of individuals rated down the overall CoE to a low/very low degree, and participants presented more explanations for the judgment of each domain. CONCLUSION: An online GRADE course improved the consistency and agreement of the CoE assessment by Brazilian researchers. Online training courses have the potential to improve skills in guideline methodology development.

Efficacy of photobiomodulation in the management of oral Lichen Planus in comparison to topical corticosteroids: Systematic review, meta-analysis, and GRADE-based assessment of certainty of evidence.

BACKGROUND: There is an urgent need to update the evidence available on the efficacy of photobiomodulation (PBM) in comparison to topical corticosteroids in the management of Oral Lichen Planus (OLP). METHODS: Cochrane Oral Health's Trials Register, Cochrane Central Register of Controlled Trials in the Cochrane Library, MEDLINE Pubmed, SCOPUS, Lilacs and Google Scholar were searched. Other sources included US National Institutes of Health Ongoing Trials Register, World Health Organization International Clinical Trials Registry Platform and Clinical Trial Registry - India were searched using variations of the keywords "Laser" and "Oral Lichen Planus". Randomized controlled trials comparing PBM and topical corticosteroids in the resolution of pain among OLP patients were included. The studies were assessed using the Cochrane Risk-of-Bias assessment tool. Publication bias was assessed using a funnel plot, and the certainty of evidence was evaluated according to the GRADE guidelines. RESULTS: Ten studies were included for qualitative assessment and of these eight were included in the meta-analysis. The included studies used laser parameters of varying strengths and duration. Meta-analysis favoured PBM (n = 274, MD =-0.48, CI -0.66- -0.30) for pain score. No adverse effects were reported for laser therapy. There was high heterogeneity and moderate certainty of evidence, and most studies had a high risk of bias. CONCLUSION: There is improvement in the clinical parameters of OLP when treated with PBM in comparison to topical corticosteroids. However, the strength of the evidence for these findings in limited. It is recommended to conduct better long-term trials with large sample size.

Compound Kushen injection combined with transarterial chemoembolization for hepatocellular carcinoma: An evidence map and overview of systematic reviews.

ETHNOPHARMACOLOGICAL RELEVANCE: For the treatment of hepatocellular carcinoma (HCC), compound Kushen injection (CKi) is commonly used in combination with transarterial chemoembolization (TACE). AIMS OF THE STUDY: Our objective was to evaluate the reporting quality, methodological quality, risk of bias, and certainty of evidence for CKi combined with TACE for the treatment of patients with HCC by conducting systematic reviews (SRs). The purpose of this study was to improve the clinical application of CKis, strengthen clinical decision-making regarding CKis, and inform future research. MATERIALS AND METHODS: We used eight databases to systematically search SRs of CKi combined with TACE for HCC through February 21, 2023. The quality of reporting of SRs was evaluated using the 2009 Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, methodological quality using A MeaSurement Tool to Assess systematic Reviews 2, risk of bias using the Risk of Bias in Systematic Review, and certainty of evidence using the Grading of Recommendations Assessment. Finally, the assessment results were visualized by the evidence mapping method. This overview has been registered on PROSPERO with the registration title "Compound Kushen injection for hepatocellular carcinoma: An overview of systematic reviews" and registration number CRD42022369120. RESULTS: A total of 12 SRs meeting the inclusion criteria were included. In terms of reporting quality, 42% of SRs reported relatively complete reports and 58% had certain deficiencies. The methodological quality of all SRs was " critically low". The risk of bias was evaluated as low in 33% of SRs and high in 67% of SRs. The results of the evidence synthesis showed that, in the "moderate" level of evidence, CKi combined with TACE resulted in a 12.7%-21.5% benefit for one-year survival rate, 11.7%-17.2% benefit for objective response rate (ORR), 20.5%-27.1% benefit for quality of life, 22.2% benefit for nausea and vomiting, and 24.7%-27.4% benefit for leukopenia in HCC patients. CONCLUSION: In conclusion, CKi combined with TACE improved survival, ORR and quality of life in patients with HCC, and reduced adverse events. The results should be interpreted with caution due to the low methodological quality of the included SRs. The clinical efficacy of CKis must be confirmed in a large number of randomized controlled trials.

GRADE concept 6: a novel application of external indirect evidence into GRADE ratings of evidence certainty in network meta-analysis.

OBJECTIVES: We describe how consideration of external evidence may play an important role in judging certainty in the process of establishing the certainty of the evidence. Our example is a network meta-analysis (NMA) addressing treatment for Ebola virus disease, which informed a World Health Organization guideline. STUDY DESIGN AND SETTING: Through Grading of Recommendations Assessment, Development, and Evaluations (GRADE) project group iterative online, in-person and email discussions, we developed this GRADE concept and obtained approval from the GRADE working group. Using the null as a threshold, we rated our certainty for network estimates in mortality, including consideration of evidence external to the NMA (i.e., did not meet eligibility criteria) and formal logical construction. RESULTS: Based on the existing GRADE guidance, we rated the network estimate for one indirect comparison as low certainty. The formal logical construction that lead us reevaluate the certainty of the evidence is as follows: if A is superior to B, and B is not inferior to C, then A must be superior to C. After considering the logic and the external indirect evidence, we concluded at least moderate certainty for the comparison. CONCLUSION: Systematic review authors and guideline developers should apply the fundamental logical construction for indirect comparisons and consider compelling external evidence in NMA certainty ratings.

GRADE Use in Evidence Syntheses Published in High-Impact-Factor Gynecology and Obstetrics Journals: A Methodological Survey.

Objective: To identify and describe the certainty of evidence of gynecology and obstetrics systematic reviews (SRs) using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach. Method: Database searches of SRs using GRADE, published between 1 January 2016 to 31 December 2020, in the 10 "gynecology and obstetrics" journals with the highest impact factor, according to the Journal Citation Report 2019. Selected studies included those SRs using the GRADE approach, used to determine the certainty of evidence. Results: Out of 952 SRs, ninety-six SRs of randomized control trials (RCTs) and/or nonrandomized studies (NRSs) used GRADE. Sixty-seven SRs (7.04%) rated the certainty of evidence for specific outcomes. In total, we identified 946 certainty of evidence outcome ratings (n = 614 RCT ratings), ranging from very-low (42.28%) to low (28.44%), moderate (17.65%), and high (11.63%). High and very low certainty of evidence ratings accounted for 2.16% and 71.60% in the SRs of NRSs, respectively, compared with 16.78% and 26.55% in the SRs of RCTs. In the SRs of RCTs and NRSs, certainty of evidence was mainly downgraded due to imprecision and bias risks. Conclusions: More attention needs to be paid to strengthening GRADE acceptance and building knowledge of GRADE methods in gynecology and obstetrics evidence synthesis.

Characteristics of the sources, evaluation, and grading of the certainty of evidence in systematic reviews in public health: A methodological study.

Objectives: To systematically explore how the sources of evidence, types of primary studies, and tools used to assess the quality of the primary studies vary across systematic reviews (SRs) in public health. Methods: We conducted a methodological survey of SRs in public health by searching the of literature in selected journals from electronic bibliographic databases. We selected a 10% random sample of the SRs that met the explicit inclusion criteria. Two researchers independently extracted data for analysis. Results: We selected 301 SRs for analysis: 94 (31.2%) of these were pre-registered, and 211 (70.1%) declared to have followed published reporting standard. All SRs searched for evidence in electronic bibliographic databases, and more than half (n = 180, 60.0%) searched also the references of the included studies. The common types of primary studies included in the SRs were primarily cross-sectional studies (n = 132, 43.8%), cohort studies (n = 126, 41.9%), randomized controlled trials (RCTs, n = 89, 29.6%), quasi-experimental studies (n = 83, 27.6%), case-control studies (n = 58, 19.3%) qualitative studies (n = 38, 12.6%) and mixed-methods studies (n = 32, 10.6%). The most frequently used quality assessment tools were the Newcastle-Ottawa Scale (used for 50.0% of cohort studies and 55.6% of case-control studies), Cochrane Collaboration's Risk of Bias tool (50.7% of RCTs) and Critical Appraisal Skills Program (38.5% of qualitative studies). Only 20 (6.6%) of the SRs assessed the certainty of the body of evidence, of which 19 (95.0%) used the GRADE approach. More than 65% of the evidence in the SRs using GRADE was of low or very low certainty. Conclusions: SRs should always assess the quality both at the individual study level and the body of evidence for outcomes, which will benefit patients, health care practitioners, and policymakers.

GRADE concept 4: rating the certainty of evidence when study interventions or comparators differ from PICO targets.

OBJECTIVES: This Grading of Recommendations Assessment, Development and Evaluation (GRADE) concept article offers systematic reviewers, guideline authors, and other users of evidence assistance in addressing randomized trial situations in which interventions or comparators differ from those in the target people, interventions, comparators, and outcomes. To clarify what GRADE considers under indirectness of interventions and comparators, we focus on a particular example: when comparator arm participants receive some or all aspects of the intervention management strategy (treatment switching). STUDY DESIGN AND SETTING: An interdisciplinary panel of the GRADE working group members developed this concept article through an iterative review of examples in multiple teleconferences, small group sessions, and e-mail correspondence. After presentation at a GRADE working group meeting in November 2022, attendees approved the final concept paper, which we support with examples from systematic reviews and individual trials. RESULTS: In the presence of safeguards against risk of bias, trials provide unbiased estimates of the effect of an intervention on the people as enrolled, the interventions as implemented, the comparators as implemented, and the outcomes as measured. Within the GRADE framework, differences in the people, interventions, comparators, and outcomes elements between the review or guideline recommendation targets and the trials as implemented constitute issues of indirectness. The intervention or comparator group management strategy as implemented, when it differs from the target comparator, constitutes one potential source of indirectness: Indirectness of interventions and comparators-comparator group receipt of the intervention constitutes a specific subcategory of said indirectness. The proportion of comparator arm participants that received the intervention and the apparent magnitude of effect bear on whether one should rate down, and if one does, to what extent. CONCLUSION: Treatment switching and other differences between review or guideline recommendation target interventions and comparators vs. interventions and comparators as implemented in otherwise relevant trials are best considered issues of indirectness.

Guidance to best tools and practices for systematic reviews.

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Use of GRADE in systematic reviews of health effects on pollutants and extreme temperatures: A cross-sectional survey.

OBJECTIVES: (i) To analyze trends and gaps in evidence of health effects on pollutants and extreme temperatures by evidence mapping; (ii) to conduct a cross-sectional survey on the use of the Grades of Recommendations Assessment Development and Evaluation (GRADE) in systematic reviews or meta-analyses (SR/MAs) of health effects on pollutants and extreme temperatures. STUDY DESIGN AND SETTING: PubMed, Embase, the Cochrane Library, Web of Science, and Cumulative Index to Nursing and Allied Health Literature (CINAHL) were searched until July 7, 2022. SR/MAs investigated health effects of pollutants and extreme temperatures were included. RESULTS: Out of 22,658 studies, 312 SR/MAs were included in evidence mapping, and the effects of pollutants on cancer and congenital malformations were new research hotspots. Among 16 SR/MAs involving 108 outcomes that were rated using GRADE, the certainty of evidence was mostly downgraded for inconsistency (50, 42.7%), imprecision (33, 28.2%), and risk of bias (24, 20.5%). In contrast, concentration-response gradient (26, 65.0%) was the main upgrade factor. CONCLUSION: GRADE is not widely used in SR/MAs of health effects on pollutants and extreme temperatures. The certainty of evidence is generally low, mainly because of the serious inconsistency or imprecision. Use of the GRADE in SR/MAs of health effects on pollutants and extreme temperatures should strengthen.