Certainty of Evidence Assessment in Systematic Reviews Published by High-Impact Sports Science Journals: A Meta-epidemiological Study.

BACKGROUND: Assessing certainty of evidence is a key element of any systematic review. The aim of this meta-epidemiology study was to understand the frequency and ways with which certainty of evidence is assessed in contemporary systematic reviews published in high-impact sports science journals. METHODS: We searched PubMed and relevant journal web sites from 1 August 2016 to 11 October 2022 for systematic reviews published in the top-ten highest-impact journals within the 2020 Journal Citation Report for the Sports Sciences category. Pairs of independent reviewers screened items using a priori established criteria. RESULTS: Of 1250 eligible documents, 258 (20.6%) assessed the certainty of evidence, defined as using two or more distinct domains to provide an overall rating of the trustworthiness of findings across studies. Nine methods were cited for assessing certainty, with the most common being the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach (61.6%). The proportion of systematic reviews assessing certainty of evidence appeared to increase over the 6-year timeframe analyzed. Across all reviews analyzed, a large majority addressed the domains of risk of bias, imprecision, and inconsistency of the results. Other certainty domains including indirectness/applicability were less commonly assessed. DISCUSSION: Only one in five recent contemporary systematic reviews in the field of exercise and sports science assessed certainty of evidence. Organizational and institutional education on methods for assessing evidence may help further increase uptake of these methods and improve both the quality and clinical impact of systematic reviews in the field.

Grading of recommendations assessment, development, and evaluation concept article 5: addressing intransitivity in a network meta-analysis.

OBJECTIVES: This article describes considerations for addressing intransitivity when assessing the certainty of the evidence from network meta-analysis (NMA) using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach. Intransitivity is induced by effect modification, that is, when the magnitude of the effect between an intervention and outcome differs depending on the level of another factor. STUDY DESIGN AND SETTING: To develop this GRADE concept paper, the lead authors conducted iterative discussions, computer simulations, and presentations to the GRADE project group and at GRADE working group meetings. The GRADE Working Group formally approved the article in July 2022. RESULTS: NMA authors can have a higher or a lower threshold to rate down the certainty of the evidence due to intransitivity, which depends on the extent of their concerns regarding the trustworthiness of indirect comparisons, and their view of the relative problems with rating down excessively or insufficiently. NMA authors should consider three main factors when addressing intransitivity: the credibility of effect modification, the strength of the effect modification, and the distribution of effect modifiers across the direct comparisons. To avoid double counting limitations of the evidence, authors should consider the relationship between intransitivity and other GRADE domains. CONCLUSION: NMA authors face theoretic and pragmatic challenges and in most situations need to assess intransitivity without the availability of empirical data. Thus, explicitness regarding perspective is crucial.

An International Needs Assessment Survey of Guideline Developers Demonstrates Variability in Resources and Challenges to Collaboration between Organizations.

BACKGROUND: The development of rigorous, high-quality clinical guidelines increases the need for resources and skilled personnel within guideline-producing organizations. While collaboration between organizations provides a unique opportunity to pool resources and save time and effort, the collaboration presents its own unique challenges. OBJECTIVE: To assess the perceived needs and current challenges of guideline producers worldwide related to guideline development and collaboration efforts. DESIGN: Survey questions were developed by the Guidelines International Network and the US GRADE Network, pilot-tested among attendees of a guideline development workshop, and disseminated electronically using convenience and snowball sampling methods. PARTICIPANTS: A total of 171 respondents representing 30 countries and more than 112 unique organizations were included in this analysis. MAIN MEASURES: The survey included free-response, multiple-choice, and seven-point Likert-scale questions. Questions assessed respondents' perceived value of guidelines, resource availability and needs, guideline development processes, and collaboration efforts of their organization. KEY RESULTS: Time required to develop high-quality systematic reviews and guidelines was the most relevant need (median=7; IQR=5.5-7). In-house resources to conduct literature searches (median=4; IQR=3-6) and the resources to develop rigorous guidelines rapidly (median=4; IQR=2-5) were perceived as the least available resources. Difficulties reconciling differences in guideline methodology (median=6; IQR=4-7) and the time required to establish collaborative agreements (median=6; IQR=5-6) were the most relevant barriers to collaboration between organizations. Results also indicated a general need for improvement in conflict of interest (COI) disclosure policies. CONCLUSION: The survey identified organizational challenges in supporting rigorous guideline development, including the time, resources, and personnel required. Connecting guideline developers to existing databases of high-quality systematic reviews and the use of freely available online platforms may facilitate guideline development. Guideline-producing organizations may also consider allocating resources to hiring or training personnel with expertise in systematic review methodologies or utilizing resources more effectively by establishing collaborations with other organizations.

Cost-effectiveness of diagnostic strategies for venous thromboembolism: a systematic review.

Guideline developers consider cost-effectiveness evidence in decision making to determine value for money. This consideration in the guideline development process can be informed either by formal and dedicated economic evaluations or by systematic reviews of existing studies. To inform the American Society of Hematology guideline on the diagnosis of venous thromboembolism (VTE), we conducted a systematic review focused on the cost-effectiveness of diagnostic strategies for VTE within the guideline scope. We systematically searched Medline (Ovid), Embase (Ovid), National Health Service Economic Evaluation Database, and the Cost-effectiveness Analysis Registry; summarized; and critically appraised the economic evidence on diagnostic strategies for VTE. We identified 49 studies that met our inclusion criteria, with 26 on pulmonary embolism (PE) and 24 on deep vein thrombosis (DVT). For the diagnosis of PE, strategies including d-dimer to exclude PE were cost-effective compared with strategies without d-dimer testing. The cost-effectiveness of computed tomography pulmonary angiogram (CTPA) in relation to ventilation-perfusion (V/Q) scan was inconclusive. CTPA or V/Q scan following ultrasound or d-dimer results could be cost-effective or even cost saving. For DVT, studies supporting strategies with d-dimer and/or ultrasound were cost-effective, supporting the recommendation that for patients at low (unlikely) VTE risk, using d-dimer as the initial test reduces the need for diagnostic imaging. Our systematic review informed the American Society of Hematology guideline recommendations about d-dimer, V/Q scan and CTPA for PE diagnosis, and d-dimer and ultrasound for DVT diagnosis.

Bleeding assessment tools in the diagnosis of VWD in adults and children: a systematic review and meta-analysis of test accuracy.

Von Willebrand disease (VWD) can be associated with significant morbidity. Patients with VWD can experience bruising, mucocutaneous bleeding, and bleeding after dental and surgical procedures. Early diagnosis and treatment are important to minimize the risk of these complications. Several bleeding assessment tools (BATs) have been used to quantify bleeding symptoms as a screening tool for VWD. We systematically reviewed diagnostic test accuracy results of BATs to screen patients for VWD. We searched Cochrane Central, MEDLINE, and EMBASE for eligible studies, reference lists of relevant reviews, registered trials, and relevant conference proceedings. Two investigators screened and abstracted data. Risk of bias was assessed using the revised tool for the quality assessment of diagnostic accuracy studies and certainty of evidence using the Grading of Recommendations Assessment, Development and Evaluation framework. We pooled estimates of sensitivity and specificity. The review included 7 cohort studies that evaluated the use of BATs to screen adult and pediatric patients for VWD. The pooled estimates for sensitivity and specificity were 75% (95% confidence interval, 66-83) and 54% (29-77), respectively. Certainty of evidence varied from moderate to high. This systematic review provides accuracy estimates for validated BATs as a screening modality for VWD. A BAT is a useful initial screening test to determine who needs specific blood testing. The pretest probability of VWD (often determined by the clinical setting/patient population), along with sensitivity and specificity estimates, will influence patient management.

GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy.

OBJECTIVES: This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can assess the results and the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA). STUDY DESIGN AND SETTING: We present an overview of the GRADE approach and guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests. Part 1 of the two parts in this 21st guidance article about how to apply GRADE focuses on understanding study design issues in test accuracy, provide an overview of the domains, and describe risk of bias and indirectness specifically. RESULTS: Supplemented by practical examples, we describe how raters of the evidence using GRADE can evaluate study designs focusing on tests and how they apply the GRADE domains risk of bias and indirectness to a body of evidence of TA studies. CONCLUSION: Rating the certainty of a body of evidence using GRADE in Cochrane and other reviews and World Health Organization and other guidelines dealing with in TA studies helped refining our approach. The resulting guidance will help applying GRADE successfully for questions and recommendations focusing on tests.

Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation.

BACKGROUND: Stakeholder engagement has become widely accepted as a necessary component of guideline development and implementation. While frameworks for developing guidelines express the need for those potentially affected by guideline recommendations to be involved in their development, there is a lack of consensus on how this should be done in practice. Further, there is a lack of guidance on how to equitably and meaningfully engage multiple stakeholders. We aim to develop guidance for the meaningful and equitable engagement of multiple stakeholders in guideline development and implementation. METHODS: This will be a multi-stage project. The first stage is to conduct a series of four systematic reviews. These will (1) describe existing guidance and methods for stakeholder engagement in guideline development and implementation, (2) characterize barriers and facilitators to stakeholder engagement in guideline development and implementation, (3) explore the impact of stakeholder engagement on guideline development and implementation, and (4) identify issues related to conflicts of interest when engaging multiple stakeholders in guideline development and implementation. DISCUSSION: We will collaborate with our multiple and diverse stakeholders to develop guidance for multi-stakeholder engagement in guideline development and implementation. We will use the results of the systematic reviews to develop a candidate list of draft guidance recommendations and will seek broad feedback on the draft guidance via an online survey of guideline developers and external stakeholders. An invited group of representatives from all stakeholder groups will discuss the results of the survey at a consensus meeting which will inform the development of the final guidance papers. Our overall goal is to improve the development of guidelines through meaningful and equitable multi-stakeholder engagement, and subsequently to improve health outcomes and reduce inequities in health.

GRADE guidelines: 21 part 2. Test accuracy: inconsistency, imprecision, publication bias, and other domains for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables.

OBJECTIVES: This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can rate the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA) on the domains imprecision, inconsistency, publication bias, and other domains. It also provides guidance for how to present synthesized information in evidence profiles and summary of findings tables. STUDY DESIGN AND SETTING: We present guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests. RESULTS: Supplemented by practical examples, we describe how raters of the evidence can apply the GRADE domains inconsistency, imprecision, and publication bias to a body of evidence of TA studies. CONCLUSION: Using GRADE in Cochrane and other reviews as well as World Health Organization and other guidelines helped refining the GRADE approach for rating the certainty of a body of evidence from TA studies. Although several of the GRADE domains (e.g., imprecision and magnitude of the association) require further methodological research to help operationalize them, judgments need to be made on the basis of what is known so far.

Defining ranges for certainty ratings of diagnostic accuracy: a GRADE concept paper.

OBJECTIVE: The objective of the study was to clarify how the Grading of Recommendations Assessment, Development and Evaluation (GRADE) concept of certainty of evidence applies to certainty ratings of test accuracy. STUDY DESIGN AND SETTING: After initial brainstorming with GRADE Working Group members, we iteratively refined and clarified the approaches for defining ranges when assessing the certainty of evidence for test accuracy within a systematic review, health technology assessment, or guideline. RESULTS: Ranges can be defined both for single test accuracy and for comparative accuracy of multiple tests. For systematic reviews and health technology assessments, approaches for defining ranges include some that do not require value judgments regarding downstream health outcomes. Key challenges arise in the context of a guideline that requires ranges for sensitivity and specificity that are set considering possible effects on all critical outcomes. We illustrate possible approaches and provide an example from a systematic review of a direct comparison between two test strategies. CONCLUSIONS: This GRADE concept paper provides a framework for assessing, presenting, and making decisions based on the certainty of evidence for test accuracy. More empirical research is needed to support future GRADE guidance on how to best operationalize the candidate approaches.

Grading of Recommendations Assessment, Development, and Evaluations (GRADE) notes: extremely serious, GRADE's terminology for rating down by three levels.

OBJECTIVES: The Grading of Recommendations Assessment, Development, and Evaluations (GRADE) system for assessing certainty in a body of evidence currently uses two levels, serious and very serious, for downgrading on a single domain. In the context of newer risk of bias instruments, such as Risk of Bias in Non-Randomized Studies I (ROBINS-I), evidence generated by nonrandomized studies may justify rating down by more than two levels on a single domain. Given the importance users of GRADE assign to terminology, our objective was to assess what term GRADE stakeholders would prefer for rating down certainty by three levels. STUDY DESIGN AND SETTING: We conducted a purposefully sampled online survey of GRADE stakeholders to assess possible terms including "critically serious," "extremely serious," "most serious," and "very, very serious" and conducted a descriptive and thematic analysis of responses. We then facilitated a GRADE working group workshop to generate consensus. RESULTS: A total of 225 respondents ranked and rated "extremely serious" highest, closely followed by "critically serious." Respondents felt that "extremely serious" was "more understandable" and "easiest to interpret". GRADE working group members described that the terms "extremely serious" appeared clearer and easier to translate in other languages. CONCLUSION: Based on this stakeholder-driven study, "extremely serious" is the preferred term to rate down certainty of evidence by three levels in the GRADE approach.

GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.

OBJECTIVE: To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process. STUDY DESIGN AND SETTING: The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group. RESULTS: We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I. CONCLUSIONS: The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

[GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines]. / Marcos GRADE de la evidencia a la decisión (EtD): un enfoque sistemático y transparente para tomar decisiones sanitarias bien informadas. 2: Guías de práctica clínica.

Clinicians do not have the time or resources to consider the underlying evidence for the myriad decisions they must make each day and, as a consequence, rely on recommendations from clinical practice guidelines. Guideline panels should consider all the relevant factors (criteria) that influence a decision or recommendation in a structured, explicit, and transparent way and provide clinicians with clear and actionable recommendations. In this article, we will describe the Evidence to Decision (EtD) frameworks for clinical practice recommendations. The general structure of the EtD framework for clinical recommendations is similar to EtD frameworks for other types of recommendations and decisions, and includes formulation of the question, an assessment of the different criteria, and conclusions. Clinical recommendations require considering criteria differently, depending on whether an individual patient or a population perspective is taken. For example, from an individual patient's perspective, out-of-pocket costs are an important consideration, whereas, from a population perspective, resource use (not only out-of-pocket costs) and cost effectiveness are important. From a population perspective, equity, acceptability, and feasibility are also important considerations, whereas the importance of these criteria is often limited from an individual patient perspective. Specific subgroups for which different recommendations may be required should be clearly identified and considered in relation to each criterion because judgments might vary across subgroups. This article is a translation of the original article published in the British Medical Journal. The EtD frameworks are currently used in the Clinical Practice Guideline Programme of the Spanish National Health System, co-ordinated by GuíaSalud.

The GRADE Working Group clarifies the construct of certainty of evidence.

OBJECTIVE: To clarify the grading of recommendations assessment, development and evaluation (GRADE) definition of certainty of evidence and suggest possible approaches to rating certainty of the evidence for systematic reviews, health technology assessments, and guidelines. STUDY DESIGN AND SETTING: This work was carried out by a project group within the GRADE Working Group, through brainstorming and iterative refinement of ideas, using input from workshops, presentations, and discussions at GRADE Working Group meetings to produce this document, which constitutes official GRADE guidance. RESULTS: Certainty of evidence is best considered as the certainty that a true effect lies on one side of a specified threshold or within a chosen range. We define possible approaches for choosing threshold or range. For guidelines, what we call a fully contextualized approach requires simultaneously considering all critical outcomes and their relative value. Less-contextualized approaches, more appropriate for systematic reviews and health technology assessments, include using specified ranges of magnitude of effect, for example, ranges of what we might consider no effect, trivial, small, moderate, or large effects. CONCLUSION: It is desirable for systematic review authors, guideline panelists, and health technology assessors to specify the threshold or ranges they are using when rating the certainty in evidence.

Stakeholders apply the GRADE evidence-to-decision framework to facilitate coverage decisions.

OBJECTIVES: Coverage decisions are complex and require the consideration of many factors. A well-defined, transparent process could improve decision-making and facilitate decision-maker accountability. STUDY DESIGN AND SETTING: We surveyed key US-based stakeholders regarding their current approaches for coverage decisions. Then, we held a workshop to test an evidence-to-decision (EtD) framework for coverage based on the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) criteria. RESULTS: A total of 42 individuals (including 19 US stakeholders as well as international health policymakers and GRADE working group members) attended the workshop. Of the 19 stakeholders, 14 (74%) completed the survey before the workshop. Almost all of their organizations (13 of 14; 93%) used systematic reviews for coverage decision-making; few (2 of 14; 14%) developed their own evidence synthesis; a majority (9 of 14; 64%) rated the certainty of evidence (using various systems); almost all (13 of 14; 93%) denied formal consideration of resource use; and half (7 of 14; 50%) reported explicit criteria for decision-making. At the workshop, stakeholders successfully applied the EtD framework to four case studies and provided narrative feedback, which centered on contextual factors affecting coverage decisions in the United States, the need for reliable data on subgroups of patients, and the challenge of decision-making without formal consideration of resource use. CONCLUSION: Stakeholders successfully applied the EtD framework to four case studies and highlighted contextual factors affecting coverage decisions and affirmed its value. Their input informed the further development of a revised EtD framework, now publicly available (http://gradepro.org/).

A guide for health professionals to interpret and use recommendations in guidelines developed with the GRADE approach.

An increasing number of organizations worldwide are using new and improved standards for developing trustworthy clinical guidelines. One of such approaches, developed by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group, offers systematic and transparent guidance in moving from evidence to recommendations. The GRADE strategy concentrates on four factors: the balance between benefits and harms, the certainty of the evidence, values and preferences, and resource considerations. However, it also considers issues around feasibility, equity, and acceptability of recommendations. GRADE distinguishes two types of recommendations: strong and weak. Strong recommendations reflect a clear preference for one alternative and should apply to all or almost all patients, obviating the need for a careful review of the evidence with each patient. Weak recommendations are appropriate when there is a close balance between desirable and undesirable consequences of alternative management strategies, uncertainty regarding the effects of the alternatives, uncertainty or variability in patients' values and preferences, or questionable cost-effectiveness. Weak recommendations usually require accessing the underlying evidence and a shared decision-making approach. Clinicians using GRADE recommendations should understand the meaning of the strength of the recommendation, be able to critically appraise the recommendation, and apply trustworthy recommendations according to their strength.

Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments.

BACKGROUND: The Grading of Recommendations Assessment, Development and Evaluation (GRADE) is widely used and reliable and accurate for assessing the certainty in the body of health evidence. The GRADE working group has provided detailed guidance for assessing the certainty in the body of evidence in systematic reviews and health technology assessments (HTAs) and how to grade the strength of health recommendations. However, there is limited advice regarding how to maximize transparency of these judgments, in particular through explanatory footnotes or explanations in Summary of Findings tables and Evidence Profiles (GRADE evidence tables). METHODS: We conducted this study to define the essential attributes of useful explanations and to develop specific guidance for explanations associated with GRADE evidence tables. We used a sample of explanations according to their complexity, type of judgment involved, and appropriateness from a database of published GRADE evidence tables in Cochrane reviews and World Health Organization guidelines. We used an iterative process and group consensus to determine the attributes and develop guidance. RESULTS: Explanations in GRADE evidence tables should be concise, informative, relevant, easy to understand, and accurate. We provide general and domain-specific guidance to assist authors with achieving these desirable attributes in their explanations associated with GRADE evidence tables. CONCLUSIONS: Adhering to the general and GRADE domain-specific guidance should improve the quality of explanations associated with GRADE evidence tables, assist authors of systematic reviews, HTA reports, or guidelines with information that they can use in other parts of their evidence synthesis. This guidance will also support editorial evaluation of evidence syntheses using GRADE and provide a minimum quality standard of judgments across tables.

Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed.

OBJECTIVES: The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group has developed GRADE evidence profiles (EP) and summary of findings (SoF) tables to present evidence summaries in systematic reviews, clinical guidelines, and health technology assessments. Explanatory notes are used to explain choices and judgments in these summaries, for example, on rating of the quality of evidence. STUDY DESIGN AND SETTING: A systematic survey of the explanations in SoF tables in 132 randomly selected Cochrane Intervention reviews and in EPs of 10 guidelines. We analyzed the content of 1,291 explanations using a predefined list of criteria. RESULTS: Most explanations were used to describe or communicate results and to explain downgrading of the quality of evidence, in particular for risk of bias and imprecision. Addressing the source of baseline risk (observational data or control group risk) was often missing. For judgments about downgrading the quality of evidence, the percentage of informative explanations ranged between 41% (imprecision) and 79% (indirectness). CONCLUSION: We found that by and large explanations were informative but detected several areas for improvement (e.g., source of baseline risk and judgments on imprecision). Guidance about explanatory footnotes and comments will be provided in the last article in this series.

Systematic reviews experience major limitations in reporting absolute effects.

OBJECTIVES: Expressing treatment effects in relative terms yields larger numbers than expressions in absolute terms, affecting the judgment of the clinicians and patients regarding the treatment options. It is uncertain how authors of systematic reviews (SRs) absolute effect estimates are reported in. We therefore undertook a systematic survey to identify and describe the reporting and methods for calculating absolute effect estimates in SRs. STUDY DESIGN AND SETTING: Two reviewers independently screened title, abstract, and full text and extracted data from a sample of Cochrane and non-Cochrane SRs. We used regression analyses to examine the association between study characteristics and the reporting of absolute estimates for the most patient-important outcome. RESULTS: We included 202 SRs (98 Cochrane and 104 non-Cochrane), most of which (92.1%) included standard meta-analyses including relative estimates of effect. Of the 202 SRs, 73 (36.1%) reported absolute effect estimates for the most patient-important outcome. SRs with statistically significant effects were more likely to report absolute estimates (odds ratio, 2.26; 95% confidence interval: 1.08, 4.74). The most commonly reported absolute estimates were: for each intervention, risk of adverse outcomes expressed as a percentage (41.1%); number needed to treat (26.0%); and risk for each intervention expressed as natural units or natural frequencies (24.7%). In 12.3% of the SRs that reported absolute effect estimates for both benefit and harm outcomes, harm outcomes were reported exclusively as absolute estimates. Exclusively reporting of beneficial outcomes as absolute estimates occurred in 6.8% of the SRs. CONCLUSIONS: Most SRs do not report absolute effects. Those that do often report them inadequately, thus requiring users of SRs to generate their own estimates of absolute effects. For any apparently effective or harmful intervention, SR authors should report both absolute and relative estimates to optimize the interpretation of their findings.

Applying Grading of Recommendations Assessment, Development and Evaluation (GRADE) to diagnostic tests was challenging but doable.

OBJECTIVES: The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group developed an approach to assess the quality of evidence of diagnostic tests. Its use in Cochrane diagnostic test accuracy reviews is new. We applied this approach to three Cochrane reviews with the aim of better understanding the application of the GRADE criteria to such reviews. STUDY DESIGN AND SETTING: We selected reviews to achieve clinical and methodological diversities. At least three assessors independently assessed each review according to the GRADE criteria of risk of bias, indirectness, imprecision, inconsistency, and publication bias. Two teleconferences were held to share experiences. RESULTS: For the interpretation of the GRADE criteria, it made a difference whether assessors looked at the evidence from a patient-important outcome perspective or from a test accuracy standpoint. GRADE criteria such as inconsistency, imprecision, and publication bias were challenging to apply as was the assessment of comparative test accuracy reviews. CONCLUSION: The perspective from which evidence is graded can influence judgments about quality. Guidance on application of GRADE to comparative test reviews and on the GRADE criteria of inconsistency, imprecision, and publication bias will facilitate the operationalization of GRADE for diagnostics.