Strong recommendations from low certainty evidence: a cross-sectional analysis of a suite of national guidelines.

BACKGROUND: Clinical guidelines should be based on a thorough evaluation of the evidence and generally include a rating of the quality of evidence and assign a strength to recommendations. Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance warns against making strong recommendations when the certainty of the evidence is low or very low, but has identified five paradigmatic situations (e.g. life-threatening situations) where this may be justified. AIMS AND OBJECTIVES: We aimed to characterize the strength of recommendations and certainty of the evidence in Irish National Clinical Guidelines using the GRADE approach. METHODS: All National Clinical Guidelines from the National Clinical Effectiveness Committee (NCEC) website using the GRADE approach (fully or partially) were included. All recommendations and their corresponding certainty of the evidence, strength of recommendations and justifications were extracted. Authors classified instances of strong recommendations with low certainty evidence (referred to as discordant recommendations) into one of the five paradigmatic situations. Descriptive statistics were calculated. RESULTS: From the 29 NCEC Clinical Guidelines available at the time of analysis, we identified 8 guidelines using GRADE with a total of 240 recommendations; 38 recommendations did not use the GRADE approach and were excluded. Half of the included guidelines focused on emergency situations. In the final dataset of 202 recommendations, 151 (74.7%) were classified as strong and 51 (25.3%) as conditional. Of the 151 strong recommendations, 55 (36.4%) were supported by high or moderate certainty evidence and 96 (63.6%) by low or very low certainty evidence and were considered discordant. Of these 96 discordant recommendations, 55 (73.7%) were consistent with one of the five paradigmatic situations. However, none were specifically described as such within the guidelines. CONCLUSIONS: The proportion of discordant recommendations identified in this analysis was higher than some previous international studies (range of all strong recommendations being discordant 30-50%), but similar to other guidelines focused on emergency situations. The majority of discordant recommendations could be mapped to one of the five situations, but no National Clinical Guideline explicitly referenced this. Guideline developers require further guidance to enable greater transparency in the reporting of the reasons for discordant recommendations.

Co-design workshops to develop evidence synthesis summary formats for use by clinical guideline development groups.

BACKGROUND: Evidence synthesis is used by decision-makers in various ways, such as developing evidence-based recommendations for clinical guidelines. Clinical guideline development groups (GDGs) typically discuss evidence synthesis findings in a multidisciplinary group, including patients, healthcare providers, policymakers, etc. A recent mixed methods systematic review (MMSR) identified no gold standard format for optimally presenting evidence synthesis findings to these groups. However, it provided 94 recommendations to help produce more effective summary formats for general evidence syntheses (e.g., systematic reviews). To refine the MMSR recommendations to create more actionable guidance for summary producers, we aimed to explore these 94 recommendations with participants involved in evidence synthesis and guideline development. METHODS: We conducted a descriptive qualitative study using online focus group workshops in February and March 2023. These groups used a participatory co-design approach with interactive voting activities to identify preferences for a summary format's essential content and style. We created a topic guide focused on recommendations from the MMSR with mixed methods support, ≥ 3 supporting studies, and those prioritized by an expert advisory group via a pragmatic prioritization exercise using the MoSCoW method (Must, Should, Could, and Will not haves). Eligible participants must be/have been involved in GDGs and/or evidence synthesis. Groups were recorded and transcribed. Two independent researchers analyzed transcripts using directed content analysis with 94 pre-defined codes from the MMSR. RESULTS: Thirty individuals participated in six focus groups. We coded 79 of the 94 pre-defined codes. Participants suggested a "less is more" structured approach that minimizes methodological steps and statistical data, promoting accessibility to all audiences by judicious use of links to further information in the full report. They emphasized concise, consistently presented formats that highlight key messages, flag readers to indicators of trust in the producers (i.e., logos, websites, and conflict of interest statements), and highlight the certainty of evidence (without extenuating details). CONCLUSIONS: This study identified guidance based on the preferences of guideline developers and evidence synthesis producers about the format of evidence synthesis summaries to support decision-making. The next steps involve developing and user-testing prototype formats through one-on-one semi-structured interviews to optimize evidence synthesis summaries and support decision-making.

An international modified Delphi process supported updating the web-based "right review" tool.

OBJECTIVES: The proliferation of evidence synthesis methods makes it challenging for reviewers to select the ''right'' method. This study aimed to update the Right Review tool (a web-based decision support tool that guides users through a series of questions for recommending evidence synthesis methods) and establish a common set of questions for the synthesis of both quantitative and qualitative studies (https://rightreview.knowledgetranslation.net/). STUDY DESIGN AND SETTING: A 2-round modified international electronic modified Delphi was conducted (2022) with researchers, health-care providers, patients, and policy makers. Panel members rated the importance/clarity of the Right Review tool's guiding questions, evidence synthesis type definitions and tool output. High agreement was defined as at least 70% agreement. Any items not reaching high agreement after round 2 were discussed by the international Project Steering Group. RESULTS: Twenty-four experts from 9 countries completed round 1, with 12 completing round 2. Of the 46 items presented in round 1, 21 reached high agreement. Twenty-seven items were presented in round 2, with 8 reaching high agreement. The Project Steering Group discussed items not reaching high agreement, including 8 guiding questions, 9 review definitions (predominantly related to qualitative synthesis), and 2 output items. Three items were removed entirely and the remaining 16 revised and edited and/or combined with existing items. The final tool comprises 42 items; 9 guiding questions, 25 evidence synthesis definitions and approaches, and 8 tool outputs. CONCLUSION: The freely accessible Right Review tool supports choosing an appropriate review method. The design and clarity of this tool was enhanced by harnessing the Delphi technique to shape ongoing development. The updated tool is expected to be available in Quarter 1, 2025.

Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol.

Introduction: The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) evidence to decision (EtD) framework provides a structured and transparent approach for clinical guideline developers to use when formulating recommendations. Understanding how stakeholders use the EtD framework will inform how best to provide future training and support. This scoping review objective is to identify the key characteristics of how the GRADE EtD framework is used and identify studies on perception of use by those involved in developing clinical guidelines. Methods: JBI methodology for scoping reviews will be followed. This scoping review will consider both peer review published literature and grey literature. This will include empirical studies on the use of EtDs (including both quantitative, qualitative, and mixed methods primary research articles) and discussion papers/ commentaries on the experience of using the EtD. It will also include a random sample of publicly available populated EtDs identified from databases and repositories of GRADE guidelines. The search strategy will aim to locate both published and unpublished documents. First, we will conduct an exploratory search of MEDLINE and Embase (Elsevier), supplemented with citation analysis of included articles. Populated EtDs will be identified through searches of databases and repositories of GRADE guidelines. Two researchers will independently screen, select, and extract identified documents. Data will be presented in tables and summarized descriptively. Conclusion: This scoping review will identify the key characteristics of how the GRADE EtD framework is currently being used in clinical guidelines. Review findings can be used to inform future guidance and requirements for using GRADE EtD, as well as training on how to consider the criteria in developing recommendations. Results will be disseminated through publications in peer - reviewed journals and conference presentations. We will present our findings to relevant stakeholders via the networks of the co-author team at a one-day workshop.