Informing pandemic management in Germany with trustworthy living evidence syntheses and guideline development: lessons learned from the COVID-19 evidence ecosystem.

OBJECTIVES: We present the 'COVID-19 evidence ecosystem' (CEOsys) as a German network to inform pandemic management and to support clinical and public health decision-making. We discuss challenges faced when organizing the ecosystem and derive lessons learned for similar networks acting during pandemics or health-related crises. STUDY DESIGN AND SETTING: Bringing together 18 university hospitals and additional institutions, CEOsys key activities included research prioritization, conducting living systematic reviews (LSRs), supporting evidence-based (living) guidelines, knowledge translation (KT), detecting research gaps, and deriving recommendations, backed by technical infrastructure and capacity building. RESULTS: CEOsys rapidly produced 31 high-quality evidence syntheses and supported three living guidelines on COVID-19-related topics, while also developing methodological procedures. Challenges included CEOsys' late initiation in relation to the pandemic outbreak, the delayed prioritization of research questions, the continuously evolving COVID-19-related evidence, and establishing a technical infrastructure. Methodological-clinical tandems, the cooperation with national guideline groups and international collaborations were key for efficiency. CONCLUSION: CEOsys provided a proof-of-concept for a functioning evidence ecosystem at the national level. Lessons learned include that similar networks should, among others, involve methodological and clinical key stakeholders early on, aim for (inter)national collaborations, and systematically evaluate their value. We particularly call for a sustainable network.

Pharmacological interventions for preventing upper gastrointestinal bleeding in people admitted to intensive care units: a network meta-analysis.

OBJECTIVES: To assess the efficacy and safety of pharmacological interventions for preventing upper gastrointestinal (GI) bleeding in people admitted to intensive care units (ICUs). DESIGN AND SETTING: Systematic review and frequentist network meta-analysis using standard methodological procedures as recommended by Cochrane for screening of records, data extraction and analysis. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach was used to assess the certainty of evidence. PARTICIPANTS: Randomised controlled trials involving patients admitted to ICUs for longer than 24 hours were included. SEARCH METHODS: The Cochrane Gut Specialised Register, Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase and Latin American and Caribbean Health Science Information database (LILACS) databases were searched from August 2017 to March 2022. The search in MEDLINE was updated in April 2023. We also searched ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform (WHO ICTRP). MAIN OUTCOME MEASURES: The primary outcome was the prevention of clinically important upper GI bleeding. RESULTS: We included 123 studies with 46 996 participants. Cimetidine (relative risk (RR) 0.56, 95% CI 0.40 to 0.77, moderate certainty), ranitidine (RR 0.54, 95% CI 0.38 to 0.76, moderate certainty), antacids (RR 0.48, 95% CI 0.33 to 0.68, moderate certainty), sucralfate (RR 0.54, 95% CI 0.39 to 0.75, moderate certainty) and a combination of ranitidine and antacids (RR 0.13, 95% CI 0.03 to 0.62, moderate certainty) are likely effective in preventing upper GI bleeding.The effect of any intervention on the prevention of nosocomial pneumonia, all-cause mortality in the ICU or the hospital, duration of the stay in the ICU, duration of intubation and (serious) adverse events remains unclear. CONCLUSIONS: Several interventions seem effective in preventing clinically important upper GI bleeding while there is limited evidence for other outcomes. Patient-relevant benefits and harms need to be assessed under consideration of the patients' underlying conditions.

Robot-assisted versus conventional laparoscopic surgery for rectal cancer.

OBJECTIVES: This is a protocol for a Cochrane Review (intervention). The objectives are as follows: To assess the benefits and harms of robot-assisted surgery for rectal cancer resection.

Sensitivity and Specificity of Using GPT-3.5 Turbo Models for Title and Abstract Screening in Systematic Reviews and Meta-analyses.

BACKGROUND: Systematic reviews are performed manually despite the exponential growth of scientific literature. OBJECTIVE: To investigate the sensitivity and specificity of GPT-3.5 Turbo, from OpenAI, as a single reviewer, for title and abstract screening in systematic reviews. DESIGN: Diagnostic test accuracy study. SETTING: Unannotated bibliographic databases from 5 systematic reviews representing 22 665 citations. PARTICIPANTS: None. MEASUREMENTS: A generic prompt framework to instruct GPT to perform title and abstract screening was designed. The output of the model was compared with decisions from authors under 2 rules. The first rule balanced sensitivity and specificity, for example, to act as a second reviewer. The second rule optimized sensitivity, for example, to reduce the number of citations to be manually screened. RESULTS: Under the balanced rule, sensitivities ranged from 81.1% to 96.5% and specificities ranged from 25.8% to 80.4%. Across all reviews, GPT identified 7 of 708 citations (1%) missed by humans that should have been included after full-text screening at the cost of 10 279 of 22 665 false-positive recommendations (45.3%) that would require reconciliation during the screening process. Under the sensitive rule, sensitivities ranged from 94.6% to 99.8% and specificities ranged from 2.2% to 46.6%. Limiting manual screening to citations not ruled out by GPT could reduce the number of citations to screen from 127 of 6334 (2%) to 1851 of 4077 (45.4%), at the cost of missing from 0 to 1 of 26 citations (3.8%) at the full-text level. LIMITATIONS: Time needed to fine-tune prompt. Retrospective nature of the study, convenient sample of 5 systematic reviews, and GPT performance sensitive to prompt development and time. CONCLUSION: The GPT-3.5 Turbo model may be used as a second reviewer for title and abstract screening, at the cost of additional work to reconcile added false positives. It also showed potential to reduce the number of citations before screening by humans, at the cost of missing some citations at the full-text level. PRIMARY FUNDING SOURCE: None.

Accuracy of Event Rate and Effect Size Estimation in Major Cardiovascular Trials: A Systematic Review.

Importance: For the design of a randomized clinical trial (RCT), estimation of the expected event rate and effect size of an intervention is needed to calculate the sample size. Overestimation may lead to an underpowered trial. Objective: To evaluate the accuracy of published estimates of event rate and effect size in contemporary cardiovascular RCTs. Evidence Review: A systematic search was conducted in MEDLINE for multicenter cardiovascular RCTs associated with MeSH (Medical Subject Headings) terms for cardiovascular diseases published in the New England Journal of Medicine, JAMA, or the Lancet between January 1, 2010, and December 31, 2019. Identified trials underwent abstract review; eligible trials then underwent full review, and those with insufficiently reported data were excluded. Data were extracted from the original publication or the study protocol, and a random-effects model was used for data pooling. This review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guideline. The primary outcome was the accuracy of event rate and effect size estimation. Accuracy was determined by comparing the observed event rate in the control group and the effect size with their hypothesized values. Linear regression was used to determine the association between estimation accuracy and trial characteristics. Findings: Of the 873 RCTs identified, 374 underwent full review and 30 were subsequently excluded, resulting in 344 trials for analysis. The median observed event rate was 9.0% (IQR, 4.3% to 21.4%), which was significantly lower than the estimated event rate of 11.0% (IQR, 6.0% to 25.0%) with a median deviation of -12.3% (95% CI, -16.4% to -5.6%; P < .001). More than half of the trials (196 [61.1%]) overestimated the expected event rate. Accuracy of event rate estimation was associated with a higher likelihood of refuting the null hypothesis (0.13 [95% CI, 0.01 to 0.25]; P = .03). The median observed effect size in superiority trials was 0.91 (IQR, 0.74 to 0.99), which was significantly lower than the estimated effect size of 0.72 (IQR, 0.60 to 0.80), indicating a median overestimation of 23.1% (95% CI, 17.9% to 28.3%). A total of 216 trials (82.1%) overestimated the effect size. Conclusions and Relevance: In this systematic review of contemporary cardiovascular RCTs, event rates of the primary end point and effect sizes of an intervention were frequently overestimated. This overestimation may have contributed to the inability to adequately test the trial hypothesis.

A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E).

BACKGROUND: Observational epidemiologic studies provide critical data for the evaluation of the potential effects of environmental, occupational and behavioural exposures on human health. Systematic reviews of these studies play a key role in informing policy and practice. Systematic reviews should incorporate assessments of the risk of bias in results of the included studies. OBJECTIVE: To develop a new tool, Risk Of Bias In Non-randomized Studies - of Exposures (ROBINS-E) to assess risk of bias in estimates from cohort studies of the causal effect of an exposure on an outcome. METHODS AND RESULTS: ROBINS-E was developed by a large group of researchers from diverse research and public health disciplines through a series of working groups, in-person meetings and pilot testing phases. The tool aims to assess the risk of bias in a specific result (exposure effect estimate) from an individual observational study that examines the effect of an exposure on an outcome. A series of preliminary considerations informs the core ROBINS-E assessment, including details of the result being assessed and the causal effect being estimated. The assessment addresses bias within seven domains, through a series of 'signalling questions'. Domain-level judgements about risk of bias are derived from the answers to these questions, then combined to produce an overall risk of bias judgement for the result, together with judgements about the direction of bias. CONCLUSION: ROBINS-E provides a standardized framework for examining potential biases in results from cohort studies. Future work will produce variants of the tool for other epidemiologic study designs (e.g. case-control studies). We believe that ROBINS-E represents an important development in the integration of exposure assessment, evidence synthesis and causal inference.

Needs and feasibility of living systematic reviews (LSRs): Experience from LSRs on COVID-19 vaccine effectiveness.

During 2021 and 2023, a team of researchers at the Robert Koch Institute (RKI) and partnering institutions conducted two living systematic reviews (LSRs) on the effectiveness of COVID-19 vaccines in different age groups to inform recommendations of the Standing Committee on Vaccination in Germany (Ständige Impfkommission, STIKO). Based on our experience from the realization of these LSRs, we developed certain criteria to assess the needs and feasibility of conducting LSRs. Combining these with previously established criteria, we developed the following set to inform future planning of LSRs for STIKO: Needs criterion (N)1: Relevance of the research question, N2: Certainty of evidence (CoE) at baseline; N3: Expected need for Population-Intervention-Comparator-Outcome (PICO) adaptations; N4: Expected new evidence over time; N5: Expected impact of new evidence on CoE; Feasibility criterion (F)1: Availability of sufficient human resources; F2: Feasibility of timely dissemination of the results to inform decision-making. For each criterion we suggest rating options which may support the decision to conduct an LSR or other forms of evidence synthesis when following the provided flowchart. The suggested criteria were developed on the basis of the experiences from exemplary reviews in a specific research field (i.e., COVID-19 vaccination), and did not follow a formal development or validation process. However, these criteria might also be useful to assess whether questions from other research fields can and should be answered using the LSR approach, or assist in determining whether the use of an LSR is sensible and feasible for specific questions in health policy and practice.

[Development of criteria for the prospective assessment of the need for updating guideline recommendations: The AGIL criteria]. / Entwicklung von Kriterien für die prospektive Einschätzung des Aktualisierungsbedarfs von Leitlinienempfehlungen: AGIL-Kriterien.

BACKGROUND: Evidence-based guideline and vaccination recommendations should continuously be updated to appropriately support health care decisions. However, resources for updating guidelines are often limited. The aim of this project was to develop a list of criteria for the prospective assessment of the need for updating individual guideline or vaccination recommendations, which can be applied from the time a guideline or guideline update is finalised. METHODS: In this article we describe the development of the AGIL criteria (Assessment of Guidelines for Updating Recommendations). The AGIL criteria were developed by experienced scientists and experts in the field of guideline development in a multi-step process. The five steps included: 1) development of an initial list of criteria by the project team; 2) online survey of guideline experts on the initial version of the criteria list; 3) revision of the criteria list based on the results of the online survey; 4) workshop on the criteria list at the EbM Congress 2023; 5) creation of version 1.0 of the AGIL criteria based on the workshop results. RESULTS: The initial list included the following three criteria: 1) relevance of the question 2) availability of new relevant evidence, and 3) impact of potentially new evidence. The response rate of the online survey for fully completed questionnaires was 31.0% (N=195; 630 guideline experts were contacted by email). For 90.3% (n=176) of the respondents, the criteria list included all essential aspects for assessing the need for updating guideline recommendations. More than three quarters of respondents rated the importance of the three criteria as "very important" or "important" (criteria 1-3: 75.3%, 86.1%, 85.2%) and - with the exception of criterion 1 - comprehensibility as "very comprehensible" or "comprehensible" (criteria 1-3: 58.4%, 75.9%, 78.5%). The results of the online survey and the workshop generally confirmed the three criteria with their two sub-questions. The incorporation of all feedback resulted in the AGIL criteria (version 1.0), recapping: 1) relevance of the question regarding a) PICO components and b) other factors, e.g. epidemiological aspects; 2) availability of new evidence a) on health-related benefits and harms and b) on other decision factors, e.g. feasibility, acceptability; 3) impact of new evidence a) on the certainty of evidence on which the recommendation is based and b) on the present recommendation, e.g. DISCUSSION: The moderate response rate of the online survey may have limited its representativeness. Nevertheless, we consider the response rate to be satisfactory in this research context. The inclusion of many experts in the online survey and the EbM Congress workshop is a strength of the project and supports the quality of the results. CONCLUSIONS: The AGIL criteria provide a structured guidance for the prospective assessment of the need for updating individual guideline recommendations and other evidence-based recommendations. The implementation and evaluation of the AGIL criteria 1.0 in a field test is planned.

Marcos GRADE de la evidencia a la decisión (EtD): un enfoque sistemático y transparente para tomar decisiones sanitarias bien informadas. 2: Guías de práctica clínica / GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines

Los médicos no disponen del tiempo ni de los recursos para considerar la evidencia subyacente en las innumerables decisiones que tienen que tomar diariamente. En consecuencia, dependen de las recomendaciones de las guías de práctica clínica. Los paneles de las guías deben considerar todos los criterios relevantes que influyen en una decisión o recomendación de manera estructurada, explícita y transparente, y proporcionar a los médicos recomendaciones factibles. En este artículo describiremos los marcos de la evidencia a la decisión (EtD) para las recomendaciones de práctica clínica. La estructura general de un marco EtD para recomendaciones clínicas es similar a la de los marcos EtD para otras recomendaciones y decisiones, e incluye la formulación de la pregunta, la evaluación de los distintos criterios y las conclusiones. Las recomendaciones clínicas requieren que los criterios se consideren de forma diferente, dependiendo de si se adopta una perspectiva individual o poblacional. Por ejemplo, desde la perspectiva individual, los gastos personales son un aspecto importante a considerar, mientras que desde la perspectiva poblacional son más importantes el uso de recursos (no solo los gastos personales) y el coste-efectividad. Son también importantes desde la perspectiva poblacional la equidad, la aceptabilidad y la factibilidad, mientras que la importancia de estos criterios suele ser limitada en el caso de la perspectiva individual. Los subgrupos específicos para los cuales pueden necesitarse recomendaciones deben estar claramente identificados y considerados con relación a cada criterio, porque los juicios pueden variar entre subgrupos. El siguiente artículo es una traducción del artículo original publicado en British Medical Journal. Los marcos EtD se utilizan actualmente en el Programa de Guías de Práctica Clínica en el Sistema Nacional de Salud, coordinado por GuíaSalud (AU)

Clinicians do not have the time or resources to consider the underlying evidence for the myriad decisions they must make each day and, as a consequence, rely on recommendations from clinical practice guidelines. Guideline panels should consider all the relevant factors (criteria) that influence a decision or recommendation in a structured, explicit, and transparent way and provide clinicians with clear and actionable recommendations. In this article, we will describe the Evidence to Decision (EtD) frameworks for clinical practice recommendations. The general structure of the EtD framework for clinical recommendations is similar to EtD frameworks for other types of recommendations and decisions, and includes formulation of the question, an assessment of the different criteria, and conclusions. Clinical recommendations require considering criteria differently, depending on whether an individual patient or a population perspective is taken. For example, from an individual patient's perspective, out-of-pocket costs are an important consideration, whereas, from a population perspective, resource use (not only out-of-pocket costs) and cost effectiveness are important. From a population perspective, equity, acceptability, and feasibility are also important considerations, whereas the importance of these criteria is often limited from an individual patient perspective. Specific subgroups for which different recommendations may be required should be clearly identified and considered in relation to each criterion because judgments might vary across subgroups. This article is a translation of the original article published in the British Medical Journal. The EtD frameworks are currently used in the Clinical Practice Guideline Programme of the Spanish National Health System, co-ordinated by GuíaSalud (AU)