Exploring decision-makers' challenges and strategies when selecting multiple systematic reviews: insights for AI decision support tools in healthcare.

BACKGROUND: Systematic reviews (SRs) are being published at an accelerated rate. Decision-makers may struggle with comparing and choosing between multiple SRs on the same topic. We aimed to understand how healthcare decision-makers (eg, practitioners, policymakers, researchers) use SRs to inform decision-making and to explore the potential role of a proposed artificial intelligence (AI) tool to assist in critical appraisal and choosing among SRs. METHODS: We developed a survey with 21 open and closed questions. We followed a knowledge translation plan to disseminate the survey through social media and professional networks. RESULTS: Our survey response rate was lower than expected (7.9% of distributed emails). Of the 684 respondents, 58.2% identified as researchers, 37.1% as practitioners, 19.2% as students and 13.5% as policymakers. Respondents frequently sought out SRs (97.1%) as a source of evidence to inform decision-making. They frequently (97.9%) found more than one SR on a given topic of interest to them. Just over half (50.8%) struggled to choose the most trustworthy SR among multiple. These difficulties related to lack of time (55.2%), or difficulties comparing due to varying methodological quality of SRs (54.2%), differences in results and conclusions (49.7%) or variation in the included studies (44.6%). Respondents compared SRs based on the relevance to their question of interest, methodological quality, and recency of the SR search. Most respondents (87.0%) were interested in an AI tool to help appraise and compare SRs. CONCLUSIONS: Given the identified barriers of using SR evidence, an AI tool to facilitate comparison of the relevance of SRs, the search and methodological quality, could help users efficiently choose among SRs and make healthcare decisions.

A systematic survey of 200 systematic reviews with network meta-analysis (published 2020-2021) reveals that few reviews report structured evidence summaries.

OBJECTIVES: To map whether and how systematic reviews (SRs) with network meta-analysis (NMA) use presentation formats to report (a) structured evidence summaries - here defined as reporting of effects estimates in absolute effects with certainty ratings and with a method to rate interventions across one or more outcome(s) - and (b) NMA results in general. STUDY DESIGN AND SETTING: We conducted a systematic survey, searching MEDLINE (Ovid) for SRs with NMA published between January 1, 2020, and December 31, 2021. We planned to include a random sample of publications, with predefined mechanisms in place for saturation, and included SRs that met prespecified quality criteria and extracted data on presentation formats that reported: (a) estimates of effects, (b) certainty of the evidence, or (c) rating of interventions. RESULTS: The 200 eligible SRs, from 158 unique Journals, utilized 1133 presentation formats. We found structured evidence summaries in 10 publications (5.0%), with 3 (1.5%) reporting structured evidence summaries across all outcomes, including benefits and harms. Sixteen of the 133 SRs (11.7%) reporting dichotomous outcomes included estimates of absolute effects. Seventy-six SRs (38.0%) reported both benefits and harms and 26 SRs (13.0%) reported certainty ratings in presentation formats, 20 (76.9%) used Grading of Recommendations Assessment, Development and Evaluation and 6 (23.1%) used Confidence In Network Meta-analysis. Surface Under the Cumulative Ranking Curve was the most common method to rate interventions (69 SRs, 34.5%). NMA results were most often reported using forest plots (108 SRs, 54.0%) and league tables (93 SRs, 46.5%). CONCLUSION: Most SRs with NMA do not report structured evidence summaries and only rarely do such summaries include reporting of both benefits and harms; those that do offer effective user-friendly communication and provide models for optimal NMA presentation practice.

Only half of the authors of overviews of exercise-related interventions use some strategy to manage overlapping primary studies-a metaresearch study.

OBJECTIVES: The conduct of systematic reviews (SRs) and overviews share several similarities. However, because the unit of analysis for overviews is the SRs, there are some unique challenges. One of the most critical issues to manage when conducting an overview is the overlap of data across the primary studies included in the SRs. This metaresearch study aimed to describe the frequency of strategies to manage the overlap in overviews of exercise-related interventions. STUDY DESIGN AND SETTING: A systematic search in MEDLINE (Ovid), Embase (Ovid), Cochrane Library, Epistemonikos, and other sources was conducted from inception to June 2022. We included overviews of SRs that considered primary studies and evaluated the effectiveness of exercise-related interventions for any health condition. The overviews were screened by two authors independently, and the extraction was performed by one author and checked by a second. We found 353 overviews published between 2005 and 2022 that met the inclusion criteria. RESULTS: One hundred and sixty-four overviews (46%) used at least one strategy to visualize, quantify, or resolve overlap, with a matrix (32/164; 20%), absolute frequency (34/164; 21%), and authors' algorithms (24/164; 15%) being the most used methods, respectively. From 2016 onwards, there has been a trend toward increasing the use of some strategies to manage overlap. Of the 108 overviews that used some strategy to resolve the overlap, ie, avoiding double or multiple counting of primary study data, 79 (73%) succeeded. In overviews where no strategies to manage overlap were reported (n = 189/353; 54%), 16 overview authors (8%) recognized this as a study limitation. CONCLUSION: Although there is a trend toward increasing its use, only half of the authors of the overviews of exercise-related interventions used a strategy to visualize, quantify, or resolve overlap in the primary studies' data. In the future, authors should report such strategies to communicate more valid results.

Methods proposed for monitoring the implementation of evidence-based research: a cross-sectional study.

OBJECTIVES: Evidence-based research (EBR) is the systematic and transparent use of prior research to inform a new study so that it answers questions that matter in a valid, efficient, and accessible manner. This study surveyed experts about existing (e.g., citation analysis) and new methods for monitoring EBR and collected ideas about implementing these methods. STUDY DESIGN AND SETTING: We conducted a cross-sectional study via an online survey between November 2022 and March 2023. Participants were experts from the fields of evidence synthesis and research methodology in health research. Open-ended questions were coded by recurring themes; descriptive statistics were used for quantitative questions. RESULTS: Twenty-eight expert participants suggested that citation analysis should be supplemented with content evaluation (not just what is cited but also in which context), content expert involvement, and assessment of the quality of cited systematic reviews. They also suggested that citation analysis could be facilitated with automation tools. They emphasized that EBR monitoring should be conducted by ethics committees and funding bodies before the research starts. Challenges identified for EBR implementation monitoring were resource constraints and clarity on responsibility for EBR monitoring. CONCLUSION: Ideas proposed in this study for monitoring the implementation of EBR can be used to refine methods and define responsibility but should be further explored in terms of feasibility and acceptability. Different methods may be needed to determine if the use of EBR is improving over time.

Methodological review of NMA bias concepts provides groundwork for the development of a list of concepts for potential inclusion in a new risk of bias tool for network meta-analysis (RoB NMA Tool).

INTRODUCTION: Network meta-analyses (NMAs) have gained popularity and grown in number due to their ability to provide estimates of the comparative effectiveness of multiple treatments for the same condition. The aim of this study is to conduct a methodological review to compile a preliminary list of concepts related to bias in NMAs. METHODS AND ANALYSIS: We included papers that present items related to bias, reporting or methodological quality, papers assessing the quality of NMAs, or method papers. We searched MEDLINE, the Cochrane Library and unpublished literature (up to July 2020). We extracted items related to bias in NMAs. An item was excluded if it related to general systematic review quality or bias and was included in currently available tools such as ROBIS or AMSTAR 2. We reworded items, typically structured as questions, into concepts (i.e. general notions). RESULTS: One hundred eighty-one articles were assessed in full text and 58 were included. Of these articles, 12 were tools, checklists or journal standards; 13 were guidance documents for NMAs; 27 were studies related to bias or NMA methods; and 6 were papers assessing the quality of NMAs. These studies yielded 99 items of which the majority related to general systematic review quality and biases and were therefore excluded. The 22 items we included were reworded into concepts specific to bias in NMAs. CONCLUSIONS: A list of 22 concepts was included. This list is not intended to be used to assess biases in NMAs, but to inform the development of items to be included in our tool.

HIGHLIGHTS: • Our research aimed to develop a preliminary list of concepts related to bias with the goal of developing the first tool for assessing the risk of bias in the results and conclusions of a network meta-analysis (NMA).• We followed the methodology proposed by Whiting (2017) and Sanderson (2007) for creating systematically developed lists of quality items, as a first step in the development of a risk of bias tool for network meta-analysis (RoB NMA Tool).• We included items related to biases in NMAs and excluded items that are equally applicable to all systematic reviews as they are covered by other tools (e.g. ROBIS, AMSTAR 2).• Fifty-seven studies were included generating 99 items, which when screened, yielded 22 included items. These items were then reworded into concepts in preparation for a Delphi process for further vetting by external experts.• A limitation of our study is the challenge in retrieving methods studies as methods collections are not regularly updated.

Exercise and aerobic capacity in individuals with spinal cord injury: A systematic review with meta-analysis and meta-regression.

BACKGROUND: A low level of cardiorespiratory fitness [CRF; defined as peak oxygen uptake ([Formula: see text]O2peak) or peak power output (PPO)] is a widely reported consequence of spinal cord injury (SCI) and a major risk factor associated with chronic disease. However, CRF can be modified by exercise. This systematic review with meta-analysis and meta-regression aimed to assess whether certain SCI characteristics and/or specific exercise considerations are moderators of changes in CRF. METHODS AND FINDINGS: Databases (MEDLINE, EMBASE, CENTRAL, and Web of Science) were searched from inception to March 2023. A primary meta-analysis was conducted including randomised controlled trials (RCTs; exercise interventions lasting >2 weeks relative to control groups). A secondary meta-analysis pooled independent exercise interventions >2 weeks from longitudinal pre-post and RCT studies to explore whether subgroup differences in injury characteristics and/or exercise intervention parameters explained CRF changes. Further analyses included cohort, cross-sectional, and observational study designs. Outcome measures of interest were absolute (A[Formula: see text]O2peak) or relative [Formula: see text]O2peak (R[Formula: see text]O2peak), and/or PPO. Bias/quality was assessed via The Cochrane Risk of Bias 2 and the National Institute of Health Quality Assessment Tools. Certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach. Random effects models were used in all meta-analyses and meta-regressions. Of 21,020 identified records, 120 studies comprising 29 RCTs, 67 pre-post studies, 11 cohort, 7 cross-sectional, and 6 observational studies were included. The primary meta-analysis revealed significant improvements in A[Formula: see text]O2peak [0.16 (0.07, 0.25) L/min], R[Formula: see text]O2peak [2.9 (1.8, 3.9) mL/kg/min], and PPO [9 (5, 14) W] with exercise, relative to controls (p < 0.001). Ninety-six studies (117 independent exercise interventions comprising 1,331 adults with SCI) were included in the secondary, pooled meta-analysis which demonstrated significant increases in A[Formula: see text]O2peak [0.22 (0.17, 0.26) L/min], R[Formula: see text]O2peak [2.8 (2.2, 3.3) mL/kg/min], and PPO [11 (9, 13) W] (p < 0.001) following exercise interventions. There were subgroup differences for R[Formula: see text]O2peak based on exercise modality (p = 0.002) and intervention length (p = 0.01), but there were no differences for A[Formula: see text]O2peak. There were subgroup differences (p ≤ 0.018) for PPO based on time since injury, neurological level of injury, exercise modality, and frequency. The meta-regression found that studies with a higher mean age of participants were associated with smaller changes in A[Formula: see text]O2peak and R[Formula: see text]O2peak (p < 0.10). GRADE indicated a moderate level of certainty in the estimated effect for R[Formula: see text]O2peak, but low levels for A[Formula: see text]O2peak and PPO. This review may be limited by the small number of RCTs, which prevented a subgroup analysis within this specific study design. CONCLUSIONS: Our primary meta-analysis confirms that performing exercise >2 weeks results in significant improvements to A[Formula: see text]O2peak, R[Formula: see text]O2peak, and PPO in individuals with SCI. The pooled meta-analysis subgroup comparisons identified that exercise interventions lasting up to 12 weeks yield the greatest change in R[Formula: see text]O2peak. Upper-body aerobic exercise and resistance training also appear the most effective at improving R[Formula: see text]O2peak and PPO. Furthermore, acutely injured, individuals with paraplegia, exercising for ≥3 sessions/week will likely experience the greatest change in PPO. Ageing seemingly diminishes the adaptive CRF responses to exercise training in individuals with SCI. REGISTRATION: PROSPERO: CRD42018104342.

Clinical utility of overviews on adverse events of pharmacological interventions.

BACKGROUND: Overviews (i.e., systematic reviews of systematic reviews, meta-reviews, umbrella reviews) are a relatively new type of evidence synthesis. Among others, one reason to conduct an overview is to investigate adverse events (AEs) associated with a healthcare intervention. Overviews aim to provide easily accessible information for healthcare decision-makers including clinicians. We aimed to evaluate the clinical utility of overviews investigating AEs. METHODS: We used a sample of 27 overviews exclusively investigating drug-related adverse events published until 2021 identified in a prior project. We defined clinical utility as the extent to which overviews are perceived to be useful in clinical practice. Each included overview was assigned to one of seven pharmacological experts with expertise on the topic of the overview. The clinical utility and value of these overviews were determined using a self-developed assessment tool. This included four open-ended questions and a ranking of three clinical utility statements completed by clinicians. We calculated frequencies for the ranked clinical utility statements and coded the answers to the open-ended questions using an inductive approach. RESULTS: The overall agreement with the provided statements was high. According to the assessments, 67% of the included overviews generated new knowledge. In 93% of the assessments, the overviews were found to add value to the existing literature. The overviews were rated as more useful than the individual included systematic reviews (SRs) in 85% of the assessments. The answers to the open-ended questions revealed two key aspects of clinical utility in the included overviews. Firstly, it was considered useful that they provide a summary of available evidence (e.g., along with additional assessments, or across different populations, or in different settings that have not been evaluated together in the included SRs). Secondly, it was found useful if overviews conducted a new meta-analysis to answer specific research questions that had not been answered previously. CONCLUSIONS: Overviews on drug-related AEs are considered valuable for clinical practice by clinicians. They can make available evidence on AEs more accessible and provide a comprehensive view of available evidence. As the role of overviews evolves, investigations such as this can identify areas of value.

Can routinely collected administrative data effectively be used to evaluate and validate endpoints used in breast cancer clinical trials? Protocol for a scoping review of the literature.

BACKGROUND: Randomized controlled trials (RCTs) are a critical component of evidence-based medicine and the evolution of patient care. However, the costs of conducting a RCT can be prohibitive. A promising approach toward reduction of costs and lessening of the burden of intensive and lengthy patient follow-up is the use of routinely collected healthcare data (RCHD), commonly called real-world data. We propose a scoping review to identify existing RCHD case definitions of breast cancer progression and survival and their diagnostic performance. METHODS: We will search MEDLINE, EMBASE, and CINAHL to identify primary studies of women with either early-stage or metastatic breast cancer, managed with established therapies, that evaluated the diagnostic accuracy of one or more RCHD-based case definitions or algorithms of disease progression (i.e., recurrence, progression-free survival, disease-free survival, or invasive disease-free survival) or survival (i.e., breast-cancer-free survival or overall survival) compared with a reference standard measure (e.g., chart review or a clinical trial dataset). Study characteristics and descriptions of algorithms will be extracted along with measures of the diagnostic accuracy of each algorithm (e.g., sensitivity, specificity, positive predictive value, negative predictive value), which will be summarized both descriptively and in structured figures/tables. DISCUSSION: Findings from this scoping review will be clinically meaningful for breast cancer researchers globally. Identification of feasible and accurate strategies to measure patient-important outcomes will potentially reduce RCT budgets as well as lessen the burden of intensive trial follow-up on patients. SYSTEMATIC REVIEW REGISTRATION: Open Science Framework ( https://doi.org/10.17605/OSF.IO/6D9RS ).

Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey.

OBJECTIVE: Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs. STUDY DESIGN AND SETTING: We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis. RESULTS: 162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%). CONCLUSION: Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.

Does type of funding affect reporting in network meta-analysis? A scoping review of network meta-analyses.

BACKGROUND: Evidence has shown that private industry-sponsored randomized controlled trials (RCTs) and meta-analyses are more likely to report intervention-favourable results compared with other sources of funding. However, this has not been assessed in network meta-analyses (NMAs). OBJECTIVES: To (a) explore the recommendation rate of industry-sponsored NMAs on their company's intervention, and (b) assess reporting in NMAs of pharmacologic interventions according to their funding type. METHODS: Design: Scoping review of published NMAs with RCTs. INFORMATION SOURCES: We used a pre-existing NMA database including 1,144 articles from MEDLINE, EMBASE and Cochrane Database of Systematic Reviews, published between January 2013 and July 2018. STUDY SELECTION: NMAs with transparent funding information and comparing pharmacologic interventions with/without placebo. SYNTHESIS: We captured whether NMAs recommended their own or another company's intervention, classified NMAs according to their primary outcome findings (i.e., statistical significance and direction of effect), and according to the overall reported conclusion. We assessed reporting using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension to NMA (PRISMA-NMA) 32-item checklist. We matched and compared industry with non-industry NMAs having the same research question, disease, primary outcome, and pharmacologic intervention against placebo/control. RESULTS: We retrieved 658 NMAs, which reported a median of 23 items in the PRISMA-NMA checklist (interquartile range [IQR]: 21-26). NMAs were categorized as 314 publicly-sponsored (PRISMA-NMA median 24.5, IQR 22-27), 208 non-sponsored (PRISMA-NMA median 23, IQR 20-25), and 136 industry/mixed-sponsored NMAs (PRISMA-NMA median 21, IQR 19-24). Most industry-sponsored NMAs recommended their own manufactured drug (92%), suggested a statistically significant positive treatment-effect for their drug (82%), and reported an overall positive conclusion (92%). Our matched NMAs (25 industry vs 25 non-industry) indicated that industry-sponsored NMAs had favourable conclusions more often (100% vs 80%) and were associated with larger (but not statistically significantly different) efficacy effect sizes (in 61% of NMAs) compared with non-industry-sponsored NMAs. CONCLUSIONS: Differences in completeness of reporting and author characteristics were apparent among NMAs with different types of funding. Publicly-sponsored NMAs had the best reporting and published their findings in higher impact-factor journals. Knowledge users should be mindful of this potential funding bias in NMAs.

Strategies used to manage overlap of primary study data by exercise-related overviews: protocol for a systematic methodological review.

INTRODUCTION: One of the most conflicting methodological issues when conducting an overview is the overlap of primary studies across systematic reviews (SRs). Overlap in the pooled effect estimates across SRs may lead to overly precise effect estimates in the overview. SRs that focus on exercise-related interventions are often included in overviews aimed at grouping and determining the effectiveness of various interventions for managing specific health conditions. The aim of this systematic methodological review is to describe the strategies used by authors of overviews focusing on exercise-related interventions to manage the overlap of primary studies. METHODS AND ANALYSIS: A comprehensive search strategy has been developed for different databases and their platforms. The databases to be consulted will be MEDLINE (Ovid), Embase (Ovid), The Cochrane Database of Systematic Reviews (Cochrane Library) and Epistemonikos. Two reviewers will independently screen the records identified through the search strategy and extract the information from the included overviews. The frequency and the type of overlap management strategies of the primary studies included in the SRs will be considered as the main outcome. In addition, the recognition of the lack of use of any overlap management strategy and the congruence between planning and conducting the overview focusing on overlap management strategies will be assessed. A subgroup analysis will be carried out according to the journal impact factor, year of publication and compliance with the Preferred Reporting Items for Overviews of Reviews statement. ETHICS AND DISSEMINATION: This study will not involve human subjects and therefore does not require ethics committee approval. However, the conduct and reporting of the findings of this review will be conducted in a rigorous, systematic and transparent manner, which relates to research ethics.The findings of this review will be presented at scientific conferences and published as one or more studies in peer-review scientific journals related to rehabilitation or research methods.

Rapid antigen-based and rapid molecular tests for the detection of SARS-CoV-2: a rapid review with network meta-analysis of diagnostic test accuracy studies.

BACKGROUND: The global spread of COVID-19 created an explosion in rapid tests with results in < 1 hour, but their relative performance characteristics are not fully understood yet. Our aim was to determine the most sensitive and specific rapid test for the diagnosis of SARS-CoV-2. METHODS: Design: Rapid review and diagnostic test accuracy network meta-analysis (DTA-NMA). ELIGIBILITY CRITERIA: Randomized controlled trials (RCTs) and observational studies assessing rapid antigen and/or rapid molecular test(s) to detect SARS-CoV-2 in participants of any age, suspected or not with SARS-CoV-2 infection. INFORMATION SOURCES: Embase, MEDLINE, and Cochrane Central Register of Controlled Trials, up to September 12, 2021. OUTCOME MEASURES: Sensitivity and specificity of rapid antigen and molecular tests suitable for detecting SARS-CoV-2. Data extraction and risk of bias assessment: Screening of literature search results was conducted by one reviewer; data abstraction was completed by one reviewer and independently verified by a second reviewer. Risk of bias was not assessed in the included studies. DATA SYNTHESIS: Random-effects meta-analysis and DTA-NMA. RESULTS: We included 93 studies (reported in 88 articles) relating to 36 rapid antigen tests in 104,961 participants and 23 rapid molecular tests in 10,449 participants. Overall, rapid antigen tests had a sensitivity of 0.75 (95% confidence interval 0.70-0.79) and specificity of 0.99 (0.98-0.99). Rapid antigen test sensitivity was higher when nasal or combined samples (e.g., combinations of nose, throat, mouth, or saliva samples) were used, but lower when nasopharyngeal samples were used, and in those classified as asymptomatic at the time of testing. Rapid molecular tests may result in fewer false negatives than rapid antigen tests (sensitivity: 0.93, 0.88-0.96; specificity: 0.98, 0.97-0.99). The tests with the highest sensitivity and specificity estimates were the Xpert Xpress rapid molecular test by Cepheid (sensitivity: 0.99, 0.83-1.00; specificity: 0.97, 0.69-1.00) among the 23 commercial rapid molecular tests and the COVID-VIRO test by AAZ-LMB (sensitivity: 0.93, 0.48-0.99; specificity: 0.98, 0.44-1.00) among the 36 rapid antigen tests we examined. CONCLUSIONS: Rapid molecular tests were associated with both high sensitivity and specificity, while rapid antigen tests were mainly associated with high specificity, according to the minimum performance requirements by WHO and Health Canada. Our rapid review was limited to English, peer-reviewed published results of commercial tests, and study risk of bias was not assessed. A full systematic review is required. REVIEW REGISTRATION: PROSPERO CRD42021289712.

Use of existing systematic reviews for the development of evidence-based vaccination recommendations: Guidance from the SYSVAC expert panel.

National immunization technical advisory groups (NITAGs) develop immunization-related recommendations and assist policy-makers in making evidence informed decisions. Systematic reviews (SRs) that summarize the available evidence on a specific topic are a valuable source of evidence in the development of such recommendations. However, conducting SRs requires significant human, time, and financial resources, which many NITAGs lack. Given that SRs already exist for many immunization-related topics, and to prevent duplication and overlap of reviews, a more practical approach may be for NITAGs to use existing SRs. Nevertheless, it can be challenging to identify relevant SRs, to select one SR from among multiple SRs, or to critically assess and effectively use them. To support NITAGs, the London School of Hygiene and Tropical Medicine, Robert Koch Institute and collaborators developed the SYSVAC project, which consists of an online registry of systematic reviews on immunization-related topics and an e-learning course, that supports the use of them (both freely accessible at https://www.nitag-resource.org/sysvac-systematic-reviews). Drawing from the e-learning course and recommendations from an expert panel, this paper outlines methods for using existing systematic reviews when making immunization-related recommendations. With specific examples and reference to the SYSVAC registry and other resources, it offers guidance on locating existing systematic reviews; assessing their relevance to a research question, up-to-dateness, and methodological quality and/or risk of bias; and considering the transferability and applicability of their findings to other populations or settings.

Knowledge user survey and Delphi process to inform development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool).

BACKGROUND: Network meta-analysis (NMA) is increasingly used in guideline development and other aspects of evidence-based decision-making. We aimed to develop a risk of bias (RoB) tool to assess NMAs (RoB NMA tool). An international steering committee recommended that the RoB NMA tool to be used in combination with the Risk of Bias in Systematic reviews (ROBIS) tool (i.e. because it was designed to assess biases only) or other similar quality appraisal tools (eg, A MeaSurement Tool to Assess systematic Reviews 2 [AMSTAR 2]) to assess quality of systematic reviews. The RoB NMA tool will assess NMA biases and limitations regarding how the analysis was planned, data were analysed and results were presented, including the way in which the evidence was assembled and interpreted. OBJECTIVES: Conduct (a) a Delphi process to determine expert opinion on an item's inclusion and (b) a knowledge user survey to widen its impact. DESIGN: Cross-sectional survey and Delphi process. METHODS: Delphi panellists were asked to rate whether items should be included. All agreed-upon item were included in a second round of the survey (defined as 70% agreement). We surveyed knowledge users' views and preferences about the importance, utility and willingness to use the RoB NMA tool to evaluate evidence in practice and in policymaking. We included 12 closed and 10 open-ended questions, and we followed a knowledge translation plan to disseminate the survey through social media and professional networks. RESULTS: 22 items were entered into a Delphi survey of which 28 respondents completed round 1, and 22 completed round 2. Seven items did not reach consensus in round 2. A total of 298 knowledge users participated in the survey (14% respondent rate). 75% indicated that their organisation produced NMAs, and 78% showed high interest in the tool, especially if they had received adequate training (84%). Most knowledge users and Delphi panellists preferred a tool to assess both bias in individual NMA results and authors' conclusions. Response bias in our sample is a major limitation as knowledge users working in high-income countries were more represented. One of the limitations of the Delphi process is that it depends on the purposive selection of experts and their availability, thus limiting the variability in perspectives and scientific disciplines. CONCLUSIONS: This Delphi process and knowledge user survey informs the development of the RoB NMA tool.

One-Third of Systematic Reviews in Rehabilitation Applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) System to Evaluate Certainty of Evidence: A Meta-Research Study.

OBJECTIVE: To determine how many systematic reviews (SRs) of the literature in rehabilitation assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to do this. DATA SOURCES: For this meta-research study, we searched PubMed and Cochrane Database of Systematic Reviews databases for SRs on rehabilitation published in 2020. STUDY SELECTION AND DATA EXTRACTION: Two reviewers independently selected the SRs and extracted the data. Reporting characteristics and appropriate use of the GRADE system were assessed. DATA SYNTHESIS: The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of which applied the GRADE system. GRADE was used by SRs of randomized controlled trials (RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according to the study design was not provided in 75% (12/16). The reasons for GRADE judgment were reported in 82% (134/163) of SRs. CONCLUSIONS: One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment was presented transparently by most SRs. Journal editors and funders should encourage the uptake of the GRADE system when considering SRs in rehabilitation for publication. The authors should pre-define GRADE assessment in a registered and/or published protocol.

A review found heterogeneous approaches and insufficient reporting in overviews on adverse events.

OBJECTIVES: To investigate reporting and methodological characteristics of overviews on adverse (drug-associated) events (AEs) of pharmacological interventions. STUDY DESIGN AND SETTING: We searched MEDLINE, Embase, Epistemonikos, and the Cochrane Database of Systematic Reviews from inception to May 17, 2021 for overviews exclusively investigating AEs of pharmacological interventions. We extracted general, reporting, and methodological characteristics and analyzed data descriptively. RESULTS: We included 27 overviews, 70% of which were published in 2016 or later. The most common nomenclature in the title was "overview" (56%), followed by "umbrella review" (26%). The median number of included systematic reviews (SRs) in each overview was 15 (interquartile range 7-34). Study selection methods were reported in 52%, methods for data extraction in 67%, and methods for critical appraisal in 63% of overviews. An assessment of methodological quality of included SRs was performed in 70% of overviews. Only 22% of overviews reported strategies for dealing with overlapping SRs. An assessment of the certainty of the evidence was performed in 33% of overviews. CONCLUSION: To ensure methodological rigor, authors of overviews on AEs should follow available guidance for the conduct and reporting of overviews.

Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement.

OBJECTIVE: To develop a reporting guideline for overviews of reviews of healthcare interventions. DESIGN: Development of the preferred reporting items for overviews of reviews (PRIOR) statement. PARTICIPANTS: Core team (seven individuals) led day-to-day operations, and an expert advisory group (three individuals) provided methodological advice. A panel of 100 experts (authors, editors, readers including members of the public or patients) was invited to participate in a modified Delphi exercise. 11 expert panellists (chosen on the basis of expertise, and representing relevant stakeholder groups) were invited to take part in a virtual face-to-face meeting to reach agreement (≥70%) on final checklist items. 21 authors of recently published overviews were invited to pilot test the checklist. SETTING: International consensus. INTERVENTION: Four stage process established by the EQUATOR Network for developing reporting guidelines in health research: project launch (establish a core team and expert advisory group, register intent), evidence reviews (systematic review of published overviews to describe reporting quality, scoping review of methodological guidance and author reported challenges related to undertaking overviews of reviews), modified Delphi exercise (two online Delphi surveys to reach agreement (≥70%) on relevant reporting items followed by a virtual face-to-face meeting), and development of the reporting guideline. RESULTS: From the evidence reviews, we drafted an initial list of 47 potentially relevant reporting items. An international group of 52 experts participated in the first Delphi survey (52% participation rate); agreement was reached for inclusion of 43 (91%) items. 44 experts (85% retention rate) completed the second Delphi survey, which included the four items lacking agreement from the first survey and five new items based on respondent comments. During the second round, agreement was not reached for the inclusion or exclusion of the nine remaining items. 19 individuals (6 core team and 3 expert advisory group members, and 10 expert panellists) attended the virtual face-to-face meeting. Among the nine items discussed, high agreement was reached for the inclusion of three and exclusion of six. Six authors participated in pilot testing, resulting in minor wording changes. The final checklist includes 27 main items (with 19 sub-items) across all stages of an overview of reviews. CONCLUSIONS: PRIOR fills an important gap in reporting guidance for overviews of reviews of healthcare interventions. The checklist, along with rationale and example for each item, provides guidance for authors that will facilitate complete and transparent reporting. This will allow readers to assess the methods used in overviews of reviews of healthcare interventions and understand the trustworthiness and applicability of their findings.