Framework for a living systematic review and meta-analysis for the surgical treatment of bladder cancer: introducing EVIglance to urology.

Background: Knowledge of current and ongoing studies is critical for identifying research gaps and enabling evidence-based decisions for individualized treatment. However, the increasing number of scientific publications poses challenges for healthcare providers and patients in all medical fields to stay updated with the latest evidence. To overcome these barriers, we aim to develop a living systematic review and open-access online evidence map of surgical therapy for bladder cancer (BC), including meta-analyses. Methods: Following the guidelines provided in the Cochrane Handbook for Systematic Reviews of Interventions and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement, a systematic literature search on uro-oncological therapy in BC will be performed across various literature databases. Within the scope of a meta-analysis and living systematic review, relevant randomized controlled trials will be identified. Data extraction and quantitative analysis will be conducted, along with a critical appraisal of the quality and risk of bias of each study. The available research evidence will be entered into an open-access framework (www.evidencemap.surgery) and will also be accessible via the EVIglance app. Regular semi-automatic updates will enable the implementation of a real-living review concept and facilitate resource-efficient screening. Discussion: A regularly updated evidence map provides professionals and patients with an open-access knowledge base on the current state of research, allowing for decision-making based on recent evidence. It will help identify an oversupply of evidence, thus avoiding redundant work. Furthermore, by identifying research gaps, new hypotheses can be formulated more precisely, enabling planning, determination of sample size, and definition of endpoints for future trials.

Methods and guidance on conducting, reporting, publishing, and appraising living systematic reviews: a scoping review.

BACKGROUND AND OBJECTIVE: The living systematic review (LSR) approach is based on ongoing surveillance of the literature and continual updating. Most currently available guidance documents address the conduct, reporting, publishing, and appraisal of systematic reviews (SRs), but are not suitable for LSRs per se and miss additional LSR-specific considerations. In this scoping review, we aim to systematically collate methodological guidance literature on how to conduct, report, publish, and appraise the quality of LSRs and identify current gaps in guidance. METHODS: A standard scoping review methodology was used. We searched MEDLINE (Ovid), EMBASE (Ovid), and The Cochrane Library on August 28, 2021. As for searching gray literature, we looked for existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses. The screening was conducted by two authors independently in Rayyan, and data extraction was done in duplicate using a pilot-tested data extraction form in Excel. Data was extracted according to four pre-defined categories for (i) conducting, (ii) reporting, (iii) publishing, and (iv) appraising LSRs. We mapped the findings by visualizing overview tables created in Microsoft Word. RESULTS: Of the 21 included papers, methodological guidance was found in 17 papers for conducting, in six papers for reporting, in 15 papers for publishing, and in two papers for appraising LSRs. Some of the identified key items for (i) conducting LSRs were identifying the rationale, screening tools, or re-revaluating inclusion criteria. Identified items of (ii) the original PRISMA checklist included reporting the registration and protocol, title, or synthesis methods. For (iii) publishing, there was guidance available on publication type and frequency or update trigger, and for (iv) appraising, guidance on the appropriate use of bias assessment or reporting funding of included studies was found. Our search revealed major evidence gaps, particularly for guidance on certain PRISMA items such as reporting results, discussion, support and funding, and availability of data and material of a LSR. CONCLUSION: Important evidence gaps were identified for guidance on how to report in LSRs and appraise their quality. Our findings were applied to inform and prepare a PRISMA 2020 extension for LSR.

Crowd-sourcing and automation facilitated the identification and classification of randomized controlled trials in a living review.

OBJECTIVES: To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR). METHODS: Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers. RESULTS: From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69). CONCLUSION: A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.

The Automated Systematic Search Deduplicator (ASySD): a rapid, open-source, interoperable tool to remove duplicate citations in biomedical systematic reviews.

BACKGROUND: Researchers performing high-quality systematic reviews search across multiple databases to identify relevant evidence. However, the same publication is often retrieved from several databases. Identifying and removing such duplicates ("deduplication") can be extremely time-consuming, but failure to remove these citations can lead to the wrongful inclusion of duplicate data. Many existing tools are not sensitive enough, lack interoperability with other tools, are not freely accessible, or are difficult to use without programming knowledge. Here, we report the performance of our Automated Systematic Search Deduplicator (ASySD), a novel tool to perform automated deduplication of systematic searches for biomedical reviews. METHODS: We evaluated ASySD's performance on 5 unseen biomedical systematic search datasets of various sizes (1845-79,880 citations). We compared the performance of ASySD with EndNote's automated deduplication option and with the Systematic Review Assistant Deduplication Module (SRA-DM). RESULTS: ASySD identified more duplicates than either SRA-DM or EndNote, with a sensitivity in different datasets of 0.95 to 0.99. The false-positive rate was comparable to human performance, with a specificity of > 0.99. The tool took less than 1 h to identify and remove duplicates within each dataset. CONCLUSIONS: For duplicate removal in biomedical systematic reviews, ASySD is a highly sensitive, reliable, and time-saving tool. It is open source and freely available online as both an R package and a user-friendly web application.

The life and death of living systematic reviews: a methodological survey.

OBJECTIVES: The objectives of this study are to describe the characteristics of living systematic reviews (LSRs) and to understand their life cycles. STUDY DESIGN AND SETTING: We conducted a comprehensive search up to April 2021 then selected articles and abstracted data in duplicate and independently. We undertook descriptive analyses and calculated delay in version update and delay since the last published version. RESULTS: We included 76 eligible LSRs with a total of 279 eligible versions. The majority of LSRs was from the clinical field (70%), was COVID-19 related (63%), and had a funding source specified (62%). The median number of versions per LSR was 2 (interquartile range (IQR) 1-4; range 1-19). The median and IQR for the ratio of the actual period of update to the planned period of update was 1.12 (0.81; 1.71). Out of all reviews with a 'planned period of update' and at least one update (N = 19), eight LSRs (42%) had a period since last published version greater than 3 times the planned period of update. No LSR included a 'retirement notice' in their latest published version. CONCLUSION: While most LSR complied with the planned period of producing updates, a substantive proportion lagged since their last update.

In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving.

OBJECTIVES: The aim of this study is to describe and pilot a novel method for continuously identifying newly published trials relevant to a systematic review, enabled by combining artificial intelligence (AI) with human expertise. STUDY DESIGN AND SETTING: We used RobotReviewer LIVE to keep a review of COVID-19 vaccination trials updated from February to August 2021. We compared the papers identified by the system with those found by the conventional manual process by the review team. RESULTS: The manual update searches (last search date July 2021) retrieved 135 abstracts, of which 31 were included after screening (23% precision, 100% recall). By the same date, the automated system retrieved 56 abstracts, of which 31 were included after manual screening (55% precision, 100% recall). Key limitations of the system include that it is limited to searches of PubMed/MEDLINE, and considers only randomized controlled trial reports. We aim to address these limitations in future. The system is available as open-source software for further piloting and evaluation. CONCLUSION: Our system identified all relevant studies, reduced manual screening work, and enabled rolling updates on publication of new primary research.

Extension of the PRISMA 2020 statement for living systematic reviews (LSRs): protocol.

Background: While the PRISMA 2020 statement is intended to guide the reporting of original systematic reviews, updated systematic reviews, and living systematic reviews (LSRs), its explanation and elaboration document notes that additional considerations for updated systematic reviews and LSRs may need to be addressed. This paper reports the protocol for developing an extension of the PRISMA 2020 statement for LSRs. Methods: We will follow the EQUATOR Network's guidance for developing health research reporting guidelines. We will review the literature to identify possible items of the PRISMA 2020 checklist that need modification, as well as new items that need to be added. Then, we will survey representatives of different stakeholder groups for their views on the proposed modifications of the PRISMA 2020 checklist. We will summarize, present, and discuss the results of the survey in an online meeting, aiming to reach consensus on the content of the LSR extension. We will then draft the checklist, explanation and elaboration for each item, and flow diagram for the PRISMA 2020 extension. Then, we will share these initial documents with stakeholder representatives for final feedback and approval. Discussion: We anticipate that the PRISMA 2020 extension for LSRs will benefit LSR authors, editors, and peer reviewers of LSRs, as well as different users of LSRs, including guideline developers, policy makers, healthcare providers, patients, and other stakeholders.

A Living, Interactive Systematic Review and Network Meta-analysis of First-line Treatment of Metastatic Renal Cell Carcinoma.

CONTEXT: Identifying the most effective first-line treatment for metastatic renal cell carcinoma (mRCC) is challenging as rapidly evolving data quickly outdate the existing body of evidence, and current approaches to presenting the evidence in user-friendly formats are fraught with limitations. OBJECTIVE: To maintain living evidence for contemporary first-line treatment for previously untreated mRCC. EVIDENCE ACQUISITION: We have created a living, interactive systematic review (LISR) and network meta-analysis for first-line treatment of mRCC using data from randomized controlled trials comparing contemporary treatment options with single-agent tyrosine kinase inhibitors. We applied an advanced programming and artificial intelligence-assisted framework for evidence synthesis to create a living search strategy, facilitate screening and data extraction using a graphical user interface, automate the frequentist network meta-analysis, and display results in an interactive manner. EVIDENCE SYNTHESIS: As of October 22, 2020, the LISR includes data from 14 clinical trials. Baseline characteristics are summarized in an interactive table. The cabozantinib + nivolumab combination (CaboNivo) is ranked the highest for the overall response rate, progression-free survival, and overall survival, whereas ipilimumab + nivolumab (NivoIpi) is ranked the highest for achieving a complete response (CR). NivoIpi, and atezolizumab + bevacizumab (AteBev) were ranked highest (lowest toxicity) and CaboNivo ranked lowest for treatment-related adverse events (AEs). Network meta-analysis results are summarized as interactive tables and plots, GRADE summary-of-findings tables, and evidence maps. CONCLUSIONS: This innovative living and interactive review provides the best current evidence on the comparative effectiveness of multiple treatment options for patients with untreated mRCC. Trial-level comparisons suggest that CaboNivo is likely to cause more AEs but is ranked best for all efficacy outcomes, except NivoIpi offers the best chance of CR. Pembrolizumab + axitinib and NivoIpi are acceptable alternatives, except NivoIpi may not be preferred for patients with favorable risk. Although network meta-analysis provides rankings with statistical adjustments, there are inherent biases in cross-trial comparisons with sparse direct evidence that does not replace randomized comparisons. PATIENT SUMMARY: It is challenging to decide the best option among the several treatment combinations of immunotherapy and targeted treatments for newly diagnosed metastatic kidney cancer. We have created interactive evidence summaries of multiple treatment options that present the benefits and harms and evidence certainty for patient-important outcomes. This evidence is updated as soon as new studies are published.

Methods and guidance on conducting, reporting, publishing and appraising living systematic reviews: a scoping review protocol.

Background: The living systematic review (LSR) approach is based on an ongoing surveillance of the literature and continual updating. A few guidance documents address the conduct, reporting, publishing and appraisal of systematic reviews (SRs), but the methodology described is either not up-to date or not suitable for LSRs and misses additional LSR-specific considerations. The objective of this scoping review is to systematically collate methodological literature and guidance on how to conduct, report, publish and appraise the quality of LSRs. The scoping review will allow the mapping of the existing evidence on the topic to support LSRs authors seeking guidance and identify related gaps.  Methods: To achieve our objectives, we will conduct a scoping review to survey and evaluate existing evidence, using the standard scoping review methodology. We will search MEDLINE, EMBASE, and Cochrane using the OVID interface. The search strategy was developed by a researcher experienced in developing literature search strategies with the help of an information specialist. As for searching grey literature, we will seek existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses using the Lens.org website. Two review authors will extract and catalogue the study data on LSR methodological aspects into a standardized and pilot-tested data extraction form. The main categories will reflect proposed methods for (i) conducting LSRs, (ii) reporting of LSRs, (iii) publishing and (iv) appraising the quality of LSRs. Data synthesis and conclusion: By collecting these data from methodological surveys and papers, as well as existing guidance documents and handbooks on LSRs, we might identify specific issues and components lacking within current LSR methodology. Thus, the systematically obtained findings of the scoping review could be used as basis for the revision of existing methods tools on LSR, for instance a PRISMA statement extension for LSRs.

A Protocol of a Guideline to Establish the Evidence Ecosystem of Acupuncture.

This is a protocol for developing a guideline to establish the evidence ecosystem of acupuncture. It describes all steps that will be followed in line with the World Health Organization Handbook for Guideline Development and the Reporting Items for practice Guidelines in Healthcare (RIGHT). The key steps included guideline protocol development, guideline registration, systematic review of acupuncture evidence issues, systematic review of methods for establishing evidence ecosystem, survey of acupuncture stakeholders regarding potential acupuncture evidence issues, development of potential items for guidelines, Delphi method for guideline item development, consensus meeting, drafting guideline, peer review, approval, and publishing. This future guideline will help to establish evidence ecosystem of acupuncture, which will facilitate the application of acupuncture in clinical practice.

Apolipoprotein E4 Polymorphism and Outcomes from Traumatic Brain Injury: A Living Systematic Review and Meta-Analysis.

The mortality of traumatic brain injury (TBI) has been largely static despite advances in monitoring and imaging techniques. Substantial variance exists in outcome, not fully accounted for by baseline characteristics or injury severity, and genetic factors likely play a role in this variance. The aims of this systematic review were to examine the evidence for a link between the apolipoprotein E4 (APOE4) polymorphism and TBI outcomes and where possible, to quantify the effect size via meta-analysis. We searched EMBASE, MEDLINE, CINAHL, and gray literature in December 2017. We included studies of APOE genotype in relation to functional adult TBI outcomes. Methodological quality was assessed using the Quality in Prognostic Studies Risk of Bias Assessment Instrument and the prognostic studies adaptation of the Grading of Recommendations Assessment, Development and Evaluation tool. In addition, we contacted investigators and included an additional 160 patients whose data had not been made available for previous analyses, giving a total sample size of 2593 patients. Meta-analysis demonstrated higher odds of a favorable outcome following TBI in those not possessing an ApoE ɛ4 allele compared with ɛ4 carriers and homozygotes (odds ratio 1.39, 95% confidence interval 1.05 to 1.84; p = 0.02). The influence of APOE4 on neuropsychological functioning following TBI remained uncertain, with multiple conflicting studies. We conclude that the ApoE ɛ4 allele confers a small risk of poor outcome following TBI, with analysis by TBI severity not possible based on the currently available published data. Further research into the long-term neuropsychological impact and risk of dementia is warranted.

A New Approach to Evidence Synthesis in Traumatic Brain Injury: A Living Systematic Review.

Living systematic reviews (LSRs) are online summaries of health care research that are updated as new research becomes available. This new development in evidence synthesis is being trialled as part of the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) project. We will develop and sustain an international TBI knowledge community that maintains up-to-date, high quality LSRs of the current state of knowledge in the most important questions in TBI. Automatic search updates will be run three-monthly, and newly identified studies incorporated into the review. Review teams will seek to publish journal updates at regular intervals, with abridged updates available more frequently online. Future project stages include the integration of LSR and other study findings into "living" clinical practice guidance. It is hoped these efforts will go some way to bridging current temporal disconnects between evidence, guidelines, and practice in TBI.

Accumulation Bias in meta-analysis: the need to consider time in error control.

Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias -Accumulation Bias- and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependencies are also inevitable, since for science to accumulate efficiently, new research needs to be informed by past results. Here, we investigate various ways in which time influences error control in meta-analysis testing. We introduce an Accumulation Bias Framework that allows us to model a wide variety of practically occurring dependencies including study series accumulation, meta-analysis timing, and approaches to multiple testing in living systematic reviews. The strength of this framework is that it shows how all dependencies affect p-value-based tests in a similar manner. This leads to two main conclusions. First, Accumulation Bias is inevitable, and even if it can be approximated and accounted for, no valid p-value tests can be constructed. Second, tests based on likelihood ratios withstand Accumulation Bias: they provide bounds on error probabilities that remain valid despite the bias. We leave the reader with a choice between two proposals to consider time in error control: either treat individual (primary) studies and meta-analyses as two separate worlds - each with their own timing - or integrate individual studies in the meta-analysis world. Taking up likelihood ratios in either approach allows for valid tests that relate well to the accumulating nature of scientific knowledge. Likelihood ratios can be interpreted as betting profits, earned in previous studies and invested in new ones, while the meta-analyst is allowed to cash out at any time and advice against future studies.

Developing and applying a 'living guidelines' approach to WHO recommendations on maternal and perinatal health.

How should the WHO most efficiently keep its global recommendations up to date? In this article we describe how WHO developed and applied a 'living guidelines' approach to its maternal and perinatal health (MPH) recommendations, based on a systematic and continuous process of prioritisation and updating. Using this approach, 25 new or updated WHO MPH recommendations have been published in 2017-2018. The new approach helps WHO ensure its guidance is responsive to emerging evidence and remains up to date for end users.