Rapid review methods series: Guidance on the use of supportive software.

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group. Rapid reviews (RRs) use modified systematic review methods to accelerate the review process while maintaining systematic, transparent and reproducible methods. This paper guides how to use supportive software for RRs.We strongly encourage the use of supportive software throughout RR production. Specifically, we recommend (1) using collaborative online platforms that enable working in parallel, allow for real-time project management and centralise review details; (2) using automation software to support, but not entirely replace a human reviewer and human judgement and (3) being transparent in reporting the methodology and potential risk for bias due to the use of supportive software.

Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier.

BACKGROUND: This study developed, calibrated and evaluated a machine learning (ML) classifier designed to reduce study identification workload in maintaining the Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies. METHODS: A ML classifier for retrieving COVID-19 research studies (the 'Cochrane COVID-19 Study Classifier') was developed using a data set of title-abstract records 'included' in, or 'excluded' from, the CCSR up to 18th October 2020, manually labelled by information and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records 'included' in, or 'excluded' from, the CCSR between October 19 and December 2, 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records 'included' in, or 'excluded' from, the CCSR between the 4th and 19th of January 2021. RESULTS: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were 'included' in the CCSR). A classification threshold was set using 16,123 calibration records (6005 of which were 'included' in the CCSR) and the classifier had a precision of 0.52 in this data set at the target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2285 (98.9%) of 2310 eligible records but missed 25 (1%), with a precision of 0.638 and a net screening workload reduction of 24.1% (1113 records correctly excluded). CONCLUSIONS: The Cochrane COVID-19 Study Classifier reduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.

An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials.

BACKGROUND AND OBJECTIVES: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a "crowd" of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decision-making.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials. STUDY DESIGN AND SETTING: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with "quasi gold-standard" decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert. RESULTS: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials from ClinicalTrials.gov (CT ID), and 97.7% for the identification of RCTs from the International Clinical Trials Registry Platform (ICTRP ID). The specificity for each task was 99% for RCT ID, 98.6% for CT ID, and 99.1% for CT ICTRP ID. The capacity of the combined Crowd and machine learning workflow has increased fivefold in 6 years, compared with baseline. The proportion of records requiring expert resolution across the tasks ranged from 16.6% to 19.7%. CONCLUSION: Cochrane Crowd is sufficiently accurate and scalable to keep pace with the current rate of publication (and registration) of new primary studies. It has also proved to be a popular, efficient, and accurate way for a large number of people to play an important voluntary role in health evidence production. Cochrane Crowd is now an established part of Cochrane's effort to manage the deluge of primary research being produced.

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews.

OBJECTIVES: This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. METHODS: A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. RESULTS: The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. CONCLUSIONS: The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.

Cochrane's Linked Data Project: How it Can Advance our Understanding of Surrogate Endpoints.

Cochrane has developed a linked data infrastructure to make the evidence and data from its rich repositories more discoverable to facilitate evidence-based health decision-making. These annotated resources can enhance the study and understanding of biomarkers and surrogate endpoints.

Facilitating Web-Based Collaboration in Evidence Synthesis (TaskExchange): Development and Analysis.

BACKGROUND: The conduct and publication of scientific research are increasingly open and collaborative. There is growing interest in Web-based platforms that can effectively enable global, multidisciplinary scientific teams and foster networks of scientists in areas of shared research interest. Designed to facilitate Web-based collaboration in research evidence synthesis, TaskExchange highlights the potential of these kinds of platforms. OBJECTIVE: This paper describes the development, growth, and future of TaskExchange, a Web-based platform facilitating collaboration in research evidence synthesis. METHODS: The original purpose of TaskExchange was to create a platform that connected people who needed help with their Cochrane systematic reviews (rigorous syntheses of health research) with people who had the time and expertise to help. The scope of TaskExchange has now been expanded to include other evidence synthesis tasks, including guideline development. The development of TaskExchange was initially undertaken in 5 agile development phases with substantial user engagement. In each phase, software was iteratively deployed as it was developed and tested, enabling close cycles of development and refinement. RESULTS: TaskExchange enables users to browse and search tasks and members by keyword or nested filters, post and respond to tasks, sign up to notification emails, and acknowledge the work of TaskExchange members. The pilot platform has been open access since August 2016, has over 2300 members, and has hosted more than 630 tasks, covering a wide range of research synthesis-related tasks. Response rates are consistently over 75%, and user feedback has been positive. CONCLUSIONS: TaskExchange demonstrates the potential for new technologies to support Web-based collaboration in health research. Development of a relatively simple platform for peer-to-peer exchange has provided opportunities for systematic reviewers to get their reviews completed more quickly and provides an effective pathway for people to join the global health evidence community.

A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery.

OBJECTIVES: There is increasing recognition that insufficient attention has been paid to the choice of outcomes measured in clinical trials. The lack of a standardized outcome classification system results in inconsistencies due to ambiguity and variation in how outcomes are described across different studies. Being able to classify by outcome would increase efficiency in searching sources such as clinical trial registries, patient registries, the Cochrane Database of Systematic Reviews, and the Core Outcome Measures in Effectiveness Trials (COMET) database of core outcome sets (COS), thus aiding knowledge discovery. STUDY DESIGN AND SETTING: A literature review was carried out to determine existing outcome classification systems, none of which were sufficiently comprehensive or granular for classification of all potential outcomes from clinical trials. A new taxonomy for outcome classification was developed, and as proof of principle, outcomes extracted from all published COS in the COMET database, selected Cochrane reviews, and clinical trial registry entries were classified using this new system. RESULTS: Application of this new taxonomy to COS in the COMET database revealed that 274/299 (92%) COS include at least one physiological outcome, whereas only 177 (59%) include at least one measure of impact (global quality of life or some measure of functioning) and only 105 (35%) made reference to adverse events. CONCLUSIONS: This outcome taxonomy will be used to annotate outcomes included in COS within the COMET database and is currently being piloted for use in Cochrane Reviews within the Cochrane Linked Data Project. Wider implementation of this standard taxonomy in trial and systematic review databases and registries will further promote efficient searching, reporting, and classification of trial outcomes.

Living systematic reviews: 2. Combining human and machine effort.

New approaches to evidence synthesis, which use human effort and machine automation in mutually reinforcing ways, can enhance the feasibility and sustainability of living systematic reviews. Human effort is a scarce and valuable resource, required when automation is impossible or undesirable, and includes contributions from online communities ("crowds") as well as more conventional contributions from review authors and information specialists. Automation can assist with some systematic review tasks, including searching, eligibility assessment, identification and retrieval of full-text reports, extraction of data, and risk of bias assessment. Workflows can be developed in which human effort and machine automation can each enable the other to operate in more effective and efficient ways, offering substantial enhancement to the productivity of systematic reviews. This paper describes and discusses the potential-and limitations-of new ways of undertaking specific tasks in living systematic reviews, identifying areas where these human/machine "technologies" are already in use, and where further research and development is needed. While the context is living systematic reviews, many of these enabling technologies apply equally to standard approaches to systematic reviewing.

Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.

The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action. Incremental advances in conventional review updating are unlikely to lead to substantial improvements in review currency. A new approach is needed. We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence. Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication. Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem.

Long-acting beta2-agonists for chronic obstructive pulmonary disease.

BACKGROUND: Chronic obstructive pulmonary disease (COPD) is a respiratory disease that causes progressive symptoms of breathlessness, cough and mucus build-up. It is the fourth or fifth most common cause of death worldwide and is associated with significant healthcare costs.Inhaled long-acting beta2-agonists (LABAs) are widely prescribed to manage the symptoms of COPD when short-acting agents alone are no longer sufficient. Twice-daily treatment with an inhaled LABA is aimed at relieving symptoms, improving exercise tolerance and quality of life, slowing decline and even improving lung function and preventing and treating exacerbations. OBJECTIVES: To assess the effects of twice-daily long-acting beta2-agonists compared with placebo for patients with COPD on the basis of clinically important endpoints, primarily quality of life and COPD exacerbations. SEARCH METHODS: We searched the Cochrane Airways Group trials register, ClinicalTrials.gov and manufacturers' websites in June 2013. SELECTION CRITERIA: Parallel, randomised controlled trials (RCTs) recruiting populations of patients with chronic obstructive pulmonary disease. Studies were required to be at least 12 weeks in duration and designed to assess the safety and efficacy of a long-acting beta2-agonist against placebo. DATA COLLECTION AND ANALYSIS: Data and characteristics were extracted independently by two review authors, and each study was assessed for potential sources of bias. Data for all outcomes were pooled and subgrouped by LABA agent (formoterol 12 µg, formoterol 24 µg and salmeterol 50 µg) and then were separately analysed by LABA agent and subgrouped by trial duration. Sensitivity analyses were conducted for the proportion of participants taking inhaled corticosteroids and for studies with high or uneven rates of attrition. MAIN RESULTS: Twenty-six RCTs met the inclusion criteria, randomly assigning 14,939 people with COPD to receive twice-daily LABA or placebo. Study duration ranged from three months to three years; the median duration was six months. Participants were more often male with moderate to severe symptoms at randomisation; mean forced expiratory volume in 1 second (FEV1) was between 33% and 55% predicted normal in the studies, and mean St George's Respiratory Questionnaire score (SGRQ) ranged from 44 to 55 when reported.Moderate-quality evidence showed that LABA treatment improved quality of life on the SGRQ (mean difference (MD) -2.32, 95% confidence interval (CI) -3.09 to -1.54; I(2) = 50%; 17 trials including 11,397 people) and reduced the number of exacerbations requiring hospitalisation (odds ratio (OR) 0.73, 95% CI 0.56 to 0.95; I(2) = 10%; seven trials including 3804 people). In absolute terms, 18 fewer people per 1000 were hospitalised as the result of an exacerbation while receiving LABA therapy over a weighted mean of 7 months (95% CI 3 to 31 fewer). Scores were also improved on the Chronic Respiratory Disease Questionnaire (CRQ), and more people receiving LABA treatment showed clinically important improvement of at least four points on the SGRQ.The number of people who had exacerbations requiring a course of oral steroids or antibiotics was also lower among those taking LABA (52 fewer per 1000 treated over 8 months; 95% CI 24 to 78 fewer, moderate quality evidence).Mortality was low, and combined findings of all studies showed that LABA therapy did not significantly affect mortality (OR 0.90, 95% CI 0.75 to 1.08; I(2) = 21%; 23 trials including 14,079 people, moderate quality evidence). LABA therapy did not affect the rate of serious adverse events (OR 0.97, 95% CI 0.83 to 1.14; I(2) = 34%, moderate quality evidence), although there was significant unexplained heterogeneity, especially between the two formoterol doses.LABA therapy improved predose FEV1 by 73 mL more than placebo (95% CI 48 to 98; I(2) = 71%, low quality evidence), and people were more likely to withdraw from placebo than from LABA therapy (OR 0.74, 95% CI 0.69 to 0.80; I(2) = 0%). Higher rates of withdrawal in the placebo arm may reduce our confidence in some results, but the disparity is more likely to reduce the magnitude of difference between LABA and placebo than inflate the true effect; removing studies at highest risk of bias on the basis of high and unbalanced attrition did not change conclusions for the primary outcomes. AUTHORS' CONCLUSIONS: Moderate-quality evidence from 26 studies showed that inhaled long-acting beta2-agonists are effective over the medium and long term for patients with moderate to severe COPD. Their use is associated with improved quality of life and reduced exacerbations, including those requiring hospitalisation. Overall, findings showed that inhaled LABAs did not significantly reduce mortality or serious adverse events.

Can decision making in general surgery be based on evidence? An empirical study of Cochrane Reviews.

BACKGROUND: This empirical study analyzes the current status of Cochrane Reviews (CRs) and their strength of recommendation for evidence-based decision making in the field of general surgery. METHODS: Systematic literature search of the Cochrane Database of Systematic Reviews and the Cochrane Collaboration's homepage to identify available CRs on surgical topics. Quantitative and qualitative characteristics, utilization, and formulated treatment recommendations were evaluated by 2 independent reviewers. Association of review characteristics with treatment recommendation was analyzed using univariate and multivariate logistic regression models. RESULTS: Ninety-three CRs, including 1,403 primary studies and 246,473 patients, were identified. Mean number of included primary studies per CR was 15.1 (standard deviation [SD] 14.5) including 2,650 (SD 3,340) study patients. Two and a half (SD 8.3) nonrandomized trials were included per analyzed CR. Seventy-two (77%) CRs were published or updated in 2005 or later. Explicit treatment recommendations were given in 45 (48%). Presence of a treatment recommendation was associated with the number of included primary studies and the proportion of randomized studies. Utilization of surgical CRs remained low and showed large inter-country differences. The most surgical CRs were accessed in UK, USA, and Australia, followed by several Western and Eastern European countries. CONCLUSION: Only a minority of available CRs address surgical questions and their current usage is low. Instead of unsystematically increasing the number of surgical CRs it would be far more efficient to focus the review process on relevant surgical questions. Prioritization of CRs needs valid methods which should be developed by the scientific surgical community.