Methods used to conceptualize dimensions of health equity impacts of public health interventions in systematic reviews.

OBJECTIVES: Our aims were to, first, identify and summarize the use of methods, frameworks, and tools as a conceptual basis for investigating dimensions of equity impacts of public health interventions in systematic reviews including an equity focus. These include PROGRESS-Plus, which identifies key sociodemographic characteristics that determine health outcomes. Second, we aimed to document challenges and opportunities encountered in the application of such methods, as reported in systematic reviews. STUDY DESIGN AND SETTING: We conducted a methodological study, comprising an overview of systematic reviews with a focus on, or that aimed to assess, the equity impacts of public health interventions. We used electronic searches of the Cochrane Database of Systematic Reviews, the Database of Promoting Health Effectiveness Reviews (DoPHER), and the Finding Accessible Inequalities Research in Public Health Database, supplemented with automated searches of the OpenAlex dataset. An active learning algorithm was used to prioritize title-abstract records for manual screening against eligibility criteria. We extracted and analyzed a core dataset from a purposively selected sample of reviews, to summarize key characteristics and approaches to conceptualizing investigations of equity. RESULTS: We assessed 322 full-text reports for eligibility, from which we included 120 reports of systematic reviews. PROGRESS-Plus was the only formalized framework used to conceptualize dimensions of equity impacts. Most reviews were able to apply their intended methods to at least some degree. Where intended methods were unable to be applied fully, this was usually because primary research studies did not report the necessary information. A general rationale for focusing on equity impacts was often included, but few reviews explicitly justified their focus on (or exclusion of) specific dimensions. In addition to practical challenges such as data not being available, authors highlighted significant measurement and conceptual issues with applying these methods which may impair the ability to investigate and interpret differential impacts within and between studies. These issues included investigating constructs that lack standardized operationalization and measurement, and the complex nature of differential impacts, with dimensions that may interact with one another, as well as with particular temporal, personal, social or geographic contexts. CONCLUSION: PROGRESS-Plus is the predominant framework used in systematic reviews to conceptualize differential impacts of public health interventions by dimensions of equity. It appears sufficiently broad to encompass dimensions of equity examined in most investigations of this kind. However, PROGRESS-Plus does not necessarily ensure or guide critical thinking about more complex pathways, including interactions between dimensions of equity, and with wider contextual factors, and important practical, measurement and conceptual challenges remain. The findings from investigations of equity impacts in systematic reviews could be made more useful through more explicitly rationalized and considered approaches to the design, conduct and reporting of both primary research and the reviews themselves.

GRADE guidance 23: considering cost-effectiveness evidence in moving from evidence to health-related recommendations.

BACKGROUND: This is the 23rd in a series of articles describing the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to grading the certainty of evidence and strength of recommendations for systematic reviews, health technology assessments, and clinical guideline development. OBJECTIVES: We outline how resource utilization and cost-effectiveness analyses are integrated into health-related recommendations, using the GRADE Evidence to Decision (EtD) frameworks. STUDY DESIGN AND SETTING: Through iterative discussions and refinement, in-person, and online meetings, and through e-mail communication, we developed draft guidance to incorporate economic evidence in the formulation of health-related recommendations. We developed scenarios to operationalize the guidance. We presented a summary of the results to members of the GRADE Economic Evaluation Project Group. RESULTS: We describe how to estimate the cost of preventing (or achieving) an event to inform assessments of cost-effectiveness of alternative treatments, when there are no published economic evaluations. Evidence profiles and Summary of Findings tables based on systematic reviews of cost-effectiveness analyses can be created to provide top-level summaries of results and quality of multiple published economic evaluations. We also describe how this information could be integrated in GRADE's EtD frameworks to inform health-related recommendations. Three scenarios representing various levels of available cost-effectiveness evidence were used to illustrate the integration process. CONCLUSION: This GRADE guidance provides practical information for presenting cost-effectiveness data and its integration in the development of health-related recommendations, using the EtD frameworks.

Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier.

BACKGROUND: This study developed, calibrated and evaluated a machine learning (ML) classifier designed to reduce study identification workload in maintaining the Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies. METHODS: A ML classifier for retrieving COVID-19 research studies (the 'Cochrane COVID-19 Study Classifier') was developed using a data set of title-abstract records 'included' in, or 'excluded' from, the CCSR up to 18th October 2020, manually labelled by information and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records 'included' in, or 'excluded' from, the CCSR between October 19 and December 2, 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records 'included' in, or 'excluded' from, the CCSR between the 4th and 19th of January 2021. RESULTS: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were 'included' in the CCSR). A classification threshold was set using 16,123 calibration records (6005 of which were 'included' in the CCSR) and the classifier had a precision of 0.52 in this data set at the target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2285 (98.9%) of 2310 eligible records but missed 25 (1%), with a precision of 0.638 and a net screening workload reduction of 24.1% (1113 records correctly excluded). CONCLUSIONS: The Cochrane COVID-19 Study Classifier reduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.

An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials.

BACKGROUND AND OBJECTIVES: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a "crowd" of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decision-making.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials. STUDY DESIGN AND SETTING: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with "quasi gold-standard" decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert. RESULTS: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials from ClinicalTrials.gov (CT ID), and 97.7% for the identification of RCTs from the International Clinical Trials Registry Platform (ICTRP ID). The specificity for each task was 99% for RCT ID, 98.6% for CT ID, and 99.1% for CT ICTRP ID. The capacity of the combined Crowd and machine learning workflow has increased fivefold in 6 years, compared with baseline. The proportion of records requiring expert resolution across the tasks ranged from 16.6% to 19.7%. CONCLUSION: Cochrane Crowd is sufficiently accurate and scalable to keep pace with the current rate of publication (and registration) of new primary studies. It has also proved to be a popular, efficient, and accurate way for a large number of people to play an important voluntary role in health evidence production. Cochrane Crowd is now an established part of Cochrane's effort to manage the deluge of primary research being produced.

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making.

OBJECTIVES: The objective of the study is to present the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty of evidence from modeling studies (i.e., certainty associated with model outputs). STUDY DESIGN AND SETTING: Expert consultations and an international multidisciplinary workshop informed development of a conceptual approach to assessing the certainty of evidence from models within the context of systematic reviews, health technology assessments, and health care decisions. The discussions also clarified selected concepts and terminology used in the GRADE approach and by the modeling community. Feedback from experts in a broad range of modeling and health care disciplines addressed the content validity of the approach. RESULTS: Workshop participants agreed that the domains determining the certainty of evidence previously identified in the GRADE approach (risk of bias, indirectness, inconsistency, imprecision, reporting bias, magnitude of an effect, dose-response relation, and the direction of residual confounding) also apply when assessing the certainty of evidence from models. The assessment depends on the nature of model inputs and the model itself and on whether one is evaluating evidence from a single model or multiple models. We propose a framework for selecting the best available evidence from models: 1) developing de novo, a model specific to the situation of interest, 2) identifying an existing model, the outputs of which provide the highest certainty evidence for the situation of interest, either "off-the-shelf" or after adaptation, and 3) using outputs from multiple models. We also present a summary of preferred terminology to facilitate communication among modeling and health care disciplines. CONCLUSION: This conceptual GRADE approach provides a framework for using evidence from models in health decision-making and the assessment of certainty of evidence from a model or models. The GRADE Working Group and the modeling community are currently developing the detailed methods and related guidance for assessing specific domains determining the certainty of evidence from models across health care-related disciplines (e.g., therapeutic decision-making, toxicology, environmental health, and health economics).

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews.

OBJECTIVES: This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. METHODS: A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. RESULTS: The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. CONCLUSIONS: The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.

Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research.

Background: Identifying new, eligible studies for integration into living systematic reviews and maps usually relies on conventional Boolean updating searches of multiple databases and manual processing of the updated results. Automated searches of one, comprehensive, continuously updated source, with adjunctive machine learning, could enable more efficient searching, selection and prioritisation workflows for updating (living) reviews and maps, though research is needed to establish this. Microsoft Academic Graph (MAG) is a potentially comprehensive single source which also contains metadata that can be used in machine learning to help efficiently identify eligible studies. This study sought to establish whether: (a) MAG was a sufficiently sensitive single source to maintain our living map of COVID-19 research; and (b) eligible records could be identified with an acceptably high level of specificity. Methods: We conducted an eight-arm cost-effectiveness analysis to assess the costs, recall and precision of semi-automated workflows, incorporating MAG with adjunctive machine learning, for continually updating our living map. Resource use data (time use) were collected from information specialists and other researchers involved in map production. Our systematic review software, EPPI-Reviewer, was adapted to incorporate MAG and associated machine learning workflows, and also used to collect data on recall, precision, and manual screening workload. Results: The semi-automated MAG-enabled workflow dominated conventional workflows in both the base case and sensitivity analyses. At one month our MAG-enabled workflow with machine learning, active learning and fixed screening targets identified 469 additional, eligible articles for inclusion in our living map, and cost £3,179 GBP per week less, compared with conventional methods relying on Boolean searches of Medline and Embase. Conclusions: We were able to increase recall and coverage of a large living map, whilst reducing its production costs. This finding is likely to be transferrable to OpenAlex, MAG's successor database platform.

Altering the availability or proximity of food, alcohol, and tobacco products to change their selection and consumption.

BACKGROUND: Overconsumption of food, alcohol, and tobacco products increases the risk of non-communicable diseases. Interventions to change characteristics of physical micro-environments where people may select or consume these products - including shops, restaurants, workplaces, and schools - are of considerable public health policy and research interest. This review addresses two types of intervention within such environments: altering the availability (the range and/or amount of options) of these products, or their proximity (the distance at which they are positioned) to potential consumers. OBJECTIVES: 1. To assess the impact on selection and consumption of altering the availability or proximity of (a) food (including non-alcoholic beverages), (b) alcohol, and (c) tobacco products.2. To assess the extent to which the impact of these interventions is modified by characteristics of: i. studies, ii. interventions, and iii. SEARCH METHODS: We searched CENTRAL, MEDLINE, Embase, PsycINFO, and seven other published or grey literature databases, as well as trial registries and key websites, up to 23 July 2018, followed by citation searches. SELECTION CRITERIA: We included randomised controlled trials with between-participants (parallel group) or within-participants (cross-over) designs. Eligible studies compared effects of exposure to at least two different levels of availability of a product or its proximity, and included a measure of selection or consumption of the manipulated product. DATA COLLECTION AND ANALYSIS: We used a novel semi-automated screening workflow and applied standard Cochrane methods to select eligible studies, collect data, and assess risk of bias. In separate analyses for availability interventions and proximity interventions, we combined results using random-effects meta-analysis and meta-regression models to estimate summary effect sizes (as standardised mean differences (SMDs)) and to investigate associations between summary effect sizes and selected study, intervention, or participant characteristics. We rated the certainty of evidence for each outcome using GRADE. MAIN RESULTS: We included 24 studies, with the majority (20/24) giving concerns about risk of bias. All of the included studies investigated food products; none investigated alcohol or tobacco. The majority were conducted in laboratory settings (14/24), with adult participants (17/24), and used between-participants designs (19/24). All studies were conducted in high-income countries, predominantly in the USA (14/24).Six studies investigated availability interventions, of which two changed the absolute number of different options available, and four altered the relative proportion of less-healthy (to healthier) options. Most studies (4/6) manipulated snack foods or drinks. For selection outcomes, meta-analysis of three comparisons from three studies (n = 154) found that exposure to fewer options resulted in a large reduction in selection of the targeted food(s): SMD -1.13 (95% confidence interval (CI) -1.90 to -0.37) (low certainty evidence). For consumption outcomes, meta-analysis of three comparisons from two studies (n = 150) found that exposure to fewer options resulted in a moderate reduction in consumption of those foods, but with considerable uncertainty: SMD -0.55 (95% CI -1.27 to 0.18) (low certainty evidence).Eighteen studies investigated proximity interventions. Most (14/18) changed the distance at which a snack food or drink was placed from the participants, whilst four studies changed the order of meal components encountered along a line. For selection outcomes, only one study with one comparison (n = 41) was identified, which found that food placed farther away resulted in a moderate reduction in its selection: SMD -0.65 (95% CI -1.29 to -0.01) (very low certainty evidence). For consumption outcomes, meta-analysis of 15 comparisons from 12 studies (n = 1098) found that exposure to food placed farther away resulted in a moderate reduction in its consumption: SMD -0.60 (95% CI -0.84 to -0.36) (low certainty evidence). Meta-regression analyses indicated that this effect was greater: the farther away the product was placed; when only the targeted product(s) was available; when participants were of low deprivation status; and when the study was at high risk of bias. AUTHORS' CONCLUSIONS: The current evidence suggests that changing the number of available food options or altering the positioning of foods could contribute to meaningful changes in behaviour, justifying policy actions to promote such changes within food environments. However, the certainty of this evidence as assessed by GRADE is low or very low. To enable more certain and generalisable conclusions about these potentially important effects, further research is warranted in real-world settings, intervening across a wider range of foods - as well as alcohol and tobacco products - and over sustained time periods.

Altering the availability or proximity of food, alcohol, and tobacco products to change their selection and consumption.

BACKGROUND: Overconsumption of food, alcohol, and tobacco products increases the risk of non-communicable diseases. Interventions to change characteristics of physical micro-environments where people may select or consume these products - including shops, restaurants, workplaces, and schools - are of considerable public health policy and research interest. This review addresses two types of intervention within such environments: altering the availability (the range and/or amount of options) of these products, or their proximity (the distance at which they are positioned) to potential consumers. OBJECTIVES: 1. To assess the impact on selection and consumption of altering the availability or proximity of (a) food (including non-alcoholic beverages), (b) alcohol, and (c) tobacco products.2. To assess the extent to which the impact of these interventions is modified by characteristics of: i. studies, ii. interventions, and iii. SEARCH METHODS: We searched CENTRAL, MEDLINE, Embase, PsycINFO, and seven other published or grey literature databases, as well as trial registries and key websites, up to 23 July 2018, followed by citation searches. SELECTION CRITERIA: We included randomised controlled trials with between-participants (parallel group) or within-participants (cross-over) designs. Eligible studies compared effects of exposure to at least two different levels of availability of a product or its proximity, and included a measure of selection or consumption of the manipulated product. DATA COLLECTION AND ANALYSIS: We used a novel semi-automated screening workflow and applied standard Cochrane methods to select eligible studies, collect data, and assess risk of bias. In separate analyses for availability interventions and proximity interventions, we combined results using random-effects meta-analysis and meta-regression models to estimate summary effect sizes (as standardised mean differences (SMDs)) and to investigate associations between summary effect sizes and selected study, intervention, or participant characteristics. We rated the certainty of evidence for each outcome using GRADE. MAIN RESULTS: We included 24 studies, with the majority (20/24) giving concerns about risk of bias. All of the included studies investigated food products; none investigated alcohol or tobacco. The majority were conducted in laboratory settings (14/24), with adult participants (17/24), and used between-participants designs (19/24). All studies were conducted in high-income countries, predominantly in the USA (14/24).Six studies investigated availability interventions, of which two changed the absolute number of different options available, and four altered the relative proportion of less-healthy (to healthier) options. Most studies (4/6) manipulated snack foods or drinks. For selection outcomes, meta-analysis of three comparisons from three studies (n = 154) found that exposure to fewer options resulted in a large reduction in selection of the targeted food(s): SMD -1.13 (95% confidence interval (CI) -1.90 to -0.37) (low certainty evidence). For consumption outcomes, meta-analysis of three comparisons from two studies (n = 150) found that exposure to fewer options resulted in a moderate reduction in consumption of those foods, but with considerable uncertainty: SMD -0.55 (95% CI -1.27 to 0.18) (low certainty evidence).Eighteen studies investigated proximity interventions. Most (14/18) changed the distance at which a snack food or drink was placed from the participants, whilst four studies changed the order of meal components encountered along a line. For selection outcomes, only one study with one comparison (n = 41) was identified, which found that food placed farther away resulted in a moderate reduction in its selection: SMD -0.65 (95% CI -1.29 to -0.01) (very low certainty evidence). For consumption outcomes, meta-analysis of 15 comparisons from 12 studies (n = 1098) found that exposure to food placed farther away resulted in a moderate reduction in its consumption: SMD -0.60 (95% CI -0.84 to -0.36) (low certainty evidence). Meta-regression analyses indicated that this effect was greater: the farther away the product was placed; when only the targeted product(s) was available; when participants were of low deprivation status; and when the study was at high risk of bias. AUTHORS' CONCLUSIONS: The current evidence suggests that changing the number of available food options or altering the positioning of foods could contribute to meaningful changes in behaviour, justifying policy actions to promote such changes within food environments. However, the certainty of this evidence as assessed by GRADE is low or very low. To enable more certain and generalisable conclusions about these potentially important effects, further research is warranted in real-world settings, intervening across a wider range of foods - as well as alcohol and tobacco products - and over sustained time periods.

The crime, mental health, and economic impacts of prearrest diversion of people with mental health problems: A systematic review.

BACKGROUND: Prearrest diversion strategies are being adopted across the Western world, enabling the police to identify and divert people suspected of having mental disorder towards health and community services rather than the criminal justice system. AIMS: To quantify longer-term criminal justice and mental health outcomes after prearrest diversion of people with suspected mental disorder and consider economic correlates. METHODS: A systematic review of published literature on longer term outcomes after prearrest diversion. RESULTS: Only two quasi-experimental studies, with four independent samples, could be included. Findings for criminal and mental health outcomes were inconclusive, but potential for adverse outcomes was identified. Ten studies with cost data suggested that prearrest diversion can lead to overall cost savings. CONCLUSIONS: There is still inadequate evidence on which to base prearrest diversion programmes. Although some benefits have been identified by the review, so have possible harms. Future research and funding strategies must build in high-quality, systematic evaluation of outcomes before implementing a theoretically attractive strategy more widely.

Still moving toward automation of the systematic review process: a summary of discussions at the third meeting of the International Collaboration for Automation of Systematic Reviews (ICASR).

The third meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 17-18 October 2017 in London, England. ICASR is an interdisciplinary group whose goal is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. The group seeks to facilitate the development and widespread acceptance of automated techniques for systematic reviews. The meeting's conclusion was that the most pressing needs at present are to develop approaches for validating currently available tools and to provide increased access to curated corpora that can be used for validation. To that end, ICASR's short-term goals in 2018-2019 are to propose and publish protocols for key tasks in systematic reviews and to develop an approach for sharing curated corpora for validating the automation of the key tasks.

Incentives for climate mitigation in the land use sector-the effects of payment for environmental services on environmental and socioeconomic outcomes in low- and middle-income countries: A mixed-methods systematic review.

Unsustainable practices in the land use sector contribute to climate change through the release of greenhouse gases. Payment for environmental services (PESs) provide economic incentives to reduce the negative environmental impacts of land use and are a popular approach to mitigate climate change in low- and middle-income countries. Some PES programmes also aim to improve socioeconomic outcomes and reduce poverty. This systematic review examines the effect of programmes on environmental and socioeconomic outcomes. We identified 44 quantitative impact evaluations and 60 qualitative studies of PES programmes for inclusion in the review, to assess both the effects of PES and identify context, design and implementation features that may influence PES effectiveness. The studies covered 18 programmes from 12 countries in Latin America and the Caribbean, East Asia and Pacific, South Asia and Sub-Saharan Africa. The review finds that PES may increase household income, reduce deforestation and improve forest cover, but the findings are, however, based on low and very low quality evidence from a small number of programmes and should be treated with caution. Qualitative evidence indicates that several factors influence whether PES programmes are likely to be effective in different contexts and suggests that the inclusion of strong governance structures and the effective targeting of both locations and participants may improve intervention effectiveness. Funders, implementing agencies and researchers should collaborate to develop a coordinated programme of rigorous, mixed-methods impact evaluation implemented across contexts. Until such evidence is available, PES programmes remain a high-risk strategy for climate change mitigation.

Living systematic review: 1. Introduction-the why, what, when, and how.

Systematic reviews are difficult to keep up to date, but failure to do so leads to a decay in review currency, accuracy, and utility. We are developing a novel approach to systematic review updating termed "Living systematic review" (LSR): systematic reviews that are continually updated, incorporating relevant new evidence as it becomes available. LSRs may be particularly important in fields where research evidence is emerging rapidly, current evidence is uncertain, and new research may change policy or practice decisions. We hypothesize that a continual approach to updating will achieve greater currency and validity, and increase the benefits to end users, with feasible resource requirements over time.

Living systematic reviews: 2. Combining human and machine effort.

New approaches to evidence synthesis, which use human effort and machine automation in mutually reinforcing ways, can enhance the feasibility and sustainability of living systematic reviews. Human effort is a scarce and valuable resource, required when automation is impossible or undesirable, and includes contributions from online communities ("crowds") as well as more conventional contributions from review authors and information specialists. Automation can assist with some systematic review tasks, including searching, eligibility assessment, identification and retrieval of full-text reports, extraction of data, and risk of bias assessment. Workflows can be developed in which human effort and machine automation can each enable the other to operate in more effective and efficient ways, offering substantial enhancement to the productivity of systematic reviews. This paper describes and discusses the potential-and limitations-of new ways of undertaking specific tasks in living systematic reviews, identifying areas where these human/machine "technologies" are already in use, and where further research and development is needed. While the context is living systematic reviews, many of these enabling technologies apply equally to standard approaches to systematic reviewing.

Quasi-experimental study designs series-paper 1: introduction: two historical lineages.

OBJECTIVES: The objective of this study was to contrast the historical development of experiments and quasi-experiments and provide the motivation for a journal series on quasi-experimental designs in health research. STUDY DESIGN AND SETTING: A short historical narrative, with concrete examples, and arguments based on an understanding of the practice of health research and evidence synthesis. RESULTS: Health research has played a key role in developing today's gold standard for causal inference-the randomized controlled multiply blinded trial. Historically, allocation approaches developed from convenience and purposive allocation to alternate and, finally, to random allocation. This development was motivated both by concerns for manipulation in allocation as well as statistical and theoretical developments demonstrating the power of randomization in creating counterfactuals for causal inference. In contrast to the sequential development of experiments, quasi-experiments originated at very different points in time, from very different scientific perspectives, and with frequent and long interruptions in their methodological development. Health researchers have only recently started to recognize the value of quasi-experiments for generating novel insights on causal relationships. CONCLUSION: While quasi-experiments are unlikely to replace experiments in generating the efficacy and safety evidence required for clinical guidelines and regulatory approval of medical technologies, quasi-experiments can play an important role in establishing the effectiveness of health care practice, programs, and policies. The papers in this series describe and discuss a range of important issues in utilizing quasi-experimental designs for primary research and quasi-experimental results for evidence synthesis.

Quasi-experimental study designs series-paper 4: uses and value.

Quasi-experimental studies are increasingly used to establish causal relationships in epidemiology and health systems research. Quasi-experimental studies offer important opportunities to increase and improve evidence on causal effects: (1) they can generate causal evidence when randomized controlled trials are impossible; (2) they typically generate causal evidence with a high degree of external validity; (3) they avoid the threats to internal validity that arise when participants in nonblinded experiments change their behavior in response to the experimental assignment to either intervention or control arm (such as compensatory rivalry or resentful demoralization); (4) they are often well suited to generate causal evidence on long-term health outcomes of an intervention, as well as nonhealth outcomes such as economic and social consequences; and (5) they can often generate evidence faster and at lower cost than experiments and other intervention studies.

Quasi-experimental study designs series-paper 9: collecting data from quasi-experimental studies.

OBJECTIVE: To identify variables that must be coded when synthesizing primary studies that use quasi-experimental designs. STUDY DESIGN AND SETTING: All quasi-experimental (QE) designs. RESULTS: When designing a systematic review of QE studies, potential sources of heterogeneity-both theory-based and methodological-must be identified. We outline key components of inclusion criteria for syntheses of quasi-experimental studies. We provide recommendations for coding content-relevant and methodological variables and outlined the distinction between bivariate effect sizes and partial (i.e., adjusted) effect sizes. Designs used and controls used are viewed as of greatest importance. Potential sources of bias and confounding are also addressed. CONCLUSION: Careful consideration must be given to inclusion criteria and the coding of theoretical and methodological variables during the design phase of a synthesis of quasi-experimental studies. The success of the meta-regression analysis relies on the data available to the meta-analyst. Omission of critical moderator variables (i.e., effect modifiers) will undermine the conclusions of a meta-analysis.

Quasi-experimental study designs series-paper 8: identifying quasi-experimental studies to inform systematic reviews.

OBJECTIVE: This article reviews the available evidence and guidance on methods to identify reports of quasi-experimental (QE) studies to inform systematic reviews of health care, public health, international development, education, crime and justice, and social welfare. STUDY DESIGN AND SETTING: Research, guidance, and examples of search strategies were identified by searching a range of databases, key guidance documents, selected reviews, conference proceedings, and personal communication. Current practice and research evidence were summarized. RESULTS: Four thousand nine hundred twenty-four records were retrieved by database searches, and additional documents were obtained by other searches. QE studies are challenging to identify efficiently because they have no standardized nomenclature and may be indexed in various ways. Reliable search filters are not available. There is a lack of specific resources devoted to collecting QE studies and little evidence on where best to search. CONCLUSION: Searches to identify QE studies should search a range of resources and, until indexing improves, use strategies that focus on the topic rather than the study design. Better definitions, better indexing in databases, prospective registers, and reporting guidance are required to improve the retrieval of QE studies and promote systematic reviews of what works based on the evidence from such studies.

Multi-arm Cost-Effectiveness Analysis (CEA) comparing different durations of adjuvant trastuzumab in early breast cancer, from the English NHS payer perspective.

BACKGROUND: Trastuzumab improves survival in HER2+ breast cancer patients, with some evidence of adverse cardiac side effects. Current recommendations are to give adjuvant trastuzumab for one year or until recurrence, although trastuzumab treatment for only 9 or 10 weeks has shown similar survival rates to 12-month treatment. We present here a multi-arm joint analysis examining the relative cost-effectiveness of different durations of adjuvant trastuzumab. METHODS AND FINDINGS: Network meta-analysis (NMA) was used to examine which trials' data to include in the cost-effectiveness analysis (CEA). A network using FinHer (9 weeks vs. zero) and BCIRG006 (12 months vs. zero) trials offered the only jointly randomisable network so these trials were used in the CEA. The 3-arm CEA compared costs and quality-adjusted life-years (QALYs) associated with zero, 9-week and 12-month adjuvant trastuzumab durations in early breast cancer, using a decision tree followed by a Markov model that extrapolated the results to a lifetime time horizon. Pairwise incremental cost-effectiveness ratios (ICERs) were also calculated for each pair of regimens and used in budget impact analysis, and the Bucher method was used to check face validity of the findings. Addition of the PHARE trial (6 months vs. 12 months) to the network, in order to create a 4-arm CEA including the 6-month regimen, was not possible as late randomisation in this trial resulted in recruitment of a different patient population as evidenced by the NMA findings. The CEA results suggest that 9 weeks' trastuzumab is cost-saving and leads to more QALYs than 12 months', i.e. the former dominates the latter. The cost-effectiveness acceptability frontier (CEAF) favours zero trastuzumab at willingness-to-pay levels below £2,500/QALY and treatment for 9 weeks above this threshold. The combination of the NMA and Bucher investigations suggests that the 9-week duration is as efficacious as the 12-month duration for distant-disease-free survival and overall survival, and safer in terms of fewer adverse cardiac events. CONCLUSIONS: Our CEA results suggest that 9-week trastuzumab dominates 12-month trastuzumab in cost-effectiveness terms at conventional thresholds of willingness to pay for a QALY, and the 9-week regimen is also suggested to be as clinically effective as the 12-month regimen according to the NMA and Bucher analyses. This finding agrees with the results of the E2198 head-to-head study that compared 10 weeks' with 14 months' trastuzumab and found no significant difference. Appropriate trial design and reporting is critical if results are to be synthesisable with existing evidence, as selection bias can lead to recruitment of a different patient population from existing trials. Our analysis was not based on head-to-head trials' data, so the results should be viewed with caution. Short-duration trials would benefit from recruiting larger numbers of participants to reduce uncertainty in the synthesised results.