Patient and caregiver engagement in the Patient-Centered Outcomes Research Institute (PCORI) Health Care Horizon Scanning System (HCHSS) process.

OBJECTIVE: The Patient-Centered Outcomes Research Institute (PCORI) horizon scanning system is an early warning system for healthcare interventions in development that could disrupt standard care. We report preliminary findings from the patient engagement process. METHODS: The system involves broadly scanning many resources to identify and monitor interventions up to 3 years before anticipated entry into U.S. health care. Topic profiles are written on included interventions with late-phase trial data and circulated with a structured review form for stakeholder comment to determine disruption potential. Stakeholders include patients and caregivers recruited from credible community sources. They view an orientation video, comment on topic profiles, and take a survey about their experience. RESULTS: As of March 2020, 312 monitored topics (some of which were archived) were derived from 3,500 information leads; 121 met the criteria for topic profile development and stakeholder comment. We invited fifty-four patients and caregivers to participate; thirty-nine reviewed at least one report. Their perspectives informed analyst nominations for fourteen topics in two 2019 High Potential Disruption Reports. Thirty-four patient stakeholders completed the user-experience survey. Most agreed (68 percent) or somewhat agreed (26 percent) that they were confident they could provide useful comments. Ninety-four percent would recommend others to participate. CONCLUSIONS: The system has successfully engaged patients and caregivers, who contributed unique and important perspectives that informed the selection of topics deemed to have high potential to disrupt clinical care. Most participants would recommend others to participate in this process. More research is needed to inform optimal patient and caregiver stakeholder recruitment and engagement methods and reduce barriers to participation.

Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer.

BACKGROUND: Improving the speed of systematic review (SR) development is key to supporting evidence-based medicine. Machine learning tools which semi-automate citation screening might improve efficiency. Few studies have assessed use of screening prioritization functionality or compared two tools head to head. In this project, we compared performance of two machine-learning tools for potential use in citation screening. METHODS: Using 9 evidence reports previously completed by the ECRI Institute Evidence-based Practice Center team, we compared performance of Abstrackr and EPPI-Reviewer, two off-the-shelf citations screening tools, for identifying relevant citations. Screening prioritization functionality was tested for 3 large reports and 6 small reports on a range of clinical topics. Large report topics were imaging for pancreatic cancer, indoor allergen reduction, and inguinal hernia repair. We trained Abstrackr and EPPI-Reviewer and screened all citations in 10% increments. In Task 1, we inputted whether an abstract was ordered for full-text screening; in Task 2, we inputted whether an abstract was included in the final report. For both tasks, screening continued until all studies ordered and included for the actual reports were identified. We assessed potential reductions in hypothetical screening burden (proportion of citations screened to identify all included studies) offered by each tool for all 9 reports. RESULTS: For the 3 large reports, both EPPI-Reviewer and Abstrackr performed well with potential reductions in screening burden of 4 to 49% (Abstrackr) and 9 to 60% (EPPI-Reviewer). Both tools had markedly poorer performance for 1 large report (inguinal hernia), possibly due to its heterogeneous key questions. Based on McNemar's test for paired proportions in the 3 large reports, EPPI-Reviewer outperformed Abstrackr for identifying articles ordered for full-text review, but Abstrackr performed better in 2 of 3 reports for identifying articles included in the final report. For small reports, both tools provided benefits but EPPI-Reviewer generally outperformed Abstrackr in both tasks, although these results were often not statistically significant. CONCLUSIONS: Abstrackr and EPPI-Reviewer performed well, but prioritization accuracy varied greatly across reports. Our work suggests screening prioritization functionality is a promising modality offering efficiency gains without giving up human involvement in the screening process.

AHRQ EPC Series on Improving Translation of Evidence: Use of a Clinical Pathway for C. Difficile Treatment to Facilitate the Translation of Research Findings into Practice.

BACKGROUND: In 2018 the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program issued a call for strategies to disseminate AHRQ EPC systematic reviews. In this pilot, findings from the 2016 AHRQ EPC report on Clostridioides difficile infection were translated into a treatment pathway and disseminated via a cloud-based platform and electronic health record (EHR). METHODS: An existing 10-step framework was used for developing and disseminating evidence-based clinical pathways. The development of the EHR intervention was informed by the Five Rights model for clinical decision support and human-computer interaction design heuristics. The researchers used observations and time measurements to describe the impact of the EPC report on pathway development and examined provider adoption using counts of pathway views. RESULTS: Two main themes emerged: (1) discrepancies between the EPC report and existing guidelines prompted critical discussions about available treatments, and (2) lack of guideline and pathway syntheses in the EPC report necessitated a rapid literature review. Pathway development required 340 hours: 205 for the rapid literature review, 63 for pathway development and EHR intervention design, and 5 for technical implementation of the intervention. Pathways were viewed 1,069 times through the cloud-based platform and 47 times through a hyperlink embedded in key EHR ordering screens. CONCLUSION: Pathways can be an approach for disseminating AHRQ EPC report findings within health care systems; however, reports should include guideline and pathway syntheses to meet their full potential. Embedding hyperlinks to pathway content within the EHR may be a viable and low-effort solution for promoting awareness of evidence-based resources.

Developing and Testing the Agency for Healthcare Research and Quality's National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument.

In 2011, the Institute of Medicine (IOM) (now the National Academy of Medicine) published standards for trustworthy guidelines and recommended that the National Guideline Clearinghouse (NGC) of the Agency for Healthcare Research and Quality clearly indicate the extent to which guidelines adhere to these standards. To accomplish this, the authors developed and tested the NGC Extent of Adherence to Trustworthy Standards (NEATS) instrument. The standards were operationalized as an instrument containing 15 items that cover disclosure of the funding source; disclosure and management of conflicts of interest; multidisciplinary input; incorporation of patient perspectives; rigorous systematic review; recommendations accompanied by rationale, assessment of benefits and harms, clear linkage to the evidence, and assessment of strength of evidence and strength of recommendation; clear articulation of recommendations; external review by diverse stakeholders; and plans for updating. After multiple rounds of feedback from experts on clinical practice guideline development, the external validity and interrater reliability of the instrument were evaluated. For each item, 80% to 100% of survey respondents judged it to be a good measure of the IOM standards. All external stakeholders stated that NEATS was suitable for its intended goal. Interrater reliability for the final NEATS instrument had a weighted κ of 0.73. The NEATS instrument is a focused tool that provides a concise evaluation of a guideline's adherence to the IOM standards for trustworthy guidelines. It has good external validity among guideline developers and good interrater reliability across trained reviewers.

AHRQ series on complex intervention systematic reviews-paper 3: adapting frameworks to develop protocols.

BACKGROUND: Once a proposed topic has been identified for a systematic review and has undergone a question formulation stage, a protocol must be developed that specifies the scope and research questions in detail and outlines the methodology for conducting the systematic review. RATIONALE: Framework modifications are often needed to accommodate increased complexity. We describe and give examples of adaptations and alternatives to traditional analytic frameworks. DISCUSSION: This article identifies and describes elements of frameworks and how they can be adapted to inform the protocol and conduct of systematic reviews of complex interventions. Modifications may be needed to adapt the population, intervention, comparators, and outcomes normally used in protocol development to successfully describe complex interventions; in some instances, alternative frameworks may be better suited. Possible approaches to analytic frameworks for complex interventions that illustrate causal and associative linkages are outlined, including time elements, which systematic reviews of complex interventions may need to address. The need for and specifics of the accommodations vary with details of a specific systematic review. This in turn helps determine whether traditional frameworks are sufficient, can be refined, or if alternate frameworks must be adopted.

Fit for purpose: perspectives on rapid reviews from end-user interviews.

BACKGROUND: There is increasing demand for rapid reviews and timely evidence synthesis. The goal of this project was to understand end-user perspectives on the utility and limitations of rapid products including evidence inventories, rapid responses, and rapid reviews. METHODS: Interviews were conducted with key informants representing: guideline developers (n = 3), health care providers/health system organizations (n = 3), research funders (n = 1), and payers/health insurers (n = 1). We elicited perspectives on important characteristics of systematic reviews, acceptable methods to streamline reviews, and uses of rapid products. We analyzed content of the interview transcripts and identified themes and subthemes. RESULTS: Key informants identified the following as critical features of evidence reviews: (1) originating from a reliable source (i.e., conducted by experienced reviewers from an established research organization), (2) addressing clinically relevant questions, and (3) trusted relationship between the user and producer. Key informants expressed strong preference for the following review methods and characteristics: use of evidence tables, quality rating of studies, assessments of total evidence quality/strength, and use of summary tables for results and conclusions. Most acceptable trade-offs to increase efficiencies were limiting the literature search (e.g., limiting search dates or language) and performing single screening of citations and full texts for relevance. Key informants perceived rapid products (particularly evidence inventories and rapid responses) as useful interim products to inform downstream investigation (e.g., whether to proceed with a full review or guideline, direction for future research). Most key informants indicated that evidence analysis/synthesis and quality/strength of evidence assessments were important for decision-making. They reported that rapid reviews in particular were useful for guideline development on narrow topics, policy decisions when a quick turn-around is needed, decision-making for practicing clinicians in nuanced clinical settings, and decisions about coverage by payers/health insurers. Rapid reviews may be more relevant within specific clinical settings or health systems; whereas, broad/national guidelines often need a traditional systematic review. CONCLUSIONS: Key informants interviewed in our study indicated that evidence inventories, rapid responses, and rapid reviews have utility in specific decisions and contexts. They indicated that the credibility of the review producer, relevance of key questions, and close working relationship between the end-user and producer are critical for any rapid product. Our findings are limited by the sample size which may have been too small to reach saturation for the themes described.

A taxonomy of rapid reviews links report types and methods to specific decision-making contexts.

OBJECTIVES: Describe characteristics of rapid reviews and examine the impact of methodological variations on their reliability and validity. STUDY DESIGN AND SETTING: We conducted a literature review and interviews with organizations that produce rapid reviews or related products to identify methods, guidance, empiric evidence, and current practices. RESULTS: We identified 36 rapid products from 20 organizations (production time, 5 minutes to 8 months). Methods differed from systematic reviews at all stages. As time frames increased, methods became more rigorous; however, restrictions on database searching, inclusion criteria, data extracted, and independent dual review remained. We categorized rapid products based on extent of synthesis. "Inventories" list what evidence is available. "Rapid responses" present best available evidence with no formal synthesis. "Rapid reviews" synthesize the quality of and findings from the evidence. "Automated approaches" generate meta-analyses in response to user-defined queries. Rapid products rely on a close relationship with end users and support specific decisions in an identified time frame. Limited empiric evidence exists comparing rapid and systematic reviews. CONCLUSIONS: Rapid products have tremendous methodological variation; categorization based on time frame or type of synthesis reveals patterns. The similarity across rapid products lies in the close relationship with the end user to meet time-sensitive decision-making needs.

An evidence-based prehospital guideline for external hemorrhage control: American College of Surgeons Committee on Trauma.

This report describes the development of an evidence-based guideline for external hemorrhage control in the prehospital setting. This project included a systematic review of the literature regarding the use of tourniquets and hemostatic agents for management of life-threatening extremity and junctional hemorrhage. Using the GRADE methodology to define the key clinical questions, an expert panel then reviewed the results of the literature review, established the quality of the evidence and made recommendations for EMS care. A clinical care guideline is proposed for adoption by EMS systems. Key words: tourniquet; hemostatic agents; external hemorrhage.

Assessing the utility of genetic tests.

Genetic testing is a rapidly expanding area with many clinical applications. While the introduction of new genetic tests creates tremendous potential for improving patient care, it is essential to adequately evaluate these tests to ensure their accuracy and utility for clinical practice. This article describes a general approach to the evaluation of genetic tests and discusses common challenges that evaluators face. This article's goal was to provide a starting point for those who are concerned with the safety and utility of genetic tests to develop an overall strategy to perform the assessment.

In-facility delirium prevention programs as a patient safety strategy: a systematic review.

Delirium, an acute decline in attention and cognition, occurs among hospitalized patients at rates estimated to range from 14% to 56% and increases the risk for morbidity and mortality. The purpose of this systematic review was to evaluate the effectiveness and safety of in-facility multicomponent delirium prevention programs. A search of 6 databases (including MEDLINE, EMBASE, and CINAHL) was conducted through September 2012. Randomized, controlled trials; controlled clinical trials; interrupted time series; and controlled before-after studies with a prospective postintervention portion were eligible for inclusion. The evidence from 19 studies that met the inclusion criteria suggests that most multicomponent interventions are effective in preventing onset of delirium in at-risk patients in a hospital setting. Evidence was insufficient to determine the benefit of such programs in other care settings. Future comparative effectiveness studies with standardized protocols are needed to identify which components in multicomponent interventions are most effective for delirium prevention.

Preventing in-facility pressure ulcers as a patient safety strategy: a systematic review.

Complications from hospital-acquired pressure ulcers cause 60,000 deaths and significant morbidity annually in the United States. The objective of this systematic review is to review evidence regarding multicomponent strategies for preventing pressure ulcers and to examine the importance of contextual aspects of programs that aim to reduce facility-acquired pressure ulcers. CINAHL, the Cochrane Library, EMBASE, MEDLINE, and PreMEDLINE were searched for articles published from 2000 to 2012. Studies (any design) that implemented multicomponent initiatives to prevent pressure ulcers in adults in U.S. acute and long-term care settings and that reported pressure ulcer rates at least 6 months after implementation were selected. Two reviewers extracted study data and rated quality of evidence. Findings from 26 implementation studies (moderate strength of evidence) suggested that the integration of several core components improved processes of care and reduced pressure ulcer rates. Key components included the simplification and standardization of pressure ulcer-specific interventions and documentation, involvement of multidisciplinary teams and leadership, use of designated skin champions, ongoing staff education, and sustained audit and feedback.

Chapter 2: medical tests guidance (2) developing the topic and structuring systematic reviews of medical tests: utility of PICOTS, analytic frameworks, decision trees, and other frameworks.

Topic development and structuring a systematic review of diagnostic tests are complementary processes. The goals of a medical test review are to identify and synthesize evidence to evaluate the impacts alternative testing strategies on health outcomes and to promote informed decision making. A common challenge is that the request for a review may state the claim for the test ambiguously. Due to the indirect impact of medical tests on clinical outcomes, reviewers need to identify which intermediate outcomes link a medical test to improved clinical outcomes. In this paper, we propose the use of five principles to deal with challenges: the PICOTS typology (patient population, intervention, comparator, outcomes, timing, setting), analytic frameworks, simple decision trees, other organizing frameworks and rules for when diagnostic accuracy is sufficient.

Observational studies in systematic [corrected] reviews of comparative effectiveness: AHRQ and the Effective Health Care Program.

OBJECTIVE: Systematic reviewers disagree about the ability of observational studies to answer questions about the benefits or intended effects of pharmacotherapeutic, device, or procedural interventions. This study provides a framework for decision making on the inclusion of observational studies to assess benefits and intended effects in comparative effectiveness reviews (CERs). STUDY DESIGN AND SETTING: The conceptual model and recommendations were developed using a consensus process by members of the methods workgroup of the Effective Health Care Program of the Agency for Healthcare Research and Quality. RESULTS: In considering whether to use observational studies in CERs for addressing beneficial effects, reviewers should answer two questions: (1) Are there gaps in the evidence from randomized controlled trials (RCTs)? (2) Will observational studies provide valid and useful information? The latter question involves the following: (a) refocusing the study questions on gaps in the evidence from RCTs, (b) assessing the risk of bias of the body of evidence of observational studies, and (c) assessing whether available observational studies address the gap review questions. CONCLUSIONS: Because it is unusual to find sufficient evidence from RCTs to answer all key questions concerning benefit or the balance of benefits and harms, comparative effectiveness reviewers should routinely assess the appropriateness of inclusion of observational studies for questions of benefit. Furthermore, reviewers should explicitly state the rationale for inclusion or exclusion of observational studies when conducting CERs.

Stereotactic body radiation therapy: scope of the literature.

Stereotactic body radiation therapy (SBRT) is derived from the techniques of stereotactic radiosurgery used to treat lesions in the brain and spine. It combines multiple finely collimated radiation beams and stereotaxy to deliver a high dose of radiation to an extracranial target in the body in a single dose or a few fractions. This review provides a broad overview of the current state of SBRT for solid malignant tumors. Reviewers identified a total of 124 relevant studies. To our knowledge, no published comparative studies address the relative effectiveness and safety of SBRT versus other forms of external-beam radiation therapy. Stereotactic body radiation therapy seems to be widely diffused as a treatment of various types of cancer, although most studies have focused only on its use for treating thoracic tumors. Comparative studies are needed to provide evidence that the theoretical advantages of SBRT over other radiation therapies actually occur in the clinical setting; this area is currently being studied in only 1 small trial.

Autologous hematopoietic cell transplantation for multiple sclerosis: a systematic review.

BACKGROUND AND OBJECTIVES: The purpose of this systematic review was to evaluate the safety and efficacy of autologous hematopoietic cell transplantation in patients with progressive multiple sclerosis (MS) refractory to conventional medical treatment. METHODS: Eight case series met our a priori inclusion criteria for the primary outcome of progression-free survival. Individual study quality was rated using an 11-item scale for case series. The strength of the overall body of evidence for each outcome was rated using a system developed by the ECRI Institute. Data from different studies were statistically combined using meta-analysis. An additional six studies were included for a summary of mortality and morbidity. RESULTS: For secondary progressive MS, immunoablative therapy with autologous bone marrow/peripheral blood stem cell transplantation was associated with higher progression-free survival (up to 3 years following treatment) when using intermediate-intensity conditioning regimens compared with high-intensity conditioning regimens. The evidence was insufficient to determine whether the treatment was effective in patients with other types of MS. Treatment-related mortality was about 2.7%. CONCLUSIONS: Patients with secondary progressive MS refractory to conventional medical treatment have longer progression-free survival following autologous stem cell transplantation with intermediate-intensity conditioning regimens than with high-intensity conditioning regimens.

Continuous positive airway pressure reduces risk of motor vehicle crash among drivers with obstructive sleep apnea: systematic review and meta-analysis.

CONTEXT: Obstructive sleep apnea (OSA) is associated with an increased risk of motor vehicle crash. OBJECTIVE: We performed a systematic review of the literature concerning the impact of continuous positive airway pressure (CPAP) treatment on motor vehicle crash risk among drivers with OSA. The primary objective was to determine whether CPAP use could reduce the risk of motor vehicle crash among drivers with OSA. A secondary objective involved determining the time on treatment required for CPAP to improve driver safety. DATA SOURCES: We searched seven electronic databases (MEDLINE, PubMed (PreMEDLINE), EMBASE, PsycINFO, CINAHL, TRIS, and the Cochrane library) and the reference lists of all obtained articles. STUDY SELECTION: We included studies (before-after, case-control, or cohort) that addressed the stated objectives. We evaluated the quality of each study and the interplay between the quality, quantity, robustness, and consistency of the evidence. We also tested for publication bias. DATA EXTRACTION: Data were extracted by two independent analysts. When appropriate, data were combined in a fixed or random effects meta-analysis. RESULTS: A meta-analysis of 9 observational studies examining crash risk of drivers with OSA pre- vs. post-CPAP found a significant risk reduction following treatment (risk ratio = 0.278, 95% CI: 0.22 to 0.35; P < 0.001). Although crash data are not available to assess the time course of change, daytime sleepiness improves significantly following a single night of treatment, and simulated driving performance improves significantly within 2 to 7 days of CPAP treatment. CONCLUSIONS: Observational studies indicate that CPAP reduces motor vehicle crash risk among drivers with OSA.

Opioid pharmacotherapy for chronic non-cancer pain in the United States: a research guideline for developing an evidence-base.

UNLABELLED: This document reports the consensus of an interdisciplinary panel of research and clinical experts charged with reviewing the use of opioids for chronic noncancer pain (CNCP) and formulating guidelines for future research. Prescribing opioids for chronic noncancer pain has recently escalated in the United States. Contrasting with increasing opioid use are: 1) The lack of evidence supporting long-term effectiveness; 2) Escalating misuse of prescription opioids including abuse and diversion; and 3) Uncertainty about the incidence and clinical salience of multiple, poorly characterized adverse drug events (ADEs) including endocrine dysfunction, immunosuppression and infectious disease, opioid-induced hyperalgesia and xerostomia, overdose, falls and fractures, and psychosocial complications. Chief among the limitations of current evidence are: 1) Sparse evidence on long-term opioid effectiveness in chronic pain patients due to the short-term time frame of clinical trials; 2) Insufficiently comprehensive outcome assessment; and 3) Incomplete identification and quantification of ADEs. The panel called for a strategic interdisciplinary approach to the problem domain in which basic scientists and clinicians cooperate to resolve urgent issues and generate a comprehensive evidence base. It offered 4 recommendations in 3 areas: 1) A research strategy for studying the effectiveness of long-term opioid pharmacotherapy; 2) Improvements in evidence-generation methodology; and 3) Potential research topics for generating new evidence. PERSPECTIVE: Prescribing opioids for CNCP has outpaced the growth of scientific evidence bearing on the benefits and harms of these interventions. The need for a strong evidence base is urgent. This guideline offers a strategic approach to creating a comprehensive evidence base to guide safe and effective management of CNCP.