Improving grading of recommendations assessment, development, and evaluation evidence tables part 4: a three-arm noninferiority randomized trial demonstrates improved understanding of content in summary of findings tables with a new format.

OBJECTIVES: To evaluate alternative formats of summary of findings (SoF) tables for single comparison with multiple outcomes. STUDY DESIGN AND SETTING: We conducted a three-arm randomized controlled noninferiority trial (RCT) in the following systematic review (SR) users: researchers, clinical practice guideline developers, health care providers, policymakers, and knowledge transfer organizations to measure understanding, accessibility, satisfaction, and preference across the current grading of recommendations assessment, development, and evaluation (GRADE) SoF, an alternative GRADE SoF, or an adapted evidence-based practice center (EPC) program SoF table. RESULTS: One Hundred Seventy-Nine participants were randomized, and 129 participants completed the RCT (n = 47 current GRADE, n = 41 alternative GRADE, n = 41 adapted EPC). Understanding the certainty of evidence and treatment effect was comparable across groups. The adapted EPC SoF table was inferior for quantifying risk and RD compared to the alternatives (<35% correct vs. >85% correct). Participants reported increased satisfaction when SoF tables presented number needed to treat (NNT), anticipated absolute effect differences, and narrative syntheses for evidence that could not be meta-analyzed. Participants reported accessibility to information as significantly better in both GRADE SoF tables, when compared with the adapted EPC SoF table. Participants preferred the alternative GRADE SoF table format. CONCLUSION: The alternative GRADE SoF table is a promising format for SR users preferring a comprehensive presentation of SR results for single comparisons.

Methods for Identifying Health Research Gaps, Needs, and Priorities: A Scoping Review.

Well-defined, systematic, and transparent processes to identify health research gaps, needs, and priorities are vital to ensuring that available funds target areas with the greatest potential for impact. This study documents a scoping review of published methods used for identifying health research gaps, establishing research needs, and determining research priorities and provides relevant information on 362 studies. Of the 362 studies, 167 were linked to funding decisionmaking and underwent a more detailed data abstraction process. The authors noted that most studies focused on physical health conditions, but few addressed psychological health conditions. The most frequent method for identifying research gaps, needs, and priorities was to convene workshops or conferences. One-third of studies employed quantitative methods, and nearly as many used the James Lind Alliance Priority Setting Partnerships approach. Other methods included literature reviews, qualitative methods, consensus methods, and reviews of source materials. The criterion most widely applied to determine health research gaps, needs, and priorities was the importance to stakeholders, followed by the potential value and feasibility of carrying out the research. The two largest stakeholder groups were researchers and clinicians. More than one-half the studies involved patients and the public as stakeholders. Very few studies have evaluated the impact of methods used to identify research gaps, needs, and priorities. This study provides a roadmap of methods used for identifying health research gaps, needs, and priorities, which may help accelerate progress toward validating methods that ensure the effective targeting of funds to meet the greatest areas of need and to maximize impact.

Methods for Identifying Health Research Gaps, Needs, and Priorities: a Scoping Review.

BACKGROUND: Well-defined, systematic, and transparent processes to identify health research gaps, needs, and priorities are vital to ensuring that available funds target areas with the greatest potential for impact. OBJECTIVE: The purpose of this review is to characterize methods conducted or supported by research funding organizations to identify health research gaps, needs, or priorities. METHOD: We searched MEDLINE, PsycINFO, and the Web of Science up to September 2019. Eligible studies reported on methods to identify health research gaps, needs, and priorities that had been conducted or supported by research funding organizations. Using a published protocol, we extracted data on the method, criteria, involvement of stakeholders, evaluations, and whether the method had been replicated (i.e., used in other studies). RESULTS: Among 10,832 citations, 167 studies were eligible for full data extraction. More than half of the studies employed methods to identify both needs and priorities, whereas about a quarter of studies focused singularly on identifying gaps (7%), needs (6%), or priorities (14%) only. The most frequently used methods were the convening of workshops or meetings (37%), quantitative methods (32%), and the James Lind Alliance approach, a multi-stakeholder research needs and priority setting process (28%). The most widely applied criteria were importance to stakeholders (72%), potential value (29%), and feasibility (18%). Stakeholder involvement was most prominent among clinicians (69%), researchers (66%), and patients and the public (59%). Stakeholders were identified through stakeholder organizations (51%) and purposive (26%) and convenience sampling (11%). Only 4% of studies evaluated the effectiveness of the methods and 37% employed methods that were reproducible and used in other studies. DISCUSSION: To ensure optimal targeting of funds to meet the greatest areas of need and maximize outcomes, a much more robust evidence base is needed to ascertain the effectiveness of methods used to identify research gaps, needs, and priorities.

Allocation of scarce resources in a pandemic: rapid systematic review update of strategies for policymakers.

OBJECTIVE: In pandemics like COVID-19, the need for medical resources quickly outpaces available supply. policymakers need strategies to inform decisions about allocating scarce resources. STUDY DESIGN AND SETTING: We updated a systematic review on evidence-based approaches and searched databases through May 2020 for evaluation of strategies for policymakers. RESULTS: The 201 identified studies evaluated reducing demand for healthcare, optimizing existing resources, augmenting resources, and adopting crisis standards of care. Most research exists to reduce demand (n = 149); 39 higher quality studies reported benefits of contact tracing, school closures, travel restrictions, and mass vaccination. Of 28 strategies to augment resources, 6 higher quality studies reported effectiveness of establishing temporary facilities, use of volunteers, and decision support software. Of 23 strategies to optimize existing resources, 12 higher quality studies reported successful scope of work expansions and building on existing interagency agreements. Of 15 COVID-19 studies, 5 higher quality studies reported on combinations of policies and benefits of community-wide mask policies. CONCLUSION: Despite the volume, the evidence base is limited; few strategies were empirically tested in robust study designs. The review provides a comprehensive overview of the effects of strategies to allocate resources and provides critical appraisal to identify the best available evidence.

Defining Access Management in Health Care Delivery Organizations.

Managing patient access to care in health care delivery organizations is instrumental in shaping patient experiences. We convened an inclusive stakeholder panel, informed by evidence, to understand the dimensions and establish definitions of access and access management. The literature varies in access definitions, but the temporal measure "time to third next available appointment" was consistently used as an indicator of access. Panel deliberations highlighted the importance of patient-centeredness and resulted in comprehensive definitions for access management, optimal access management, and optimal access. Health care organizations and researchers can use the developed definitions and concepts as starting points for initiatives to improve access management.

Eight Priorities for Improving Primary Care Access Management in Healthcare Organizations: Results of a Modified Delphi Stakeholder Panel.

OBJECTIVE: To identify priorities for improving healthcare organization management of patient access to primary care based on prior evidence and a stakeholder panel. BACKGROUND: Studies on healthcare access show its importance for ensuring population health. Few studies show how healthcare organizations can improve access. METHODS: We conducted a modified Delphi stakeholder panel anchored by a systematic review. Panelists (N = 20) represented diverse stakeholder groups including patients, providers, policy makers, purchasers, and payers of healthcare services, predominantly from the Veterans Health Administration. A pre-panel survey addressed over 80 aspects of healthcare organization management of access, including defining access management. Panelists discussed survey-based ratings during a 2-day in-person meeting and re-voted afterward. A second panel process focused on each final priority and developed recommendations and suggestions for implementation. RESULTS: The panel achieved consensus on definitions of optimal access and access management on eight urgent and important priorities for guiding access management improvement, and on 1-3 recommendations per priority. Each recommendation is supported by referenced, panel-approved suggestions for implementation. Priorities address two organizational structure targets (interdisciplinary primary care site leadership; clearly identified group practice management structure); four process improvements (patient telephone access management; contingency staffing; nurse management of demand through care coordination; proactive demand management by optimizing provider visit schedules), and two outcomes (quality of patients' experiences of access; provider and staff morale). Recommendations and suggestions for implementation, including literature references, are summarized in a panelist-approved, ready-to-use tool. CONCLUSIONS: A stakeholder panel informed by a pre-panel systematic review identified eight action-oriented priorities for improving access and recommendations for implementing each priority. The resulting tool is suitable for guiding the VA and other integrated healthcare delivery organizations in assessing and initiating improvements in access management, and for supporting continued research.

Sequelae of an Evidence-based Approach to Management for Access to Care in the Veterans Health Administration.

BACKGROUND: Access to health care is a critical concept in the design, delivery, and evaluation of high quality care. Meaningful evaluation of access requires research evidence and the integration of perspectives of patients, providers, and administrators. OBJECTIVE: Because of high-profile access challenges, the Department of Veterans Affairs (VA) invested in research and implemented initiatives to address access management. We describe a 2-year evidence-based approach to improving access in primary care. METHODS: The approach included an Evidence Synthesis Program (ESP) report, a 22-site in-person qualitative evaluation of VA initiatives, and in-person and online stakeholder panel meetings facilitated by the RAND corporation. Subsequent work products were disseminated in a targeted strategy to increase impact on policy and practice. RESULTS: The ESP report summarized existing research evidence in primary care management and an evaluation of ongoing initiatives provided organizational data and novel metrics. The stakeholder panel served as a source of insights and information, as well as a knowledge dissemination vector. Work products included the ESP report, a RAND report, peer-reviewed manuscripts, presentations at key conferences, and training materials for VA Group Practice Managers. Resulting policy and practice implications are discussed. CONCLUSIONS: The commissioning of an evidence report was the beginning of a cascade of work including exploration of unanswered questions, novel research and measurement discoveries, and policy changes and innovation. These results demonstrate what can be achieved in a learning health care system that employs evidence and expertise to address complex issues such as access management.

Identifying Research Gaps and Prioritizing Psychological Health Evidence Synthesis Needs.

BACKGROUND: Evidence synthesis is key in promoting evidence-based health care, but it is resource-intense. Methods are needed to identify and prioritize evidence synthesis needs within health care systems. We describe a collaboration between an agency charged with facilitating the implementation of evidence-based research and practices across the Military Health System and a research center specializing in evidence synthesis. METHODS: Scoping searches targeted 15 sources, including the Veterans Affairs/Department of Defense Guidelines and National Defense Authorization Acts. We screened for evidence gaps in psychological health management approaches relevant to the target population. We translated gaps into potential topics for evidence maps and/or systematic reviews. Gaps amenable to evidence synthesis format provided the basis for stakeholder input. Stakeholders rated topics for their potential to inform psychological health care in the military health system. Feasibility scans determined whether topics were ready to be pursued, that is, sufficient literature exists, and duplicative efforts are avoided. RESULTS: We identified 58 intervention, 9 diagnostics, 12 outcome, 19 population, and 24 health services evidence synthesis gaps. Areas included: posttraumatic stress disorder (PTSD) (19), suicide prevention (14), depression (9), bipolar disorder (9), substance use (24), traumatic brain injury (20), anxiety (1), and cross-cutting (14) synthesis topics. Stakeholder input helped prioritize 19 potential PTSD topics and 22 other psychological health topics. To date, 46 topics have undergone feasibility scans. We document lessons learned across clinical topics and research methods. CONCLUSION: We describe a transparent and structured approach to evidence synthesis topic selection for a health care system using scoping searches, translation into evidence synthesis format, stakeholder input, and feasibility scans.

Two alternatives versus the standard Grading of Recommendations Assessment, Development and Evaluation (GRADE) summary of findings (SoF) tables to improve understanding in the presentation of systematic review results: a three-arm, randomised, controlled, non-inferiority trial.

OBJECTIVE: Summary of findings (SoF) tables present results of systematic reviews in a concise and explicit format. Adopted by many review groups including the Cochrane Collaboration and the Agency for Healthcare Research and Quality (AHRQ), optimal understanding of SoF table may be influenced by the type of information being conveyed and objectives or preferences of the end user. This study aims to compare three SoF table formats in terms of understanding, accessibility, satisfaction and preference with systematic review users. METHODS: The primary objective of this three-arm randomised controlled non-inferiority trial is to investigate whether an alternative Grading of Recommendations Assessment, Development and Evaluation (GRADE) SoF table or Evidence-based Practice Center SoF table is non-inferior to the current GRADE SoF table in the understanding of the information presented to systematic review users, particularly for descriptive findings. Researchers, clinical practice guideline developers, policy-makers or knowledge transfer professionals will be recruited. Data will be collected electronically at baseline and after randomisation. Non-inferiority would be declared if the difference in the proportion of participants who understand the information displayed in the alternative SoF table is 10% or less. ETHICS AND DISSEMINATION: The Hamilton Integrated Research Ethics Board reviewed this protocol. The findings from this study will be disseminated through a publication in a peer-reviewed journal. TRIAL REGISTRATION NUMBER: NCT02813941.

Fit for purpose: perspectives on rapid reviews from end-user interviews.

BACKGROUND: There is increasing demand for rapid reviews and timely evidence synthesis. The goal of this project was to understand end-user perspectives on the utility and limitations of rapid products including evidence inventories, rapid responses, and rapid reviews. METHODS: Interviews were conducted with key informants representing: guideline developers (n = 3), health care providers/health system organizations (n = 3), research funders (n = 1), and payers/health insurers (n = 1). We elicited perspectives on important characteristics of systematic reviews, acceptable methods to streamline reviews, and uses of rapid products. We analyzed content of the interview transcripts and identified themes and subthemes. RESULTS: Key informants identified the following as critical features of evidence reviews: (1) originating from a reliable source (i.e., conducted by experienced reviewers from an established research organization), (2) addressing clinically relevant questions, and (3) trusted relationship between the user and producer. Key informants expressed strong preference for the following review methods and characteristics: use of evidence tables, quality rating of studies, assessments of total evidence quality/strength, and use of summary tables for results and conclusions. Most acceptable trade-offs to increase efficiencies were limiting the literature search (e.g., limiting search dates or language) and performing single screening of citations and full texts for relevance. Key informants perceived rapid products (particularly evidence inventories and rapid responses) as useful interim products to inform downstream investigation (e.g., whether to proceed with a full review or guideline, direction for future research). Most key informants indicated that evidence analysis/synthesis and quality/strength of evidence assessments were important for decision-making. They reported that rapid reviews in particular were useful for guideline development on narrow topics, policy decisions when a quick turn-around is needed, decision-making for practicing clinicians in nuanced clinical settings, and decisions about coverage by payers/health insurers. Rapid reviews may be more relevant within specific clinical settings or health systems; whereas, broad/national guidelines often need a traditional systematic review. CONCLUSIONS: Key informants interviewed in our study indicated that evidence inventories, rapid responses, and rapid reviews have utility in specific decisions and contexts. They indicated that the credibility of the review producer, relevance of key questions, and close working relationship between the end-user and producer are critical for any rapid product. Our findings are limited by the sample size which may have been too small to reach saturation for the themes described.

Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis.

BACKGROUND: There are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses. METHODS: We generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions. RESULTS: The power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% 'high quality', 75% 'low quality' trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92%, τ(2) = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively. CONCLUSIONS: Although study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.

Hospital fall prevention: a systematic review of implementation, components, adherence, and effectiveness.

OBJECTIVES: To systematically document the implementation, components, comparators, adherence, and effectiveness of published fall prevention approaches in U.S. acute care hospitals. DESIGN: Systematic review. Studies were identified through existing reviews, searching five electronic databases, screening reference lists, and contacting topic experts for studies published through August 2011. SETTING: U.S. acute care hospitals. PARTICIPANTS: Studies reporting in-hospital falls for intervention groups and concurrent (e.g., controlled trials) or historic comparators (e.g., before-after studies). INTERVENTION: Fall prevention interventions. MEASUREMENTS: Incidence rate ratios (IRR, ratio of fall rate postintervention or treatment group to the fall rate preintervention or control group) and ratings of study details. RESULTS: Fifty-nine studies met inclusion criteria. Implementation strategies were sparsely documented (17% not at all) and included staff education, establishing committees, seeking leadership support, and occasionally continuous quality improvement techniques. Most interventions (81%) included multiple components (e.g., risk assessments (often not validated), visual risk alerts, patient education, care rounds, bed-exit alarms, and postfall evaluations). Fifty-four percent did not report on fall prevention measures applied in the comparison group, and 39% neither reported fidelity data nor described adherence strategies such as regular audits and feedback to ensure completion of care processes. Only 45% of concurrent and 15% of historic control studies reported sufficient data to compare fall rates. The pooled postintervention incidence rate ratio (IRR) was 0.77 (95% confidence interval = 0.52-1.12, P = .17; eight studies; I(2) : 94%). Meta-regressions showed no systematic association between implementation intensity, intervention complexity, comparator information, or adherence levels and IRR. CONCLUSION: Promising approaches exist, but better reporting of outcomes, implementation, adherence, intervention components, and comparison group information is necessary to establish evidence on how hospitals can successfully prevent falls.

Advancing the science of patient safety.

Despite a decade's worth of effort, patient safety has improved slowly, in part because of the limited evidence base for the development and widespread dissemination of successful patient safety practices. The Agency for Healthcare Research and Quality sponsored an international group of experts in patient safety and evaluation methods to develop criteria to improve the design, evaluation, and reporting of practice research in patient safety. This article reports the findings and recommendations of this group, which include greater use of theory and logic models, more detailed descriptions of interventions and their implementation, enhanced explanation of desired and unintended outcomes, and better description and measurement of context and of how context influences interventions. Using these criteria and measuring and reporting contexts will improve the science of patient safety.

How does context affect interventions to improve patient safety? An assessment of evidence from studies of five patient safety practices and proposals for research.

BACKGROUND: Logic and experience suggest that it is easier in some situations than in others to change behaviour and organisation to improve patient safety. Knowing which 'context factors' help and hinder implementation of different changes would help implementers, as well as managers, policy makers, regulators and purchasers of healthcare. It could help to judge the likely success of possible improvements, given the conditions that they have, and to decide which of these conditions could be modified to make implementation more effective. METHODS: The study presented in this paper examined research to discover any evidence reported about whether or how context factors influence the effectiveness of five patient safety interventions. RESULTS: The review found that, for these five diverse interventions, there was little strong evidence of the influence of different context factors. However, the research was not designed to investigate context influence. CONCLUSIONS: The paper suggests that significant gaps in research exist and makes proposals for future research better to inform decision-making.

Psychosocial interventions for non-professional carers of people with Parkinson's disease: a systematic scoping review.

AIM: This paper is a report of a scoping review to systematically identify and collate the evidence on psychosocial interventions for non-professional carers of people with Parkinson's disease. BACKGROUND: Carers are critical to people with Parkinson's disease maintaining independent living and quality of life. Parkinson's disease imposes a challenging constellation of symptoms and no summary of effective interventions for carers and their unique support needs exists. DATA SOURCES: Thirty electronic databases were searched from their inception to July 2006, and bibliographies and specific internet sites were scanned. METHODS: Eligible studies were categorized according to design, type of economic evaluation where applicable, number of participants, country of evaluation, intervention, orientation, provider, setting, method of delivery, carer population, patient population, carer outcomes, patient outcomes and authors' conclusions. Data were extracted by one reviewer and checked by another reviewer; discrepancies were resolved through discussion or arbitration by a third reviewer. FINDINGS: Thirty studies met the inclusion criteria. Most investigated relatively unique interventions involving multiple elements; the majority were not aimed primarily at carers but were embedded in patient treatment programmes. Many were pilot studies, employing weak research designs and involving very small numbers of participants and most were not designed to assess the clinical or cost effectiveness of the intervention for the carers. CONCLUSION: Several interventions merit further investigation but there is currently little evidence to show which approaches are effective and cost effective in supporting carers. Future studies need to employ appropriate and rigorous research designs with adequate samples and outcome measures, and with more focus on the carer.