A Quantitative Framework to Identify and Prioritize Opportunities in Biomedical Product Innovation: A Proof-of-Concept Study.

Importance: Prioritization and funding for health initiatives, including biomedical innovation, may not consistently target unmet public health needs. Objective: To (1) develop a quantitative, databased framework to identify and prioritize opportunities for biomedical product innovation investments based on a multicriteria decision-making model (MCDM) that includes comprehensive measures of public health burden and health care costs, and (2) pilot test the model. Design, Setting, and Participants: The Department of Health and Human Services (HHS) convened public and private experts to develop a model, select measures, and complete a longitudinal pilot study to identify and prioritize opportunities for investment in biomedical product innovations that have the greatest public health benefit. Cross-sectional and longitudinal data (2012-2019) for 13 pilot medical disorders were obtained from the Institute for Health Metrics Global Burden of Disease database (IHME GBD) and the National Center for Health Statistics (NCHS). Main Outcome Measures: The main outcome measure was an overall gap score reflecting high public health burden (composite measure of mortality, prevalence, years lived with disability, and health disparities), or high health care costs (composite measure of total, public, and out-of-pocket health spending) relative to low biomedical innovation. Sixteen innovation metrics were selected to reflect the pipeline of biomedical products from research and development to market approval. A higher score indicates a greater gap. Normalized composite scores were calculated for public health burden, cost, and innovation investment using the MCDM Technique for Order of Preference by Similarity to Ideal Solution method. Results: Among the 13 conditions tested in the pilot study, diabetes (0.61), osteoarthritis (0.46), and drug-use disorders (0.39) had the highest overall gap score reflecting high public health burden, or high health care costs relative to low biomedical innovation in these medical disorders. Chronic kidney disease (0.05), chronic obstructive pulmonary disease (0.09), and cirrhosis and other liver diseases (0.10) had the least amount of biomedical product innovation despite similar public health burden and health care cost scores. Conclusions: In this cross-sectional pilot study, we developed and implemented a data-driven, proof-of-concept model that can help identify, quantify, and prioritize opportunities for biomedical product innovation. Quantifying the relative alignment between biomedical product innovation, public health burden, and health care cost may help identify and prioritize investments that can have the greatest public health benefit.

Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program.

OBJECTIVES: We developed guidance to inform decisions regarding the inclusion of nonrandomized studies of interventions (NRSIs) in systematic reviews (SRs) of the effects of interventions. STUDY DESIGN AND SETTING: The guidance workgroup comprised SR experts and used an informal consensus generation method. RESULTS: Instead of recommending NRSI inclusion only if randomized controlled trials (RCTs) are insufficient to address the SR key question, different topics may require different decisions regarding NRSI inclusion. We identified important considerations to inform such decisions from topic refinement through protocol development. During topic scoping and refinement, considerations were related to the clinical decisional dilemma, adequacy of RCTs to address the key questions, risk of bias in NRSIs, and the extent to which NRSIs are likely to complement RCTs. When NRSIs are included, during SR team formation, familiarity with topic-specific data sources and advanced analytic methods for NRSIs should be considered. During protocol development, the decision regarding NRSI inclusion or exclusion should be justified, and potential implications explained. When NRSIs are included, the protocol should describe the processes for synthesizing evidence from RCTs and NRSIs and determining the overall strength of evidence. CONCLUSION: We identified specific considerations for decisions regarding NRSI inclusion in SRs and highlight the importance of flexibility and transparency.

Challenges and opportunities for advancing patient-centered clinical decision support: findings from a horizon scan.

OBJECTIVE: We conducted a horizon scan to (1) identify challenges in patient-centered clinical decision support (PC CDS) and (2) identify future directions for PC CDS. MATERIALS AND METHODS: We engaged a technical expert panel, conducted a scoping literature review, and interviewed key informants. We qualitatively analyzed literature and interview transcripts, mapping findings to the 4 phases for translating evidence into PC CDS interventions (Prioritizing, Authoring, Implementing, and Measuring) and to external factors. RESULTS: We identified 12 challenges for PC CDS development. Lack of patient input was identified as a critical challenge. The key informants noted that patient input is critical to prioritizing topics for PC CDS and to ensuring that CDS aligns with patients' routine behaviors. Lack of patient-centered terminology standards was viewed as a challenge in authoring PC CDS. We found a dearth of CDS studies that measured clinical outcomes, creating significant gaps in our understanding of PC CDS' impact. Across all phases of CDS development, there is a lack of patient and provider trust and limited attention to patients' and providers' concerns. DISCUSSION: These challenges suggest opportunities for advancing PC CDS. There are opportunities to develop industry-wide practices and standards to increase transparency, standardize terminologies, and incorporate patient input. There is also opportunity to engage patients throughout the PC CDS research process to ensure that outcome measures are relevant to their needs. CONCLUSION: Addressing these challenges and embracing these opportunities will help realize the promise of PC CDS-placing patients at the center of the healthcare system.

Harmonized Outcome Measures for Use in Depression Patient Registries and Clinical Practice.

Major depressive disorder is a common mental health condition that affects an estimated 16.2 million adults and 3.1 million adolescents in the United States. Yet, a lack of uniformity remains in measurements and monitoring for depression both in clinical practice and in research settings. This project aimed to develop a minimum set of standardized outcome measures relevant to both patients and clinicians that can be collected in depression registries and clinical practice. Twenty-nine depression registries and related data collection efforts were identified and invited to submit outcome measures. Additional measures were identified through literature searches and reviews of quality measures. A multistakeholder panel representing clinicians; payers; government agencies; industry; and medical specialty, health care quality, and patient advocacy organizations categorized the 27 identified measures using the Agency for Healthcare Research and Quality's supported Outcome Measures Framework. The panel identified 10 broadly relevant measures and harmonized definitions for these measures through in-person and virtual meetings. The harmonized measures represent a minimum set of outcomes that are relevant to clinicians and patients and appropriate for use in depression research and clinical practice. Routine and consistent collection of these measures in registries and other systems would support creation of a national research infrastructure to efficiently address new questions, improve patient management and outcomes, and facilitate care coordination.

Next Steps in Improving Healthcare Value: AHRQ Evidence-based Practice Center Program-Applying the Knowledge to Practice to Data Cycle to Strengthen the Value of Patient Care.

For more than 20 years, the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program has been identifying and synthesizing evidence to inform evidence-based healthcare. Recognizing that many healthcare settings continue to face challenges in disseminating and implementing evidence into practice, AHRQ's EPC program has also embarked on initiatives to facilitate the translation of evidence into practice and to measure and monitor how practice changes impact health outcomes. The program has structured its efforts around the three phases of the Learning Healthcare System cycle: knowledge, practice, and data. Here, we use a topic relevant to the field of hospital medicine-Clostridium difficile colitis prevention and treatment-as an exemplar of how the EPC program has used this framework to move evidence into practice and develop systems to facilitate continuous learning in healthcare systems.

Does information from ClinicalTrials.gov increase transparency and reduce bias? Results from a five-report case series.

BACKGROUND: We investigated whether information in ClinicalTrials.gov would impact the conclusions of five ongoing systematic reviews. METHOD: We considered five reviews that included 495 studies total. Each review team conducted a search of ClinicalTrials.gov up to the date of the review's last literature search, screened the records using the review's eligibility criteria, extracted information, and assessed risk of bias and applicability. Each team then evaluated the impact of the evidence found in ClinicalTrials.gov on the conclusions in the review. RESULTS: Across the five reviews, the number of studies that had both a registry record and a publication varied widely, from none in one review to 43% of all studies identified in another. Among the studies with both a record and publication, there was also wide variability in the match between published outcomes and those listed in ClinicalTrials.gov. Of the 173 total ClinicalTrials.gov records identified across the five projects, between 11 and 43% did not have an associated publication. In the 14% of records that contained results, the new data provided in the ClinicalTrials.gov records did not change the results or conclusions of the reviews. Finally, a large number of published studies were not registered in ClinicalTrials.gov, but many of these were published before ClinicalTrials.gov's inception date of 2000. CONCLUSION: Improved prospective registration of trials and consistent reporting of results in ClinicalTrials.gov would help make ClinicalTrials.gov records more useful in finding unpublished information and identifying potential biases. In addition, consistent indexing in databases, such as MEDLINE, would allow for better matching of records and publications, leading to increased utility of these searches for systematic review projects.

Fit for purpose: perspectives on rapid reviews from end-user interviews.

BACKGROUND: There is increasing demand for rapid reviews and timely evidence synthesis. The goal of this project was to understand end-user perspectives on the utility and limitations of rapid products including evidence inventories, rapid responses, and rapid reviews. METHODS: Interviews were conducted with key informants representing: guideline developers (n = 3), health care providers/health system organizations (n = 3), research funders (n = 1), and payers/health insurers (n = 1). We elicited perspectives on important characteristics of systematic reviews, acceptable methods to streamline reviews, and uses of rapid products. We analyzed content of the interview transcripts and identified themes and subthemes. RESULTS: Key informants identified the following as critical features of evidence reviews: (1) originating from a reliable source (i.e., conducted by experienced reviewers from an established research organization), (2) addressing clinically relevant questions, and (3) trusted relationship between the user and producer. Key informants expressed strong preference for the following review methods and characteristics: use of evidence tables, quality rating of studies, assessments of total evidence quality/strength, and use of summary tables for results and conclusions. Most acceptable trade-offs to increase efficiencies were limiting the literature search (e.g., limiting search dates or language) and performing single screening of citations and full texts for relevance. Key informants perceived rapid products (particularly evidence inventories and rapid responses) as useful interim products to inform downstream investigation (e.g., whether to proceed with a full review or guideline, direction for future research). Most key informants indicated that evidence analysis/synthesis and quality/strength of evidence assessments were important for decision-making. They reported that rapid reviews in particular were useful for guideline development on narrow topics, policy decisions when a quick turn-around is needed, decision-making for practicing clinicians in nuanced clinical settings, and decisions about coverage by payers/health insurers. Rapid reviews may be more relevant within specific clinical settings or health systems; whereas, broad/national guidelines often need a traditional systematic review. CONCLUSIONS: Key informants interviewed in our study indicated that evidence inventories, rapid responses, and rapid reviews have utility in specific decisions and contexts. They indicated that the credibility of the review producer, relevance of key questions, and close working relationship between the end-user and producer are critical for any rapid product. Our findings are limited by the sample size which may have been too small to reach saturation for the themes described.

A taxonomy of rapid reviews links report types and methods to specific decision-making contexts.

OBJECTIVES: Describe characteristics of rapid reviews and examine the impact of methodological variations on their reliability and validity. STUDY DESIGN AND SETTING: We conducted a literature review and interviews with organizations that produce rapid reviews or related products to identify methods, guidance, empiric evidence, and current practices. RESULTS: We identified 36 rapid products from 20 organizations (production time, 5 minutes to 8 months). Methods differed from systematic reviews at all stages. As time frames increased, methods became more rigorous; however, restrictions on database searching, inclusion criteria, data extracted, and independent dual review remained. We categorized rapid products based on extent of synthesis. "Inventories" list what evidence is available. "Rapid responses" present best available evidence with no formal synthesis. "Rapid reviews" synthesize the quality of and findings from the evidence. "Automated approaches" generate meta-analyses in response to user-defined queries. Rapid products rely on a close relationship with end users and support specific decisions in an identified time frame. Limited empiric evidence exists comparing rapid and systematic reviews. CONCLUSIONS: Rapid products have tremendous methodological variation; categorization based on time frame or type of synthesis reveals patterns. The similarity across rapid products lies in the close relationship with the end user to meet time-sensitive decision-making needs.

Agency for Healthcare Research and Quality Evidence-based Practice Center methods for systematically reviewing complex multicomponent health care interventions.

OBJECTIVES: The purpose of this Agency for Healthcare Research and Quality Evidence-based Practice Center methods white paper was to outline approaches to conducting systematic reviews of complex multicomponent health care interventions. STUDY DESIGN AND SETTING: We performed a literature scan and conducted semistructured interviews with international experts who conduct research or systematic reviews of complex multicomponent interventions (CMCIs) or organizational leaders who implement CMCIs in health care. RESULTS: Challenges identified include lack of consistent terminology for such interventions (eg, complex, multicomponent, multidimensional, multifactorial); a wide range of approaches used to frame the review, from grouping interventions by common features to using more theoretical approaches; decisions regarding whether and how to quantitatively analyze the interventions, from holistic to individual component analytic approaches; and incomplete and inconsistent reporting of elements critical to understanding the success and impact of multicomponent interventions, such as methods used for implementation the context in which interventions are implemented. CONCLUSION: We provide a framework for the spectrum of conceptual and analytic approaches to synthesizing studies of multicomponent interventions and an initial list of critical reporting elements for such studies. This information is intended to help systematic reviewers understand the options and tradeoffs available for such reviews.

A framework for creating standardized outcome measures for patient registries.

AIM: Our objectives were to create a conceptual framework for development of standard outcome measures and to design and pilot test a tool for displaying outcome measures. MATERIALS & METHODS: Information on outcome measures used in registries was gathered through stakeholder discussions, which informed the development of the outcome measurement framework and the related tool. RESULTS: The outcome measurement framework is a conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health areas. The related tool facilitates collecting, displaying and searching for information on outcome measures. CONCLUSION: The model developed through this process offers a framework that can be used to define outcome measures in a standard way across medical conditions.

A proposed approach may help systematic reviews retain needed expertise while minimizing bias from nonfinancial conflicts of interest.

OBJECTIVES: Groups such as the Institute of Medicine emphasize the importance of attention to financial conflicts of interest. Little guidance exists, however, on managing the risk of bias for systematic reviews from nonfinancial conflicts of interest. We sought to create practical guidance on ensuring adequate clinical or content expertise while maintaining independence of judgment on systematic review teams. STUDY DESIGN AND SETTING: Workgroup members built on existing guidance from international and domestic institutions on managing conflicts of interest. We then developed practical guidance in the form of an instrument for each potential source of conflict. RESULTS: We modified the Institute of Medicine's definition of conflict of interest to arrive at a definition specific to nonfinancial conflicts. We propose questions for funders and systematic review principal investigators to evaluate the risk of nonfinancial conflicts of interest. Once risks have been identified, options for managing conflicts include disclosure followed by no change in the systematic review team or activities, inclusion on the team along with other members with differing viewpoints to ensure diverse perspectives, exclusion from certain activities, and exclusion from the project entirely. CONCLUSION: The feasibility and utility of this approach to ensuring needed expertise on systematic reviews and minimizing bias from nonfinancial conflicts of interest must be investigated.

mobilityRERC state of the science conference: Considerations for developing an evidence base for wheeled mobility and seating service delivery.

PURPOSE: This article, developed as background content for discussion during the Mobility Rehabilitation Engineering Research Center State of the Science Conference, reviews research surrounding wheeled mobility and seating (WMS) service delivery, discusses the challenges of improving clinical decision-making, and discusses research approaches used to study and improve health services in other practice areas that might be leveraged to develop the evidence base for WMS. METHODS: Narrative literature review. RESULTS AND CONCLUSIONS: An overview of existing research found general agreement across models of WMS service delivery but little high quality evidence to support the recommended approaches and few studies of the relationship between service delivery steps and individual patient outcomes. The definition of successful clinical decision-making is different for different stakeholders. Clinical decision-making should incorporate the best available evidence along with patient values, preferences, circumstances, and clinical expertise. To advance the evidence base for WMS service delivery, alternatives to randomized controlled trials should be considered and reliable and valid outcome measures developed. Technological advances offer tremendous opportunities for individuals with complex rehabilitation technology needs. However, with ongoing scrutiny of WMS service delivery there is an increased need for evidence to support the clinical decision-making process and to support evidence-based coverage policies for WMS services and technologies. IMPLICATIONS FOR REHABILITATION: An evidence base for wheeled mobility and seating services is an important component of the clinical decision-making process. At present, there is little evidence regarding essential components of the wheeled mobility and seating evaluation or the relationship between the evaluation process and patient outcomes. Many factors can confound this relationship and present challenges to research in this area. All stakeholders (i.e. clinicians, rehabilitation technology suppliers, manufacturers, researchers, payers, policy makers, and wheelchair users) need to work together to develop and support an evidence base for wheeled mobility and seating service delivery.

GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes.

GRADE requires guideline developers to make an overall rating of confidence in estimates of effect (quality of evidence-high, moderate, low, or very low) for each important or critical outcome. GRADE suggests, for each outcome, the initial separate consideration of five domains of reasons for rating down the confidence in effect estimates, thereby allowing systematic review authors and guideline developers to arrive at an outcome-specific rating of confidence. Although this rating system represents discrete steps on an ordinal scale, it is helpful to view confidence in estimates as a continuum, and the final rating of confidence may differ from that suggested by separate consideration of each domain. An overall rating of confidence in estimates of effect is only relevant in settings when recommendations are being made. In general, it is based on the critical outcome that provides the lowest confidence.

GRADE guidelines: 9. Rating up the quality of evidence.

The most common reason for rating up the quality of evidence is a large effect. GRADE suggests considering rating up quality of evidence one level when methodologically rigorous observational studies show at least a two-fold reduction or increase in risk, and rating up two levels for at least a five-fold reduction or increase in risk. Systematic review authors and guideline developers may also consider rating up quality of evidence when a dose-response gradient is present, and when all plausible confounders or biases would decrease an apparent treatment effect, or would create a spurious effect when results suggest no effect. Other considerations include the rapidity of the response, the underlying trajectory of the condition, and indirect evidence.

Assessing applicability when comparing medical interventions: AHRQ and the Effective Health Care Program.

OBJECTIVE: To describe a systematic approach for identifying, reporting, and synthesizing information to allow consistent and transparent consideration of the applicability of the evidence in a systematic review according to the Population, Intervention, Comparator, Outcome, Setting domains. STUDY DESIGN AND SETTING: Comparative effectiveness reviews need to consider whether available evidence is applicable to specific clinical or policy questions to be useful to decision makers. Authors reviewed the literature and developed guidance for the Effective Health Care program. RESULTS: Because applicability depends on the specific questions and needs of the users, it is difficult to devise a valid uniform scale for rating the overall applicability of individual studies or body of evidence. We recommend consulting stakeholders to identify the factors most relevant to applicability for their decisions. Applicability should be considered separately for benefits and harms. Observational studies can help determine whether trial populations and interventions are representative of "real world" practice. Reviewers should describe differences between available evidence and the ideally applicable evidence for the question being asked and offer a qualitative judgment about the importance and potential effect of those differences. CONCLUSION: Careful consideration of applicability may improve the usefulness of systematic reviews in informing practice and policy.

Planning the Safety of Atrial Fibrillation Ablation Registry Initiative (SAFARI) as a Collaborative Pan-Stakeholder Critical Path Registry Model: a Cardiac Safety Research Consortium "Incubator" Think Tank.

Atrial fibrillation (AF) is a major public health problem in the United States that is associated with increased mortality and morbidity. Of the therapeutic modalities available to treat AF, the use of percutaneous catheter ablation of AF is expanding rapidly. Randomized clinical trials examining the efficacy and safety of AF ablation are currently underway; however, such trials can only partially determine the safety and durability of the effect of the procedure in routine clinical practice, in more complex patients, and over a broader range of techniques and operator experience. These limitations of randomized trials of AF ablation, particularly with regard to safety issues, could be addressed using a synergistically structured national registry, which is the intention of the SAFARI. To facilitate discussions about objectives, challenges, and steps for such a registry, the Cardiac Safety Research Consortium and the Duke Clinical Research Institute, Durham, NC, in collaboration with the US Food and Drug Administration, the American College of Cardiology, and the Heart Rhythm Society, organized a Think Tank meeting of experts in the field. Other participants included the National Heart, Lung and Blood Institute, the Centers for Medicare and Medicaid Services, the Agency for Healthcare Research and Quality, the Society of Thoracic Surgeons, the AdvaMed AF working group, and additional industry representatives. The meeting took place on April 27 to 28, 2009, at the US Food and Drug Administration headquarters in Silver Spring, MD. This article summarizes the issues and directions presented and discussed at the meeting.

Diffusion of new technology and payment policies: coronary stents.

Medicare payment is often cited as a major driver of medical technology diffusion. Stakeholders claimed that beneficiaries would be denied access to stents because Medicare payment did not initially cover the cost of stents. Nevertheless, stents diffused rapidly, including to untested indications. Outcomes with stents improved over time, primarily because of a fundamental property of technology diffusion termed "reinvention," in which new technology is modified by users. The traditional system of regulatory approval and reimbursement does not account for this dynamic process. There has been no incentive for systematic collection of data to determine which modifications are most beneficial.

Preventing tomorrow's sudden cardiac death today: part II: Translating sudden cardiac death risk assessment strategies into practice and policy.

Although current evidence supporting a more precise strategy for identifying patients at highest risk for sudden cardiac death (SCD) is sparse, strategies for translating existing and future evidence into clinical practice and policy are needed today. A great many unanswered questions exist. Examples include the following: At what level of risk for SCD should we pursue further testing or therapy? How should clinical strategies ethically and economically balance alternative outcomes? How can we best translate optimal strategies into clinical practice so as to prevent tomorrow's SCDs? On July 20 and 21, 2006, a group of individuals with expertise in clinical cardiovascular medicine, biostatistics, economics, and health policy was joined by government (Food and Drug Administration; Centers for Medicare and Medicaid Services; National Heart, Lung, and Blood Institute; Agency for Healthcare Research and Quality), professional societies (Heart Rhythm Society), and industry to discuss strategies for risk assessment and prevention of SCD. The meeting was organized by the Duke Center for the Prevention of Sudden Cardiac Death and the Duke Clinical Research Institute. This article, the second of 2 documents, summarizes the policy discussions of that meeting, discusses an analytic framework for evaluating the risks and benefits associated with SCD prevention and risk stratification, and addresses the translation of SCD risk assessment strategies into practice and policy.