Factors to Consider During Identification and Invitation of Individuals in a Multi-stakeholder Research Partnership.

BACKGROUND: Health research teams increasingly partner with stakeholders to produce research that is relevant, accessible, and widely used. Previous work has covered stakeholder group identification. OBJECTIVE: We aimed to develop factors for health research teams to consider during identification and invitation of individual representatives in a multi-stakeholder research partnership, with the aim of forming equitable and informed teams. DESIGN: Consensus development. PARTICIPANTS: We involved 16 stakeholders from the international Multi-Stakeholder Engagement (MuSE) Consortium, including patients and the public, providers, payers of health services/purchasers, policy makers, programme managers, peer review editors, and principal investigators. APPROACH: We engaged stakeholders in factor development and as co-authors of this manuscript. Using a modified Delphi approach, we gathered stakeholder views concerning a preliminary list of 18 factors. Over two feedback rounds, using qualitative and quantitative analysis, we concentrated these into ten factors. KEY RESULTS: We present seven highly desirable factors: 'expertise or experience', 'ability and willingness to represent the stakeholder group', 'inclusivity (equity, diversity and intersectionality)', 'communication skills', 'commitment and time capacity', 'financial and non-financial relationships and activities, and conflict of interest', 'training support and funding needs'. Additionally, three factors are desirable: 'influence', 'research relevant values', 'previous stakeholder engagement'. CONCLUSIONS: We present factors for research teams to consider during identification and invitation of individual representatives in a multi-stakeholder research partnership. Policy makers and guideline developers may benefit from considering the factors in stakeholder identification and invitation. Research funders may consider stipulating consideration of the factors in funding applications. We outline how these factors can be implemented and exemplify how their use has the potential to improve the quality and relevancy of health research.

Use of Real-World Evidence for Regulatory Approval and Coverage of Medical Devices: A Landscape Assessment.

OBJECTIVES: To enhance the generalizability of the evidence it reviews, the US Food and Drug Administration (FDA) has encouraged manufacturers to expand the submission of real-world evidence (RWE). The extent to which this evidence, which is generated outside of research settings, can support decision making remains unclear. We described the current use of RWE for medical devices, assessed manufacturers' challenges in generating and using it for regulatory and coverage decisions, and identified opportunities to expand its use. METHODS: We conducted 27 solo and group interviews with FDA officials and representatives of device manufacturers, payers, and health technology assessment organizations. All interviews used a semistructured protocol and were transcribed to allow thematic analysis. RESULTS: Accessing and linking real-world data sources, identifying unique devices, capturing longitudinal data, limited staff expertise, and uncertain return on investment have hampered efforts to use real-world data. Many companies in our sample were conducting research using real-world data, but none had submitted RWE as the primary evidence supporting a premarket approval. FDA guidance was helpful, but regulatory requirements remained ambiguous and examples of successful regulatory decisions based on RWE were limited. Payers mainly used RWE to supplement experimental evidence in coverage decisions, evaluated both types of evidence in similar ways, and had concerns about the rigor of RWE. CONCLUSIONS: Technical challenges may slow efforts to generate and use RWE in the near term. Additional regulatory guidance and examples, greater use of rigorous study designs and analytic methods, and continued stakeholder engagement could accelerate the use of RWE.

Engaging Patients and Other Non-Researchers in Health Research: Defining Research Engagement.

With the increase in patient and consumer activism through the late twentieth century and into this century, patient roles in research evolved into a new model of research engagement, with patients serving as active advisors and co-leading or leading clinical research. By requiring active engagement of patients and other stakeholders, several government research funders have advanced this model, particularly in Canada, the United States (US), United Kingdom (UK), and Australia. A consortium of individuals from these countries formed a Multi-Stakeholder Engagement (MuSE) consortium to examine critical issues in engaged research, establish consensus on definitions, and provide guidance for the field, beginning with an overview of how to involve stakeholders in health research (Concannon et al. J Gen Intern Med. 2019;34(3):458-463) and continuing here with an examination of definitions of research engagement. The political and advocacy roots of engaged research are reflected in definitions. Engagement is conceptualized with reference to research project goals, from informing specific clinical decisions to informing health-system level decisions. Political and cultural differences across countries are evident. Some of these government funders focus on empirical rather than ethical rationales. In countries with centralized health technology assessment, the link between societal values and engaged research is explicit. Ethical rationales for engagement are explicit in most of the published literature on research engagement. Harmonization of definitions is recommended so that research engagement elements, methods, and outcomes and impacts can be clearly examined and understood, and so that the field of research engagement can proceed from a clear conceptual foundation. Specific recommendations for terminology definitions are provided. Placing engaged research on a continuum from specific clinical decisions to more global public and social justice concerns clarifies the type of engaged research, supports appropriate comparisons, and improves the rigor of engaged research methods. The results help identify knowledge gaps in this growing field.

Are Publicly Funded Health Databases Geographically Detailed and Timely Enough to Support Patient-Centered Outcomes Research?

Emerging health care research paradigms such as comparative effectiveness research (CER), patient-centered outcome research (PCOR), and precision medicine (PM) share one ultimate goal: constructing evidence to provide the right treatment to the right patient at the right time. We argue that to succeed at this goal, it is crucial to have both timely access to individual-level data and fine geographic granularity in the data. Existing data will continue to be an important resource for observational studies as new data sources are developed. We examined widely used publicly funded health databases and population-based survey systems and found four ways they could be improved to better support the new research paradigms: (1) finer and more consistent geographic granularity, (2) more complete geographic coverage of the US population, (3) shorter time from data collection to data release, and (4) improved environments for restricted data access. We believe that existing data sources, if utilized optimally, and newly developed data infrastructures will both play a key role in expanding our insight into what treatments, at what time, work for each patient.

Practical Guidance for Involving Stakeholders in Health Research.

Stakeholder engagement is increasingly common in health research, with protocols for engaging multiple stakeholder groups becoming normative in patient-centered outcomes research. Previous work has focused on identifying relevant stakeholder groups with whom to work and on working with stakeholders in evidence implementation. This paper draws on the expertise of a team from four countries-Canada, Australia, the UK, and the USA-to provide researchers with practical guidance for carrying out multi-stakeholder-engaged projects: we present a list of questions to assist in selecting appropriate roles and modes of engagement; we introduce a matrix to help summarize engagement activities; and we provide a list of online resources. This guidance, matrix, and list of resources can assist researchers to consider more systematically which stakeholder groups to involve, in what study roles, and by what modes of engagement. By documenting how stakeholders are paired up with specific roles, the matrix also provides a potential structure for evaluating the impact of stakeholder engagement.

A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research.

OBJECTIVES: We conducted a review of the peer-reviewed literature since 2003 to catalogue reported methods of stakeholder engagement in comparative effectiveness research and patient-centered outcomes research. METHODS AND RESULTS: We worked with stakeholders before, during and after the review was conducted to: define the primary and key research questions; conduct the literature search; screen titles, abstracts and articles; abstract data from the articles; and analyze the data. The literature search yielded 2,062 abstracts. The review was conducted on 70 articles that reported on stakeholder engagement in individual research projects or programs. FINDINGS: Reports of stakeholder engagement are highly variable in content and quality. We found frequent engagement with patients, modestly frequent engagement with clinicians, and infrequent engagement with stakeholders in other key decision-making groups across the healthcare system. Stakeholder engagement was more common in earlier (prioritization) than in later (implementation and dissemination) stages of research. The roles and activities of stakeholders were highly variable across research and program reports. RECOMMENDATIONS: To improve on the quality and content of reporting, we developed a 7-Item Stakeholder Engagement Reporting Questionnaire. We recommend three directions for future research: 1) descriptive research on stakeholder-engagement in research; 2) evaluative research on the impact of stakeholder engagement on the relevance, transparency and adoption of research; and 3) development and validation of tools that can be used to support stakeholder engagement in future work.

Policy Decisionmaking in Long-Term Care: Lessons from Infection Control During the COVID-19 Pandemic.

The COVID-19 pandemic focused attention on long-term care facilities' need for infection-control policies that balanced community safety and individual well-being. Infection-control policies were often developed, implemented, and mandated without the input or involvement of those who are most affected: residents and their family members, administrators, and staff. This failure led to declines in residents' physical and mental health. The pandemic exposed an opportunity-and an imperative-to reimagine long-term care in a way that is centered on the needs and preferences of those who receive care, their family members, and those who provide care. This study lays the groundwork for cultural change and a move toward inclusive policy decisionmaking in long-term care through a review of infection-control policy decisions and action items proposed in guided discussions with a diversity of stakeholders-long-term care residents, direct care staff, and consumer advocates to facility administrators, clinicians, researchers, and industry organizations. Transforming the culture of long-term care to elevate the needs of residents will require attention to facility leadership, along with steps to increase inclusiveness, transparency, and accountability in decisionmaking.

A protocol for ongoing systematic scoping reviews of World Trade Center Health research.

BACKGROUND: The World Trade Center (WTC) Health Program ("Program") seeks to assess the inventory, quality, and impact of its funded research in the context of all clinical and translational research involving WTC populations. This paper presents a protocol for ongoing scoping reviews of WTC-related health research. METHODS: Using terms relevant to the September 11 attacks, we will search OVID MEDLINE, PsycINFO, Scopus, Web of Science, CINAHL, and Embase for records of peer-reviewed publications. Title, abstract, and full text screening will be used to exclude records according to a priori criteria. Data abstraction will be performed on all articles that meet inclusion criteria using a standardized query form that was developed in collaboration with NIOSH. A team of reviewers will be trained to abstract data from included articles. Articles will be double-reviewed, and disagreements will be adjudicated. RESULTS: We will summarize existing research involving WTC populations. The summary will assess the extent, nature, and signals of impact of WTC-related health research. CONCLUSIONS: Our review will lay the groundwork for additional study of research impact by identifying population, clinical, and translational topics that can be assessed through future focused reviews. It will also support planning activities by Program policy makers and stakeholders as they work to achieve the Program's research goals. SYSTEMATIC REVIEW REGISTRATION: This publication serves as documentation of the protocol.

Treatment goals for rheumatoid arthritis: patient engagement and goal collection.

Aim: We developed the Patient-Engaged Health Technology Assessment strategy for survey-based goal collection from patients to yield patient-important outcomes suitable for use in multi-criteria decision analysis. Methods: Rheumatoid arthritis patients were recruited from online patient networks for proof-of-concept testing of goal collection and prioritization using a survey. A Project Steering Committee and Expert Panel rated the feasibility of scaling to larger samples. Results: Survey respondents (n = 47) completed the goal collection exercise. Finding effective treatments was rated by respondents as the most important goal, and reducing stiffness was rated as the least important. Feedback from our steering committee and expert panel support the approach's feasibility for goal identification and ranking. Conclusion: Goals relevant for treatment evaluation can be identified and rated for importance by patients to permit wide input from patients with lived experience of disease.

Protocol for the development of guidance for collaborator and partner engagement in health care evidence syntheses.

BACKGROUND: Involving collaborators and partners in research may increase relevance and uptake, while reducing health and social inequities. Collaborators and partners include people and groups interested in health research: health care providers, patients and caregivers, payers of health research, payers of health services, publishers, policymakers, researchers, product makers, program managers, and the public. Evidence syntheses inform decisions about health care services, treatments, and practice, which ultimately affect health outcomes. Our objectives are to: A. Identify, map, and synthesize qualitative and quantitative findings related to engagement in evidence syntheses B. Explore how engagement in evidence synthesis promotes health equity C. Develop equity-oriented guidance on methods for conducting, evaluating, and reporting engagement in evidence syntheses METHODS: Our diverse, international team will develop guidance for engagement with collaborators and partners throughout multiple sequential steps using an integrated knowledge translation approach: 1. Reviews. We will co-produce 1 scoping review, 3 systematic reviews and 1 evidence map focusing on (a) methods, (b) barriers and facilitators, (c) conflict of interest considerations, (d) impacts, and (e) equity considerations of engagement in evidence synthesis. 2. Methods study, interviews, and survey. We will contextualise the findings of step 1 by assessing a sample of evidence syntheses reporting on engagement with collaborators and partners and through conducting interviews with collaborators and partners who have been involved in producing evidence syntheses. We will use these findings to develop draft guidance checklists and will assess agreement with each item through an international survey. 3. CONSENSUS: The guidance checklists will be co-produced and finalised at a consensus meeting with collaborators and partners. 4. DISSEMINATION: We will develop a dissemination plan with our collaborators and partners and work collaboratively to improve adoption of our guidance by key organizations. CONCLUSION: Our international team will develop guidance for collaborator and partner engagement in health care evidence syntheses. Incorporating partnership values and expectations may result in better uptake, potentially reducing health inequities.

Key issues for stakeholder engagement in the development of health and healthcare guidelines.

Established in 2015, the Multi-Stakeholder Engagement (MuSE) Consortium is an international network of over 120 individuals interested in stakeholder engagement in research and guidelines. The MuSE group is developing guidance for stakeholder engagement in the development of health and healthcare guideline development. The development of this guidance has included multiple meetings with stakeholders, including patients, payers/purchasers of health services, peer review editors, policymakers, program managers, providers, principal investigators, product makers, the public, and purchasers of health services and has identified a number of key issues. These include: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. In this paper, we discuss these issues and our plan to develop guidance to facilitate stakeholder engagement in all stages of the development of health and healthcare guideline development.

A group of international researchers, patient partners, and other stakeholders are working together to create a checklist for when and how to involve stakeholders in health guideline development. Health guidelines include clinical practice guidelines, which your healthcare provider uses to determine treatments for health conditions. While working on this checklist, the team identified key issues to work on, including: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. This paper describes each issue and how the team plans to produce guidance papers to address them.

A new taxonomy for stakeholder engagement in patient-centered outcomes research.

Despite widespread agreement that stakeholder engagement is needed in patient-centered outcomes research (PCOR), no taxonomy exists to guide researchers and policy makers on how to address this need. We followed an iterative process, including several stages of stakeholder review, to address three questions: (1) Who are the stakeholders in PCOR? (2) What roles and responsibilities can stakeholders have in PCOR? (3) How can researchers start engaging stakeholders? We introduce a flexible taxonomy called the 7Ps of Stakeholder Engagement and Six Stages of Research for identifying stakeholders and developing engagement strategies across the full spectrum of research activities. The path toward engagement will not be uniform across every research program, but this taxonomy offers a common starting point and a flexible approach.

Willingness-to-pay for predictive tests with no immediate treatment implications: a survey of US residents.

We assessed how much, if anything, people would pay for a laboratory test that predicted their future disease status. A questionnaire was administered via an internet-based survey to a random sample of adult US respondents. Each respondent answered questions about two different scenarios, each of which specified: one of four randomly selected diseases (Alzheimer's, arthritis, breast cancer, or prostate cancer); an ex ante risk of developing the disease (randomly designated 10 or 25%); and test accuracy (randomly designated perfect or 'not perfectly accurate'). Willingness-to-pay (WTP) was elicited with a double-bounded, dichotomous-choice approach. Of 1463 respondents who completed the survey, most (70-88%, depending on the scenario) were inclined to take the test. Inclination to take the test was lower for Alzheimer's and higher for prostate cancer compared with arthritis, and rose somewhat with disease prevalence and for the perfect versus imperfect test [Correction made here after initial online publication.]. Median WTP varied from $109 for the imperfect arthritis test to $263 for the perfect prostate cancer test. Respondents' preferences for predictive testing, even in the absence of direct treatment consequences, reflected health and non-health related factors, and suggests that conventional cost-effectiveness analyses may underestimate the value of testing.

Translational Impacts of World Trade Center Health Program Research: A Mixed Methods Study.

The World Trade Center (WTC) Health Program (hereafter Program) provides medical monitoring and treatment for its Members: the responders and survivors who were at the WTC and related sites or located within the New York City disaster area during and after the terrorist attacks of September 11, 2001. Scientific research supported by the Program provides an evidence base for the health monitoring and clinical care of Members. In this article, the authors describe the findings and recommendations of a four-year study of the Program's research portfolio and its translational impact. They present results from a mixed methods assessment that integrates (1) a scoping review of nearly 1,000 peer-reviewed publications and thousands of pages of grey literature that reference research related to the health effects of 9/11, (2) stakeholder perspectives gathered through focus groups and interviews, and (3) a review of Program documentation. The study is intended to guide planning by Program leadership as it aims to maximize the impacts of Program research and achieve its goal of translating research into care for those affected by the attacks on 9/11.

Impact of World Trade Center-Related Health Research: An Application of the NIEHS Translational Framework.

The World Trade Center Health Program (WTCHP) has a research mission to identify physical and mental health conditions that may be related to the 9/11 terrorist attacks as well as effective diagnostic procedures and treatments for WTC-related health conditions. The ability of the WTCHP to serve its members and realize positive impacts on all of its stakeholders depends on effective translation of research findings. As part of an ongoing assessment of the translational impact of World Trade Center (WTC)-related research, we applied the National Institute of Environmental Health Sciences (NIEHS) translational framework to two case studies: WTC-related research on post-traumatic stress disorder (PTSD) and cancer. We conducted a review of 9/11 health-related research in the peer-reviewed literature through October 2017, grey literature, and WTCHP program documentation. We mapped peer-reviewed studies in the literature to the NIEHS framework and used WTCHP program documentation and grey literature to find evidence of translation of research into clinical practice and policy. Using the NIEHS framework, we identified numerous translational milestones and bridges, as well as areas of opportunity, within each case study. This application demonstrates the utility of the NIEHS framework for documenting progress toward public health impact and for setting future research goals.

Identifying patient decisions and related information needs during decision making related to total knee arthroplasty.

Aim: Research regarding decisions patients make about total knee arthroplasty, apart from having the procedure or not, are limited. Understanding patient decision making and related information needs is essential for shared decision making. Methods: Focus groups with an online community-based sample identified decisions about total knee arthroplasty beyond the decision to have the surgery itself. An online survey was used to determine relative importance of five major decisions and evaluate related information available. Results: Patients did not feel they have enough information to make important decisions of surgeon, device type, surgical approach, facility, or timing, for their total knee arthroplasty. Conclusion: Although further research is needed to generalize these findings, physicians should consider these questions during shared decision making with patients considering total knee arthroplasty.

Why clinical trials may not help patients make treatment decisions: results from focus group discussions with 22 patients.

Aim: Despite broad interest in advancing personalized medicine, most evidence is currently derived from average results of clinical trials that may obscure heterogeneity of trial participants. Little is known currently about how patients view heterogeneity in trials and whether they can participate in methodological discussions about this concept. Materials & methods: In structured discussions with three focus groups involving 22 participants, we assessed how representatives of patient communities have used research to guide individual treatment decisions. Discussion themes were organized into a framework describing patient decision-making in four steps: decisions patients make in the course of care; information used to make decisions; sources for information; and quality of information. Results/conclusion: Patients prioritize information that reflects their own characteristics, preferences and values. They struggle applying clinical research to their own case.

Facilitating stakeholder engagement in early stage translational research.

INTRODUCTION: Stakeholder engagement can play an important role in increasing public trust and the understanding of scientific research and its impact. Frameworks for stakeholder identification exist, but these frameworks may not apply well to basic science and early stage translational research. METHODS: Four Clinical and Translational Science Award (CTSA) hubs led six focus groups and two semi-structured interviews using a semi-structured discussion guide to learn from basic science researchers about stakeholder engagement in their work. The 24 participants represented fourteen clinical and academic disciplines. RESULTS: Early stage translational researchers reported engagement with a broad array of stakeholders. Those whose research has a clinical focus reported working with a more diverse range of stakeholders than those whose work did not. Common barriers to stakeholder engagement were grouped into three major themes: a poor definition of concepts, absence of guidance, and limited resources. DISCUSSION: The National Center for Advancing Translational Sciences (NCATS), the consortium of CTSAs, and the individual CTSA "hubs" are three actors that can help early stage translational researchers develop shared terms of reference, build the necessary skills, and assemble the appropriate resources for engaging stakeholders in Clinical and Translational Research. Getting this right will involve a coordinated push by all three entities.

Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation.

BACKGROUND: Stakeholder engagement has become widely accepted as a necessary component of guideline development and implementation. While frameworks for developing guidelines express the need for those potentially affected by guideline recommendations to be involved in their development, there is a lack of consensus on how this should be done in practice. Further, there is a lack of guidance on how to equitably and meaningfully engage multiple stakeholders. We aim to develop guidance for the meaningful and equitable engagement of multiple stakeholders in guideline development and implementation. METHODS: This will be a multi-stage project. The first stage is to conduct a series of four systematic reviews. These will (1) describe existing guidance and methods for stakeholder engagement in guideline development and implementation, (2) characterize barriers and facilitators to stakeholder engagement in guideline development and implementation, (3) explore the impact of stakeholder engagement on guideline development and implementation, and (4) identify issues related to conflicts of interest when engaging multiple stakeholders in guideline development and implementation. DISCUSSION: We will collaborate with our multiple and diverse stakeholders to develop guidance for multi-stakeholder engagement in guideline development and implementation. We will use the results of the systematic reviews to develop a candidate list of draft guidance recommendations and will seek broad feedback on the draft guidance via an online survey of guideline developers and external stakeholders. An invited group of representatives from all stakeholder groups will discuss the results of the survey at a consensus meeting which will inform the development of the final guidance papers. Our overall goal is to improve the development of guidelines through meaningful and equitable multi-stakeholder engagement, and subsequently to improve health outcomes and reduce inequities in health.

Epidemiology of Clostridium difficile colitis in hospitalized patients with inflammatory bowel diseases.

PURPOSE: A notable increase in-hospital admissions for Clostridium difficile colitis has occurred in the United States. In this paper we evaluate changes in the epidemiology of Clostridium difficile colitis in a subset of hospitalized patients with inflammatory bowel diseases. METHODS: A retrospective cohort analysis was conducted for all inflammatory bowel disease patients with Clostridium difficile colitis in the Nationwide Inpatient Sample, a 20 percent stratified random sample of national hospital discharge abstracts from 1993 through 2003. Using standard diagnostic codes, we identified yearly admissions for Clostridium difficile, other bacterial infections, and parasitic infections in inflammatory bowel disease patients. Next, we calculated prevalence, case fatality, and operative mortality for inflammatory bowel disease patients diagnosed with Clostridium difficile. RESULTS: We found that the prevalence of Clostridium difficile rose significantly in patients with ulcerative colitis and in those Crohn's disease patients with some component of large bowel involvement but not in patients with Crohn's disease limited to the small bowel alone. During the study period, case fatality also rose significantly in patients with ulcerative colitis and Clostridium difficile but not in patients with Crohn's disease and Clostridium difficile. Operative mortality for ulcerative colitis patients with Clostridium difficile reached 25.7 percent. CONCLUSIONS: The prevalence and case fatality of patients with inflammatory bowel disease and Clostridium difficile rose significantly during the study period. Changes in Clostridium difficile epidemiology were particularly noteworthy for those patients with ulcerative colitis, who experienced elevated rates of hospitalization and case fatality.