Unique Review Criteria and Patient and Stakeholder Reviewers: Analysis of PCORI's Approach to Research Funding.

OBJECTIVE: The Patient-Centered Outcomes Research Institute (PCORI) uses a unique approach to Merit Review that includes patients and stakeholders as reviewers with scientists, and includes unique review criteria (patient-centeredness and active engagement of end users in the research). This study assessed the extent to which different reviewer types influence review scores and funding outcomes, the emphasis placed on technical merit compared to other criteria by a multistakeholder panel, and the impact of the in-person discussion on agreement among different reviewer types. METHODS: Cross-sectional analysis of administrative data from PCORI online and in-person Merit Review (N = 1312 applications from the five funding cycles from November 2013 to August 2015). Linear and logistic regression models were used to analyze the data. RESULTS: For all reviewer types, final review scores were associated with at least one review criterion score from each of the three reviewer types. The strongest predictor of final overall scores for all reviewer types was scientists' prediscussion ratings of technical merit. All reviewers' prediscussion ratings of the potential to improve health care and outcomes, and scientists' ratings of technical merit and patient-centeredness, were associated with funding success. For each reviewer type, overall impact scores from the online scoring were changed on at least half of the applications at the in-person panel discussion. Score agreement across reviewer types was greater after panel discussion. CONCLUSIONS: Scientist, patient, and stakeholder views all contribute to PCORI Merit Review of applications for research funding. Technical merit is critical to funding success but patient and stakeholder ratings of other criteria also influence application disposition.

Methods and impact of engagement in research, from theory to practice and back again: early findings from the Patient-Centered Outcomes Research Institute.

PURPOSE: Since 2012, PCORI has been funding patient-centered comparative effectiveness research with a requirement for engaging patients and other stakeholders in the research, a requirement that is unique among the US funders of clinical research. This paper presents PCORI's evaluation framework for assessing the short- and long-term impacts of engagement; describes engagement in PCORI projects (types of stakeholders engaged, when in the research process they are engaged and how they are engaged, contributions of their engagement); and identifies the effects of engagement on study design, processes, and outcomes selection, as reported by both PCORI-funded investigators and patient and other stakeholder research partners. METHODS: Detailed quantitative and qualitative information collected annually from investigators and their partners was analyzed via descriptive statistics and cross-sectional qualitative content and thematic analysis, and compared against the outcomes expected from the evaluation framework and its underlying conceptual model. RESULTS: The data support the role of engaged research partners in refinements to the research questions, selection of interventions to compare, choice of study outcomes and how they are measured, contributions to strategies for recruitment, and ensuring studies are patient-centered. CONCLUSIONS: The evaluation framework and the underlying conceptual model are supported by results to date. PCORI will continue to assess the effects of engagement as the funded projects progress toward completion, dissemination, and uptake into clinical decision making.

The PCORI Engagement Rubric: Promising Practices for Partnering in Research.

PURPOSE: Engaging patients, caregivers, and other health care stakeholders as partners in planning, conducting, and disseminating research is a promising way to improve clinical decision making and outcomes. Many researchers, patients, and other stakeholders, however, lack clarity about when and how to engage as partners within the clinical research process. To address the need for guidance on creating meaningful stakeholder partnerships in patient-centered clinical comparative effectiveness research, the Patient-Centered Outcomes Research Institute (PCORI) developed the PCORI Engagement Rubric (Rubric). METHODS: PCORI developed the Rubric drawing from a synthesis of the literature, a qualitative study with patients, a targeted review of engagement plans from PCORI-funded project applications, and a moderated discussion and review with PCORI's Advisory Panel on Patient Engagement. RESULTS: The Rubric provides a framework for operationalizing engagement to incorporate patients and other stakeholders in all phases of research. It includes: principles of engagement; definitions of stakeholder types; key considerations for planning, conducting, and disseminating engaged research; potential engagement activities; and examples of promising practices from PCORI-funded projects. CONCLUSIONS: PCORI designed the Rubric to illustrate opportunities for engagement to researchers interested in applying for PCORI funding and to patients and other stakeholders interested in greater involvement in research. By encouraging PCORI applicants, awardees, and others to apply the rubric, PCORI hopes to shift the research paradigm from one of conducting research on patients as subjects to a pursuit carried out in collaboration with patients and other stakeholders to better reflect the values, preferences, and outcomes that matter to the patient community.

Health policy, ethics, and the Kansas Legislative Health Academy.

We describe a unique program, the Kansas Legislative Health Academy, that brings together state legislators from across the political spectrum to build their capacity in advancing policies that can improve the health of Kansans. To that end, the academy helps legislators develop new skills to deliberate the ethics of health policy, use systems thinking to understand the long- and short-term effects of policy action and inaction, and engage in acts of civic leadership. The academy also seeks to foster an environment of respectful open dialogue and to build new cross-chamber and cross-party relationships. Among the most important outcomes cited by program participants is the value of sustained, personal interaction and problem solving with individuals holding differing political views.

Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute.

PURPOSE: To provide an overview of PCORI's approach to engagement in research. METHODS: The Patient-Centered Outcomes Research Institute (PCORI) was established in 2010 to fund patient-centered comparative effectiveness research. Requirements for research funding from PCORI include meaningful engagement of patients and other stakeholders in the research. PCORI's approach to engagement in research is guided by a conceptual model of patient-centered outcomes research (PCOR), that provides a structure for understanding engagement in research. RESULTS: To understand and improve engagement in research PCORI is learning from awardees and other stakeholders. Those efforts are described along with PCORI's capacity building and guidance to awardees via the Engagement Rubric. PCORI's unique model of engaging patients and other stakeholders in merit review of funding applications is also described. Additional support for learning about engagement in research is provided through specific research funding and through PCORI's major infrastructure initiative, PCORnet. CONCLUSION: PCORI requires engagement of stakeholders in the research it funds. In addition PCORI engages stakeholders in activities including review of funding applications and establishment of CER research infrastructure through PCORnet. The comprehensive approach to engagement is being evaluated to help guide the field toward promising practices in research engagement.

Data monitoring committees for pragmatic clinical trials.

In any clinical trial, it is essential to monitor the accumulating data to be sure that the trial continues to be safe for participants and that the trial is being conducted properly. Data monitoring committees, independent expert panels who undertake regular reviews of the data as the trial progresses, serve an important role in safeguarding the interests of research participants and ensuring trial integrity in many trials. Many pragmatic clinical trials, which aim to inform healthcare decisions by comparing alternate interventions in heterogeneous healthcare delivery settings, will warrant review by an independent data monitoring committee due to their potential impact on clinical practice. However, the very features that make a trial "pragmatic" may pose challenges in terms of which aspects of a trial to monitor and when it is appropriate for a data monitoring committee to intervene. Using the Pragmatic-Explanatory Continuum Indicator Summary tool that draws distinctions between pragmatic and explanatory clinical trials, we review characteristics of pragmatic clinical trials that may have implications for data monitoring committees and interim monitoring plans. These include broad eligibility criteria, a focus on subjective patient-centered outcomes, and in some cases a lack of standardized follow-up procedures across study sites. Additionally, protocol adherence is often purposefully not addressed in pragmatic trials in order to accurately represent the clinical practice setting and maintain practicability of implementation; there are differing viewpoints as to whether adherence should be assessed and acted upon by data monitoring committees in these trials. Some other issues not specifically related to the Pragmatic-Explanatory Continuum Indicator Summary criteria may also merit special consideration in pragmatic trials. Thresholds for early termination of a pragmatic clinical trial might be controversial. The distinguishing features of pragmatic clinical trials require careful consideration when developing interim data monitoring plans, and trial sponsors, investigators, and data monitoring committees should agree on a plan before trial inception. Finally, special expertise, such as an informatics, may be helpful on data monitoring committees for some pragmatic clinical trials. Patient representatives may provide particularly valuable insights in the monitoring process.

Treatment goals for rheumatoid arthritis: patient engagement and goal collection.

Aim: We developed the Patient-Engaged Health Technology Assessment strategy for survey-based goal collection from patients to yield patient-important outcomes suitable for use in multi-criteria decision analysis. Methods: Rheumatoid arthritis patients were recruited from online patient networks for proof-of-concept testing of goal collection and prioritization using a survey. A Project Steering Committee and Expert Panel rated the feasibility of scaling to larger samples. Results: Survey respondents (n = 47) completed the goal collection exercise. Finding effective treatments was rated by respondents as the most important goal, and reducing stiffness was rated as the least important. Feedback from our steering committee and expert panel support the approach's feasibility for goal identification and ranking. Conclusion: Goals relevant for treatment evaluation can be identified and rated for importance by patients to permit wide input from patients with lived experience of disease.

Human centered design workshops as a meta-solution to diagnostic disparities.

OBJECTIVES: Diagnostic errors - inaccurate or untimely diagnoses or failures to communicate diagnoses - are harmful and costly for patients and health systems. Diagnostic disparities occur when diagnostic errors are experienced at disproportionate rates by certain patient subgroups based, for example, on patients' age, sex/gender, or race/ethnicity. We aimed to develop and test the feasibility of a human centered design workshop series that engages diverse stakeholders to develop solutions for mitigating diagnostic disparities. METHODS: We employed a series of human centered design workshops supplemented by semi-structured interviews and literature evidence scans. Co-creation sessions and rapid prototyping by patient, clinician, and researcher stakeholders were used to generate design challenges, solution concepts, and prototypes. RESULTS: A series of four workshops attended by 25 unique participants was convened in 2019-2021. Workshops generated eight design challenges, envisioned 29 solutions, and formulated principles for developing solutions in an equitable, patient-centered manner. Workshops further resulted in the conceptualization of 37 solutions for addressing diagnostic disparities and prototypes for two of the solutions. Participants agreed that the workshop processes were replicable and could be implemented in other settings to allow stakeholders to generate context-specific solutions. CONCLUSIONS: The incorporation of human centered design through a series of workshops promises to be a productive way of engaging patient-researcher stakeholders to mitigate and prevent further exacerbation of diagnostic disparities. Healthcare stakeholders can apply human centered design principles to guide thinking about improving diagnostic performance and to center diverse patients' needs and experiences when implementing quality and safety improvements.

Engaging patients and parents to improve mental health intervention for youth with rheumatological disease.

BACKGROUND: Mental health disorders are common in youth with rheumatological disease yet optimal intervention strategies are understudied in this population. We examined patient and parent perspectives on mental health intervention for youth with rheumatological disease. METHODS: We conducted a mixed methods cross-sectional study, via anonymous online survey, developed by researchers together with patient/parent partners, to quantitatively and qualitatively examine youth experiences with mental health services and resources in North America. Patients ages 14-24 years with juvenile idiopathic arthritis, juvenile dermatomyositis, or systemic lupus erythematous, and parents of patients ages 8-24 with these diseases were eligible (not required to participate in pairs). Participants self-reported mental health problems (categorized into clinician-diagnosed disorders vs self-diagnosed symptoms) and treatments (e.g. therapy, medications) received for the youth. Multivariate linear regression models compared patient and parent mean Likert ratings for level of: i) comfort with mental health providers, and ii) barriers to seeking mental health services, adjusting for potential confounders (patient age, gender, disease duration, and patient/parent visual analog score for disease-related health). Participants indicated usefulness of mental health resources; text responses describing these experiences were analyzed by qualitative description. RESULTS: Participants included 123 patients and 324 parents. Patients reported clinician-diagnosed anxiety (39%) and depression (35%); another 27 and 18% endorsed self-diagnosed symptoms of these disorders, respectively. 80% of patients with clinician-diagnosed disorders reported receiving treatment, while 11% of those with self-diagnosed symptoms reported any treatment. Patients were less comfortable than parents with all mental health providers. The top two barriers to treatment for patients and parents were concerns about mental health providers not understanding the rheumatological disease, and inadequate insurance coverage. Over 60% had used patient mental health resources, and over 60% of these participants found them to be helpful, although text responses identified a desire for resources tailored to patients with rheumatological disease. CONCLUSION: Self-reported mental health problems are prevalent for youth in this sample with rheumatological disease, and obstacles to mental health treatment include disease-related and logistic factors. Strategies are needed to improve acceptance and accessibility of mental health intervention, including routine mental health screening and availability of disease-specific mental health resources.

Identifying Research Priorities among Patients and Families of Children with Rheumatic Diseases Living in the United States.

OBJECTIVE: To improve the quality and participation in pediatric rheumatology research, patient-prioritized studies should be emphasized. We collaborated with United States-based pediatric rheumatology advocacy organizations to survey patients and caregivers of children with rheumatic diseases to identify what research topics were most important to them. METHODS: We conducted Web-based surveys and focus groups (FG) of patients and caregivers of children with juvenile myositis (JM), juvenile arthritis (JA), and childhood-onset systemic lupus erythematosus (cSLE). Surveys were emailed to listservs and posted to social media sites of JM, JA, and cSLE patient advocacy organizations. An initial survey asked open-ended questions about patient/caregiver research preferences. Responses were further characterized through FG. A final ranking survey asked respondents to rank from a list of research themes the 7 most important to them. RESULTS: There were 365 JM respondents, 44 JA respondents, and 32 cSLE respondents to the final ranking survey. The top research priority for JM was finding new treatments, and for JA and cSLE, the priority was understanding genetic/environmental etiology. The 3 prioritized research themes common across all disease groups were medication side effects, disease flare, and disease etiology. CONCLUSION: Patient-centered research prioritization is recognized as valuable in conducting high-quality research, yet there is a paucity of data describing patient/family preferences, especially in pediatrics. We used multimodal methodologies to assess current patient/caregiver research priorities to help frame the agenda for the pediatric rheumatology research community. Patients and caregivers from all surveyed disease groups prioritized the study of medication side effects, disease flares, and disease etiology.

Toward Accelerated Authorization and Access to New Medicines for Juvenile Idiopathic Arthritis.

A meeting was organized to bring together multiple stakeholders involved in the testing and authorization of new medications for juvenile idiopathic arthritis (JIA) to discuss current issues surrounding clinical trials and access to new medications for children and adolescents with JIA. The Childhood Arthritis and Rheumatology Research Alliance invited representatives of regulatory agencies (Food and Drug Administration and European Medicines Agency), and major pharmaceutical companies with JIA-approved products or products in development, patient and parent representatives, representatives of an advocacy organization (Arthritis Foundation), and pediatric rheumatology clinicians/investigators to a 1-day meeting in April 2018. The participants engaged in discussion regarding issues in clinical trials. As the pharmacologic options to treat inflammatory arthritis rapidly expand, registration trial designs to test medications in JIA patients must adapt. Many methodologies successfully used in the recent past are no longer feasible. The pool of patients meeting entry criteria who are willing to participate is shrinking while the number of medications to be tested is growing. Suggested solutions included proposing innovative clinical trial methods to regulatory agencies, as well as open discussions among stakeholders. Ensuring that new medications are authorized in a timely manner to meet the needs of JIA patients worldwide is critical. Approaches should include open dialog between regulatory agencies, pharmaceutical companies, and other stakeholders to develop and implement novel study designs, including patient and clinician perspectives to define meaningful trial outcomes, and changing existing study plans.