Evaluation of patient engagement in medicine development: A multi-stakeholder framework with metrics.

BACKGROUND: Patient engagement is becoming more customary in medicine development. However, embedding it in organizational decision-making remains challenging, partly due to lack of agreement on its value and the means to evaluate it. The objective of this project was to develop a monitoring and evaluation framework, with metrics, to demonstrate impact and enhance learning. METHODS: A consortium of five patient groups, 15 biopharmaceutical companies and two academic groups iteratively created a framework in a multi-phase participatory process, including analysis of its application in 24 cases. RESULTS: The framework includes six components, with 87 metrics and 15 context factors distributed among (sub)components: (a) Input: expectations, preparations, resources, representativeness of stakeholders; (b) Activities/process: structure, management, interactions, satisfaction; (c) Learnings and changes; (d) Impacts: research relevance, study ethics and inclusiveness, study quality and efficiency, quality of evidence and uptake of products, empowerment, reputation and trust, embedding of patient engagement; (e) Context: policy, institutional, community, decision-making contextual factors. Case study findings show a wide variation in use of metrics. There is no 'one size fits all' set of metrics appropriate for every initiative or organization. Presented sample sets of metrics can be tailored to individual situations. CONCLUSION: Introducing change into any process is best done when the value of that change is clear. This framework allows participants to select what metrics they value and assess to what extent patient engagement has contributed. PATIENT CONTRIBUTION: Five patient groups were involved in all phases of the study (design, conduct, interpretation of data) and in writing the manuscript.

Monitoring and Evaluation of Patient Engagement in Health Product Research and Development: Co-Creating a Framework for Community Advisory Boards.

PURPOSE: While patient engagement is becoming more customary in developing health products, its monitoring and evaluation to understand processes and enhance impact are challenging. This article describes a patient engagement monitoring and evaluation (PEME) framework, co-created and tailored to the context of community advisory boards (CABs) for rare diseases in Europe. It can be used to stimulate learning and evaluate impacts of engagement activities. METHODS: A participatory approach was used in which data collection and analysis were iterative. The process was based on the principles of interactive learning and action and guided by the PEME framework. Data were collected via document analysis, reflection sessions, a questionnaire, and a workshop. RESULTS: The tailored framework consists of a theory of change model with metrics explaining how CABs can reach their objectives. Of 61 identified metrics, 17 metrics for monitoring the patient engagement process and short-term outcomes were selected, and a "menu" for evaluating long-term impacts was created. Example metrics include "Industry representatives' understanding of patients' unmet needs;" "Feeling of trust between stakeholders;" and "Feeling of preparedness." "Alignment of research programs with patients' needs" was the highest-ranked metric for long-term impact. CONCLUSIONS: Findings suggest that process and short-term outcome metrics could be standardized across CABs, whereas long-term impact metrics may need to be tailored to the collaboration from a proposed menu. Accordingly, we recommend that others adapt and refine the PEME framework as appropriate. The next steps include implementing and testing the evaluation framework to stimulate learning and share impacts.

Estrogen Receptor Pathway Activity Score to Predict Clinical Response or Resistance to Neoadjuvant Endocrine Therapy in Primary Breast Cancer.

Endocrine therapy is important for management of patients with estrogen receptor (ER)-positive breast cancer; however, positive ER staining does not reliably predict therapy response. We assessed the potential to improve prediction of response to endocrine treatment of a novel test that quantifies functional ER pathway activity from mRNA levels of ER pathway-specific target genes. ER pathway activity was assessed on datasets from three neoadjuvant-treated ER-positive breast cancer patient cohorts: Edinburgh: 3-month letrozole, 55 pre-/2-week/posttreatment matched samples; TEAM IIa: 3- to 6-month exemestane, 49 pre-/28 posttreatment paired samples; and NEWEST: 16-week fulvestrant, 39 pretreatment samples. ER target gene mRNA levels were measured in fresh-frozen tissue (Edinburgh, NEWEST) with Affymetrix microarrays, and in formalin-fixed paraffin-embedded samples (TEAM IIa) with qRT-PCR. Approximately one third of ER-positive patients had a functionally inactive ER pathway activity score (ERPAS), which was associated with a nonresponding status. Quantitative ERPAS decreased significantly upon therapy (P < 0.001 Edinburgh and TEAM IIa). Responders had a higher pretreatment ERPAS and a larger 2-week decrease in activity (P = 0.02 Edinburgh). Progressive disease was associated with low baseline ERPAS (P = 0.03 TEAM IIa; P = 0.02 NEWEST), which did not decrease further during treatment (P = 0.003 TEAM IIa). In contrast, the staining-based ER Allred score was not significantly associated with therapy response (P = 0.2). The ERPAS identified a subgroup of ER-positive patients with a functionally inactive ER pathway associated with primary endocrine resistance. Results confirm the potential of measuring functional ER pathway activity to improve prediction of response and resistance to endocrine therapy.