The use of implementation science theories, models, and frameworks in implementation research for medicinal products: A scoping review.

BACKGROUND: The uptake, adoption and integration of new medicines and treatment regimens within healthcare delivery can take a decade or more. Increasingly, implementation science (IS) research is being used to bridge this gap between the availability of new therapeutic evidence and its actual application in clinical practice. Little is known, however, about the quality of IS research in this area, including the degree to which theories, models and frameworks (TMFs) are being used. The objective of this study was to conduct a scoping review of the use of TMFs in implementation research involving medicinal products. METHODS: A search was conducted for English language abstracts and manuscripts describing the application of TMFs in IS studies for medicinal products. Eligible publications were those published between 1 January 1974 and 12 December 2022. All records were screened at the title and abstract stage; included full-text papers were abstracted using data extraction tables designed for the study. Study quality was appraised using the Implementation Research Development Tool. RESULTS: The initial scoping search identified 2697 publications, of which 9 were ultimately eligible for inclusion in the review. Most studies were published after 2020 and varied in their objectives, design and therapeutic area. Most studies had sample sizes of fewer than 50 participants, and all focused on the post-marketing phase of drug development. The TMF most frequently used was the Consolidated Framework for Implementation Research (CFIR). Although most studies applied all TMF domains, TMF use was limited to instrument development and/or qualitative analysis. Quality appraisals indicated the need for engaging patients and other stakeholders in the implementation research, reporting on the cost of implementation strategies, and evaluating the unintended consequences of implementation efforts. CONCLUSIONS: We found that few IS studies involving medicinal products reported using TMFs. Those that did encompassed a wide variety of therapeutic indications and medicinal products; all were in the post-marketing phase and involved limited application of the TMFs. Researchers should consider conducting IS in earlier phases of drug development and integrating the TMFs throughout the research process. More consistent and in-depth use of TMFs may help advance research in this area.

A citation analysis and scoping systematic review of the operationalization of the Practical, Robust Implementation and Sustainability Model (PRISM).

BACKGROUND: The Practical, Robust Implementation and Sustainability Model (PRISM) was developed in 2008 as a contextually expanded version of the broadly used Reach, Adoption, Effectiveness, Implementation, and Maintenance (RE-AIM) framework. PRISM provides researchers a pragmatic and intuitive model to improve translation of research interventions into clinical and community practice. Since 2008, the use of PRISM increased across diverse topics, populations, and settings. This citation analysis and scoping systematic review aimed to assess the use of the PRISM framework and to make recommendations for future research. METHODS: A literature search was conducted using three databases (PubMed, Web of Science, Scopus) for the period of 2008 and September 2020. After exclusion, reverse citation searches and invitations to experts in the field were used to identify and obtain recommendations for additional articles not identified in the original search. Studies that integrated PRISM into their study design were selected for full abstraction. Unique research studies were abstracted for information on study characteristics (e.g., setting/population, design), PRISM contextual domains, and RE-AIM outcomes. RESULTS: A total of 180 articles were identified to include PRISM to some degree. Thirty-two articles representing 23 unique studies integrated PRISM within their study design. Study characteristics varied widely and included studies conducted in diverse contexts, but predominately in high-income countries and in clinical out-patient settings. With regards to use, 19 used PRISM for evaluation, 10 for planning/development, 10 for implementation, four for sustainment, and one for dissemination. There was substantial variation across studies in how and to what degree PRISM contextual domains and RE-AIM outcomes were operationalized and connected. Only two studies directly connected individual PRISM context domains with RE-AIM outcomes, and another four included RE-AIM outcomes without direct connection to PRISM domains. CONCLUSIONS: This is the first systematic review of the use of PRISM in various contexts. While there were low levels of 'integrated' use of PRISM and few reports on linkage to RE-AIM outcomes, most studies included important context domains of implementation and sustainability infrastructure and external environment. Recommendations are provided for more consistent and comprehensive use of and reporting on PRISM to inform both research and practice on contextual factors in implementation.

Understanding and applying the RE-AIM framework: Clarifications and resources.

INTRODUCTION: Understanding, categorizing, and using implementation science theories, models, and frameworks is a complex undertaking. The issues involved are even more challenging given the large number of frameworks and that some of them evolve significantly over time. As a consequence, researchers and practitioners may be unintentionally mischaracterizing frameworks or basing actions and conclusions on outdated versions of a framework. METHODS: This paper addresses how the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework has been described, summarizes how the model has evolved over time, and identifies and corrects several misconceptions. RESULTS: We address 13 specific areas where misconceptions have been noted concerning the use of RE-AIM and summarize current guidance on these issues. We also discuss key changes to RE-AIM over the past 20 years, including the evolution to Pragmatic Robust Implementation and Sustainability Model, and provide resources for potential users to guide application of the framework. CONCLUSIONS: RE-AIM and many other theories and frameworks have evolved, been misunderstood, and sometimes been misapplied. To some degree, this is inevitable, but we conclude by suggesting some actions that reviewers, framework developers, and those selecting or applying frameworks can do to prevent or alleviate these problems.

Comparative Clinical Effectiveness Research Focused on Community-Based Delivery of Palliative Care: Overview of the Patient-Centered Outcomes Research Institute's Funding Initiative.

Palliative care is a growing specialty that addresses the needs of individuals diagnosed with advanced illness and their caregivers. Although palliative care has been shown to improve a variety of patient- and caregiver-centered outcomes, access to comprehensive palliative care services for patients is often limited. There is a need to identify the most effective approaches to delivering palliative care to patients in community settings. In fiscal year 2017, based on extensive input from a diverse set of stakeholders, the Patient-Centered Outcomes Research Institute (PCORI) funded nine multisite comparative clinical effectiveness research (CER) trials focused on community-based delivery of palliative care for a total investment of $80 million. These studies, focusing on advance care planning and models of palliative care delivery, represent some of the largest most complex palliative care trials funded to date. Each study evaluates both patient and caregiver outcomes, and together, these trials include a broad range of health conditions, interventions, and settings of care. PCORI has also fostered a learning network of the funded awardees to facilitate the successful conduct of these CER studies and to support awardee efforts to develop collaborative products relevant to advancing the field of palliative care research and practice. The protocols of each of the nine trials, detailed in this issue, demonstrate the expansive reach of the investment PCORI has made in an effort to further the research agenda and provide substantive research evidence in stakeholder-identified areas of need in the field of palliative care.

RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review.

The RE-AIM planning and evaluation framework was conceptualized two decades ago. As one of the most frequently applied implementation frameworks, RE-AIM has now been cited in over 2,800 publications. This paper describes the application and evolution of RE-AIM as well as lessons learned from its use. RE-AIM has been applied most often in public health and health behavior change research, but increasingly in more diverse content areas and within clinical, community, and corporate settings. We discuss challenges of using RE-AIM while encouraging a more pragmatic use of key dimensions rather than comprehensive applications of all elements. Current foci of RE-AIM include increasing the emphasis on cost and adaptations to programs and expanding the use of qualitative methods to understand "how" and "why" results came about. The framework will continue to evolve to focus on contextual and explanatory factors related to RE-AIM outcomes, package RE-AIM for use by non-researchers, and integrate RE-AIM with other pragmatic and reporting frameworks.

National Working Group on the RE-AIM Planning and Evaluation Framework: Goals, Resources, and Future Directions.

The National Working Group on RE-AIM Planning and Evaluation Framework (herein Workgroup) was established in 2004 to support the application of the framework and advance dissemination and implementation science (D&I). Workgroup members developed and disseminated products and resources (and continue to do so) to advocate for consistent application of RE-AIM and allow for cross study comparisons. The purpose of this paper is to summarize key Workgroup activities, products, and services (e.g., webinars, consultations, planning tools) and enhance bidirectional communication between the Workgroup and RE-AIM users. The ultimate goal of this work is to serve as a forum for dissemination to improve the balance between RE-AIM user demand (needs) and the currently limited RE-AIM Workgroup supply (consultation and resources) to demonstrate and expand the utility of RE-AIM as a D&I planning and evaluation framework. A summary of resources is provided as well as specific examples of how the Workgroup has been responsive to user needs.

Development, Feasibility, and Small-Scale Implementation of a Web-Based Prognostic Tool-Surveillance, Epidemiology, and End Results Cancer Survival Calculator.

BACKGROUND: Population datasets and the Internet are playing an ever-growing role in the way cancer information is made available to providers, patients, and their caregivers. The Surveillance, Epidemiology, and End Results Cancer Survival Calculator (SEER*CSC) is a Web-based cancer prognostic tool that uses SEER data, a large population dataset, to provide physicians with highly valid, evidence-based prognostic estimates for increasing shared decision-making and improving patient-provider communication of complex health information. OBJECTIVE: The aim of this study was to develop, test, and implement SEER*CSC. METHODS: An iterative approach was used to develop the SEER*CSC. Based on input from cancer patient advocacy groups and physicians, an initial version of the tool was developed. Next, providers from 4 health care delivery systems were recruited to do formal usability testing of SEER*CSC. A revised version of SEER*CSC was then implemented in two health care delivery sites using a real-world clinical implementation approach, and usage data were collected. Post-implementation follow-up interviews were conducted with site champions. Finally, patients from two cancer advocacy groups participated in usability testing. RESULTS: Overall feedback of SEER*CSC from both providers and patients was positive, with providers noting that the tool was professional and reliable, and patients finding it to be informational and helpful to use when discussing their diagnosis with their provider. However, use during the small-scale implementation was low. Reasons for low usage included time to enter data, not having treatment options in the tool, and the tool not being incorporated into the electronic health record (EHR). Patients found the language in its current version to be too complex. CONCLUSIONS: The implementation and usability results showed that participants were enthusiastic about the use and features of SEER*CSC, but sustained implementation in a real-world clinical setting faced significant challenges. As a result of these findings, SEER*CSC is being redesigned with more accessible language for a public facing release. Meta-tools, which put different tools in context of each other, are needed to assist in understanding the strengths and limitations of various tools and their place in the clinical decision-making pathway. The continued development and eventual release of prognostic tools should include feedback from multidisciplinary health care teams, various stakeholder groups, patients, and caregivers.

An overview and discussion of the Patient-Centered Outcomes Research Institute's decision aid portfolio.

Decision aids (DAs) help patients make informed healthcare decisions in a manner consistent with their values and preferences. Despite their promise, DAs developed with public research dollars are not being implemented and adopted in real-world patient care settings at a rate consistent with which they are being developed. To appraise the sum of the parts of the portfolio and create a strategic imperative surrounding future funding, the Patient-Centered Outcomes Research Institute (PCORI) tasked the Duke Evidence Synthesis Group with evaluating its DA portfolio. This paper describes PCORI's portfolio of DAs according to the Duke Evidence Synthesis Group's analysis in the context of PCORI's mission and the field of decision science. The results revealed a diversity within PCORI's portfolio of funded DA projects. Findings support the movement toward more rigorous DA development, assessment and maintenance. PCORI's funding priorities related to DAs are clarified and comparative questions of interest are posed.

Health literacy-listening skill and patient questions following cancer prevention and screening discussions.

OBJECTIVE: Patient question-asking is essential to shared decision making. We sought to describe patients' questions when faced with cancer prevention and screening decisions, and to explore differences in question-asking as a function of health literacy with respect to spoken information (health literacy-listening). METHODS: Four-hundred and thirty-three (433) adults listened to simulated physician-patient interactions discussing (i) prophylactic tamoxifen for breast cancer prevention, (ii) PSA testing for prostate cancer and (iii) colorectal cancer screening, and identified questions they would have. Health literacy-listening was assessed using the Cancer Message Literacy Test-Listening (CMLT-Listening). Two authors developed a coding scheme, which was applied to all questions. Analyses examined whether participants scoring above or below the median on the CMLT-Listening asked a similar variety of questions. RESULTS: Questions were coded into six major function categories: risks/benefits, procedure details, personalizing information, additional information, decision making and credibility. Participants who scored higher on the CMLT-Listening asked a greater variety of risks/benefits questions; those who scored lower asked a greater variety of questions seeking to personalize information. This difference persisted after adjusting for education. CONCLUSION: Patients' health literacy-listening is associated with distinctive patterns of question utilization following cancer screening and prevention counselling. Providers should not only be responsive to the question functions the patient favours, but also seek to ensure that the patient is exposed to the full range of information needed for shared decision making.

Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review.

BACKGROUND: The reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework was developed to determine potential public health impact of interventions (i.e., programs, policy, and practice). The purpose of this systematic review was to determine (1) comparative results across accurately reported RE-AIM indicators, (2) relevant information when there remains under-reporting or misclassification of data across each dimension, (3) the degree to which authors intervened to improve outcomes related to each dimension, and (4) the number of articles reporting RE-AIM dimensions for a given study. METHODS: In April 2013, a systematic search of the RE-AIM framework was completed in PubMed, PSYCHInfo, EbscoHost, Web of Science, and Scopus. Evidence was analyzed until January 2015. RESULTS: Eighty-two interventions that included empirical data related to at least one of the RE-AIM dimensions were included in the review. Across these interventions, they reached a median sample size of 320 participants (M = 4894 ± 28,256). Summarizing the effectiveness indicators, we found that: the average participation rate was 45 % (±28 %), 89 % of the interventions reported positive changes in the primary outcome and 11 interventions reported broader outcomes (e.g., quality of life). As for individual-level maintenance, 11 % of studies showed effects ≥6 months post-program. Average setting and staff adoption rates were 75 % (±32 %) and 79 % (±28 %), respectively. Interventions reported being delivered as intended (82 % (±16 %)) and 22 % intervention reported adaptations to delivery. There were insufficient data to determine average maintenance at the organizational level. Data on costs associated with each dimension were infrequent and disparate: four studies reported costs of recruitment, two reported intervention costs per participant, and two reported adoption costs. CONCLUSIONS: The RE-AIM framework has been employed in a variety of populations and settings for the planning, delivery, and evaluation of behavioral interventions. This review highlights inconsistencies in the degree to which authors reported each dimension in its entirety as well as inaccuracies in reporting indicators within each dimension. Further, there are few interventions that aim to improve outcomes related to reach, adoption, implementation, and maintenance.

Network analysis of RE-AIM framework: chronology of the field and the connectivity of its contributors.

The reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework has been widely used for translational research. We used social network analysis (SNA) to explore how innovative research frameworks, such as RE-AIM, have diffused over time in academic literature. A structured literature review was conducted on RE-AIM between 1999 and 2012. SNA indices of degree score, betweenness, centrality, and authorship ties were used to examine use of RE-AIM. Use of RE-AIM has grown since its inception and spread from a few research centers to use internationally. Investigation of co-authorship revealed many have published on RE-AIM, but a much smaller core of RE-AIM researchers have published together two or more times. SNA revealed how the RE-AIM framework has been used over time and identified areas to further expand use of the framework. SNA can be useful to understand how research frameworks diffuse over time.

E-mail to Promote Colorectal Cancer Screening Within Social Networks: Acceptability and Content.

Effective techniques to encourage colorectal cancer screening in underscreened populations have included social support interventions and e-mail reminders from physicians. Personalized e-mail messages to promote colorectal cancer screening within social networks could be even more effective but have not been studied. The authors interviewed 387 e-mail users, aged 42-73 years in Georgia, Hawaii, and Massachusetts. Participants were asked to edit a sample message in which the sender shares a recent colonoscopy experience and urges the recipient to discuss colorectal cancer screening with a doctor. For those reporting willingness to send this message, changes to the message and suggested subject lines were recorded. Edited text was analyzed for content and concordance with original message. The majority of participants (74.4%) were willing to e-mail a modifiable message. Of those willing, 63.5% edited the message. Common edits included deletion (17.7%) or modification (17.4%) of a negatively framed sentence on colon cancer risks and addition or modification of personalizing words (15.6%). Few edits changed the meaning of the message (5.6%), and even fewer introduced factual inaccuracies (1.7%). Modifiable e-mail messages offer a way for screened individuals to promote colorectal cancer screening to social network members. The accuracy and effects of such messages should be further studied.

How pragmatic is it? Lessons learned using PRECIS and RE-AIM for determining pragmatic characteristics of research.

BACKGROUND: The need for high-quality evidence that is applicable in real-world, routine settings continues to increase. Pragmatic trials are designed to evaluate the effectiveness of interventions in real-world settings, whereas explanatory trials aim to test whether an intervention works under optimal situations. There is a continuum between explanatory and pragmatic trials. Most trials have aspects of both, making it challenging to label and categorize a trial and to evaluate its potential for translation into practice. METHODS: We summarize our experience applying the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) combined with external validity items based on the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to three studies to provide a more robust and comprehensive assessment of trial characteristics related to translation of research. We summarize lessons learned using domains from the combined frameworks for use in study planning, evaluating specific studies, and reviewing the literature and make recommendations for future use. RESULTS: A variety of coders can be trained to use the PRECIS and RE-AIM domains. These domains can also be used for diverse purposes, content areas, and study types, but are not without challenges. Both PRECIS and RE-AIM domains required modification in two of the three studies to evaluate and rate domains specific to study type. Lessons learned involved: dedicating enough time for training activities related to the domains; use of reviewers with a range of familiarity with specific study protocols; how to best adapt ratings that reflect complex study designs; and differences of opinion regarding the value of creating a composite score for these criteria. CONCLUSIONS: Combining both frameworks can specifically help identify where and how a study is and is not pragmatic. Using both PRECIS and RE-AIM allows for standard reporting of key study characteristics related to pragmatism and translation. Such measures should be used more consistently to help plan more pragmatic studies, evaluate progress, increase transparency of reporting, and integrate literature to facilitate translation of research into practice and policy.

A systematic review of eHealth cancer prevention and control interventions: new technology, same methods and designs?

There has been a recent surge of eHealth programs in cancer and other content areas, but few reviews have focused on the methodologies and designs employed in these studies. We conducted a systematic review of studies on eHealth interventions on cancer prevention and control published between 2001 and 2010 applying the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) criteria and external validity components from the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework. We identified 113 studies that focused on cancer prevention and control of eHealth interventions. Most studies fell midway along the explanatory/pragmatic trial continuum, but few reported on various practical feasibility criteria for translation. Despite vast interest in cancer eHealth and the applied nature of this field, few studies considered key external validity issues. There is a need for use of alternative pragmatic study designs and transparent reporting of external validity components to produce more rapid and generalizable results.

Adult Willingness to Use Email and Social Media for Peer-to-Peer Cancer Screening Communication: Quantitative Interview Study.

BACKGROUND: Adults over age 40 are increasing their use of email and social media, raising interest in use of peer-to-peer Internet-based messaging to promote cancer screening. OBJECTIVE: The objective of our study was to assess current practices and attitudes toward use of email and other e-communication for peer-to-peer dialogues on cancer screening. METHODS: We conducted in-person interviews with 438 insured adults ages 42-73 in Georgia, Hawaii, and Massachusetts. Participants reported on use of email and other e-communication including social media to discuss with peers routine health topics including breast and colorectal cancer (CRC). We ascertained willingness to share personal CRC screening experiences via conversation, postcard, email, or other e-communication. Health literacy scores were measured. RESULTS: Email had been used by one-third (33.8%, 148/438) to discuss routine health topics, by 14.6% (64/438) to discuss breast cancer screening, and by 12.6% (55/438) to discuss CRC screening. Other e-communication was used to discuss routine health topics (11.6%, 51/438), screening for breast cancer (3.9%, 17/438), and CRC (2.3%, 10/438). In the preceding week, 84.5% (370/438) of participants had used email, 55.9% (245/438) had used e-communication of some type; 44.3% (194/438) text, 32.9% (144/438) Facebook, 12.3% (54/438) instant message, 7.1% (31/438) video chat, and 4.8% (21/438) Twitter. Many participants were willing to share their CRC screening experiences via email (32.4%, 142/438 might be willing; 36.3%, 159/438 very willing) and via other e-communication (15.8%, 69/438 might be willing; 14.4%, 63/438 very willing). Individuals willing to send CRC screening emails scored significantly higher on tests of health literacy compared to those willing to send only postcards (P<.001). CONCLUSIONS: Many adults are willing to use email and e-communication to promote cancer screening to peers. Optimal approaches for encouraging peer-to-peer transmission of accurate and appropriate cancer screening messages must be studied.

The association between health literacy and cancer-related attitudes, behaviors, and knowledge.

Using a multidimensional assessment of health literacy (the Cancer Message Literacy Test-Listening, the Cancer Message Literacy Test-Reading, and the Lipkus Numeracy Scale), the authors assessed a stratified random sample of 1013 insured adults (40-70 years of age). The authors explored whether low health literacy across all 3 domains (n =111) was associated with sets of variables likely to affect engagement in cancer prevention and screening activities: (a) attitudes and behaviors relating to health care encounters and providers, (b) attitudes toward cancer and health, (c) knowledge of cancer screening tests, and (d) attitudes toward health related media and actual media use. Adults with low health literacy were more likely to report avoiding doctor's visits, to have more fatalistic attitudes toward cancer, to be less accurate in identifying the purpose of cancer screening tests, and more likely to avoid information about diseases they did not have. Compared with other participants, those with lower health literacy were more likely to say that they would seek information about cancer prevention or screening from a health care professional and less likely to turn to the Internet first for such information. Those with lower health literacy reported reading on fewer days and using the computer on fewer days than did other participants. The authors assessed the association of low health literacy with colorectal cancer screening in an age-appropriate subgroup for which colorectal cancer screening is recommended. In these insured subjects receiving care in integrated health care delivery systems, those with low health literacy were less likely to be up to date on screening for colorectal cancer, but the difference was not statistically significant.

Predicting cancer prognosis using interactive online tools: a systematic review and implications for cancer care providers.

Cancer prognosis is of keen interest for patients with cancer, their caregivers, and providers. Prognostic tools have been developed to guide patient-physician communication and decision-making. Given the proliferation of prognostic tools, it is timely to review existing online cancer prognostic tools and discuss implications for their use in clinical settings. Using a systematic approach, we searched the Internet, Medline, and consulted with experts to identify existing online prognostic tools. Each was reviewed for content and format. Twenty-two prognostic tools addressing 89 different cancers were identified. Tools primarily focused on prostate (n = 11), colorectal (n = 10), breast (n = 8), and melanoma (n = 6), although at least one tool was identified for most malignancies. The input variables for the tools included cancer characteristics (n = 22), patient characteristics (n = 18), and comorbidities (n = 9). Effect of therapy on prognosis was included in 15 tools. The most common predicted outcome was cancer-specific survival/mortality (n = 17). Only a few tools (n = 4) suggested patients as potential target users. A comprehensive repository of online prognostic tools was created to understand the state-of-the-art in prognostic tool availability and characteristics. Use of these tools may support communication and understanding about cancer prognosis. Dissemination, testing, refinement of existing, and development of new tools under different conditions are needed.