RE-AIM for rural health innovations: perceptions of (mis) alignment between the RE-AIM framework and evaluation reporting in the Department of Veterans Affairs Enterprise-Wide Initiatives program.

Background: The Department of Veterans Affairs (VA) Office of Rural Health (ORH) supports national VA program offices' efforts to expand health care to rural Veterans through its Enterprise-Wide Initiatives (EWIs) program. In 2017, ORH selected Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM), an implementation science framework, to structure the EWI evaluation and reporting process. As part of its mandate to improve EWI program evaluation, the Center for the Evaluation of Enterprise-Wide Initiatives conducted a qualitative evaluation to better understand EWI team' perceptions of, and barriers and facilitators to, the EWI evaluation process. Methods: We conducted 43 semi-structured interviews with 48 team members (e.g., evaluators, program office leads, and field-based leads) representing 21 EWIs from April-December 2020. Questions focused on participants' experiences using strategies targeting each RE-AIM dimension. Interviews were inductively analyzed in MAXQDA. We also systematically reviewed 51 FY19-FY20 EWI annual reports to identify trends in misapplications of RE-AIM. Results: Participants had differing levels of experience with RE-AIM. While participants understood ORH's rationale for selecting a common framework to structure evaluations, the perceived misalignment between RE-AIM and EWIs' work emerged as an important theme. Concerns centered around 3 sub-themes: (1) (Mis)Alignment with RE-AIM Dimensions, (2) (Mis)Alignment between RE-AIM and the EWI, and (3) (Mis)Alignment with RE-AIM vs. other Theories, Models, or Frameworks. Participants described challenges differentiating between and operationalizing dimensions in unique contexts. Participants also had misconceptions about RE-AIM and its relevance to their work, e.g., that it was meant for established programs and did not capture aspects of initiative planning, adaptations, or sustainability. Less commonly, participants shared alternative models or frameworks to RE-AIM. Despite criticisms, many participants found RE-AIM useful, cited training as important to understanding its application, and identified additional training as a future need. Discussion: The selection of a shared implementation science framework can be beneficial, but also challenging when applied to diverse initiatives or contexts. Our findings suggest that establishing a common understanding, operationalizing framework dimensions for specific programs, and assessing training needs may better equip partners to integrate a shared framework into their evaluations.

Applying Translational Science Approaches to Protect Workers Exposed to Nanomaterials.

Like nanotechnology, translational science is a relatively new and transdisciplinary field. Translational science in occupational safety and health (OSH) focuses on the process of taking scientific knowledge for the protection of workers from the lab to the field (i.e., the worksite/workplace) and back again. Translational science has been conceptualized as having multiple phases of research along a continuum, beyond scientific discovery (T0), to efficacy (T1), to effectiveness (T2), to dissemination and implementation (D&I) (T3), to outcomes and effectiveness research in populations (T4). The translational research process applied to occupational exposure to nanomaterials might involve similar phases. This builds on basic and efficacy research (T0 and T1) in the areas of toxicology, epidemiology, industrial hygiene, medicine and engineering. In T2, research and evidence syntheses and guidance and recommendations to protect workers may be developed and assessed for effectiveness. In T3, emphasis is needed on D&I research to explore the multilevel barriers and facilitators to nanotechnology risk control information/research adoption, use, and sustainment in workplaces. D&I research for nanomaterial exposures should focus on assessing sources of information and evidence to be disseminated /implemented in complex and dynamic workplaces, how policy-makers and employers use this information in diverse contexts to protect workers, how stakeholders inform these critical processes, and what barriers impede and facilitate multilevel decision-making for the protection of nanotechnology workers. The T4 phase focuses on how effective efforts to prevent occupational exposure to nanomaterials along the research continuum contribute to large-scale impact in terms of worker safety, health and wellbeing (T4). Stakeholder input and engagement is critical to all stages of the translational research process. This paper will provide: (1) an illustration of the translational research continuum for occupational exposure to nanomaterials; and (2) a discussion of opportunities for applying D&I science to increase the effectiveness, uptake, integration, sustainability, and impact of interventions to protect the health and wellbeing of workers in the nanotechnology field.

Using a longitudinal multi-method approach to document, assess, and understand adaptations in the Veterans Health Administration Advanced Care Coordination program.

Background: Understanding adaptations supports iterative refinement of the implementation process and informs scale out of programs. Systematic documentation of adaptations across the life course of programs is not routinely done, and efficient capture of adaptations in real world studies is not well understood. Methods: We used a multi-method longitudinal approach to systematically document adaptations during pre-implementation, implementation, and sustainment for the Veteran Health Administration (VA) Advanced Care Coordination program. This approach included documenting adaptations through a real-time tracking instrument, process maps, Implementation and Evaluation (I&E) team meeting minutes, and adaptation interviews. Data collection was guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) enhanced framework for reporting adaptations and modifications to evidence-based interventions (FRAME) model. Adaptations were evaluated across 9 categories, and analytic team consensus and member-checking were used to validate the results. Results: A total of 144 individual adaptations were identified across two implementation sites and the four data sources; analytic team consensus and member-checking processes resulted in 50 unique adaptations. Most adaptations took place during the early implementation and mid-implementation phases and were: 1) planned; 2) made to address changes in program delivery; 3) made to extend a component; 4) related to the core component of the intervention concerning notification of the community emergency department visit; 5) initiated by the entire or most of the I&E team; 6) made on the basis of: pragmatic/practical considerations; 7) made with an intent to improve implementation domain (to make the intervention delivered more consistently; to better fit the local practice, patient flow or Electronic Health Record (EHR) and/or for practical reasons); 8) a result of internal influences; 9) perceived to impact the RE-AIM implementation dimension (consistent delivery of quality care or costs). I&E team meeting minutes and process maps captured the highest numbers of unique adaptations (n = 19 and n = 13, respectively). Conclusion: Our longitudinal, multi-method approach provided a feasible way to collect adaptations data through engagement of multiple I&E team members, allowing and a broader understanding of adaptations that took place. Recommendations for future research include pragmatic assessment of the impact of adaptations and meaningful data collection without overburdening the implementing teams and front-line staff.

Use of the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework to guide iterative adaptations: Applications, lessons learned, and future directions.

Introduction: Implementation science frameworks have been used widely for planning and evaluation, but seldom to guide adaptations during program implementation. There is great potential for these frameworks to be used to inform conceptual and data-driven decisions about adaptations. Methods: We summarize recent applications using Iterative RE-AIM to capture and guide adaptations. Iterative RE-AIM can be repeated at multiple time points customized to each project and involves the following activities: identification of key implementation partners; rating importance of and progress on each RE-AIM dimension (reach, effectiveness, adoption, implementation, and maintenance); use of summary data on ratings to identify one or two RE-AIM dimensions for adaptations and implementation strategies; and evaluation of progress and impact of adaptations. We summarize recent and ongoing Iterative RE-AIM applications across multiple care coordination and pain management projects within the Veterans Health Administration, a hypertension control trial in Guatemala, a hospital-based lung ultrasound implementation pilot, and a colorectal cancer screening program in underserved communities. Results: Iterative RE-AIM appears feasible, helpful, and broadly applicable across diverse health care issues, interventions, contexts, and populations. In general, the RE-AIM dimension showing the largest gap between importance and progress has been Reach. The dimensions most frequently selected for improvement have been Reach and Implementation. We discuss commonalities, differences and lessons learned across these various applications of Iterative RE-AIM. Challenges include having objective real time data on which to make decisions, having key implementation staff available for all assessments, and rapidly scoring and providing actionable feedback. We discuss print and online resources and materials to support Iterative RE-AIM. Conclusions: The use of Iterative RE-AIM to guide and support understanding of adaptations has proven feasible across diverse projects and in multiple case studies, but there are still questions about its strengths, limitations, essential components, efficiency, comparative effectiveness, and delivery details. Future directions include investigating the optimal frequency and timing for iterative applications; adding contextual assessments; developing more continuous and rapid data on which to make adaptation decisions; identifying opportunities to enhance health equity; and determining the level of facilitation that is most cost-effective.

Making Implementation Science More Rapid: Use of the RE-AIM Framework for Mid-Course Adaptations Across Five Health Services Research Projects in the Veterans Health Administration.

Introduction: Implementation science frameworks have helped advance translation of research to practice. They have been widely used for planning and post-hoc evaluation, but seldom to inform and guide mid-course adjustments to intervention and implementation strategies. Materials and Methods: This study developed an innovative methodology using the RE-AIM framework and related tools to guide mid-course assessments and adaptations across five diverse health services improvement projects in the Veterans Health Administration (VA). Using a semi-structured guide, project team members were asked to assess the importance of and progress on each RE-AIM dimension (i.e., reach, effectiveness, adoption, implementation, maintenance) at the current phase of their project. Based on these ratings, each team identified one or two RE-AIM dimensions for focused attention. Teams developed proximal goals and implementation strategies to improve progress on their selected dimension(s). A follow-up meeting with each team occurred approximately 6 weeks after the goal setting meeting to evaluate the usefulness of the iterative process. Results were evaluated using both descriptive quantitative analyses and qualitative assessments from interviews and meeting notes. Results: A median of seven team members participated in the two meetings. Qualitative and descriptive data revealed that the process was feasible, understandable and useful to teams in adjusting their interventions and implementation strategies. The RE-AIM dimensions identified as most important were adoption and effectiveness, and the dimension that had the largest gap between importance and rated progress was reach. The dimensions most frequently selected for improvement were reach and adoption. Examples of action plans were summarizing stakeholder interviews for leadership, revising exclusion criteria, and conducting in-service trainings. Follow-up meetings indicated that teams found the process very useful and were able to implement the action plans they set. Discussion: The iterative use of RE-AIM to support adjustments during project implementation proved feasible and useful across diverse projects in the VA setting. Building on this and related examples, future research should replicate these findings and further develop the methodology, as well as explore the optimal frequency and timing for these iterative applications of RE-AIM. More generally, greater focus on more rapid and iterative use of implementation science frameworks is encouraged to facilitate successful translation of research to practice.