The STS case study: an analysis method for longitudinal qualitative research for implementation science.

BACKGROUND: Ethnographic approaches offer a method and a way of thinking about implementation. This manuscript applies a specific case study method to describe the impact of the longitudinal interplay between implementation stakeholders. Growing out of science and technology studies (STS) and drawing on the latent archaeological sensibilities implied by ethnographic methods, the STS case-study is a tool for implementors to use when a piece of material culture is an essential component of an innovation. METHODS: We conducted an ethnographic process evaluation of the clinical implementation of tele-critical care (Tele-CC) services in the Department of Veterans Affairs. We collected fieldnotes and conducted participant observation at virtual and in-person education and planning events (n = 101 h). At Go-Live and 6-months post-implementation, we conducted site visits to the Tele-CC hub and 3 partnered ICUs. We led semi-structured interviews with ICU staff at Go-Live (43 interviews with 65 participants) and with ICU and Tele-CC staff 6-months post-implementation (44 interviews with 67 participants). We used verification strategies, including methodological coherence, appropriate sampling, collecting and analyzing data concurrently, and thinking theoretically, to ensure the reliability and validity of our data collection and analysis process. RESULTS: The STS case-study helped us realize that we must think differently about how a Tele-CC clinician could be noticed moving from communal to intimate space. To understand how perceptions of surveillance impacted staff acceptance, we mapped the materials through which surveillance came to matter in the stories staff told about cameras, buttons, chimes, motors, curtains, and doorbells. CONCLUSIONS: STS case-studies contribute to the literature on longitudinal qualitive research (LQR) in implementation science, including pen portraits and periodic reflections. Anchored by the material, the heterogeneity of an STS case-study generates questions and encourages exploring differences. Begun early enough, the STS case-study method, like periodic reflections, can serve to iteratively inform data collection for researchers and implementors. The next step is to determine systematically how material culture can reveal implementation barriers and direct attention to potential solutions that address tacit, deeply rooted challenges to innovations in practice and technology.

Utilized or Underutilized: A Qualitative Analysis of Building Coherence During Early Implementation of a Tele-Intensive Care Unit.

Background: Generating, reading, or interpreting data is a component of Telemedicine-Intensive Care Unit (Tele-ICU) utilization that has not been explored in the literature. Introduction: Using the idea of "coherence," a construct of Normalization Process Theory, we describe how intensive care unit (ICU) and Tele-ICU staff made sense of their shared work and how they made use of Tele-ICU together. Materials and Methods: We interviewed ICU and Tele-ICU staff involved in the implementation of Tele-ICU during site visits to a Tele-ICU hub and 3 ICUs, at preimplementation (43 interviews with 65 participants) and 6 months postimplementation (44 interviews with 67 participants). Data were analyzed using deductive coding techniques and lexical searches. Results: In the early implementation of Tele-ICU, ICU and Tele-ICU staff lacked consensus about how to share information and consequently how to make use of innovations in data tracking and interpretation offered by the Tele-ICU (e.g., acuity systems). Attempts to collaborate and create opportunities for utilization were supported by quality improvement (QI) initiatives. Discussion: Characterizing Tele-ICU utilization as an element of a QI process limited how ICU staff understood Tele-ICU as an innovation. It also did not promote an understanding of how the Tele-ICU used data and may therefore attenuate the larger promise of Tele-ICU as a potential tool for leveraging big data in critical care. Conclusions: Shared data practices lay the foundation for Tele-ICU program utilization but raise new questions about how the promise of big data can be operationalized for bedside ICU staff.

RE-AIM for rural health innovations: perceptions of (mis) alignment between the RE-AIM framework and evaluation reporting in the Department of Veterans Affairs Enterprise-Wide Initiatives program.

Background: The Department of Veterans Affairs (VA) Office of Rural Health (ORH) supports national VA program offices' efforts to expand health care to rural Veterans through its Enterprise-Wide Initiatives (EWIs) program. In 2017, ORH selected Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM), an implementation science framework, to structure the EWI evaluation and reporting process. As part of its mandate to improve EWI program evaluation, the Center for the Evaluation of Enterprise-Wide Initiatives conducted a qualitative evaluation to better understand EWI team' perceptions of, and barriers and facilitators to, the EWI evaluation process. Methods: We conducted 43 semi-structured interviews with 48 team members (e.g., evaluators, program office leads, and field-based leads) representing 21 EWIs from April-December 2020. Questions focused on participants' experiences using strategies targeting each RE-AIM dimension. Interviews were inductively analyzed in MAXQDA. We also systematically reviewed 51 FY19-FY20 EWI annual reports to identify trends in misapplications of RE-AIM. Results: Participants had differing levels of experience with RE-AIM. While participants understood ORH's rationale for selecting a common framework to structure evaluations, the perceived misalignment between RE-AIM and EWIs' work emerged as an important theme. Concerns centered around 3 sub-themes: (1) (Mis)Alignment with RE-AIM Dimensions, (2) (Mis)Alignment between RE-AIM and the EWI, and (3) (Mis)Alignment with RE-AIM vs. other Theories, Models, or Frameworks. Participants described challenges differentiating between and operationalizing dimensions in unique contexts. Participants also had misconceptions about RE-AIM and its relevance to their work, e.g., that it was meant for established programs and did not capture aspects of initiative planning, adaptations, or sustainability. Less commonly, participants shared alternative models or frameworks to RE-AIM. Despite criticisms, many participants found RE-AIM useful, cited training as important to understanding its application, and identified additional training as a future need. Discussion: The selection of a shared implementation science framework can be beneficial, but also challenging when applied to diverse initiatives or contexts. Our findings suggest that establishing a common understanding, operationalizing framework dimensions for specific programs, and assessing training needs may better equip partners to integrate a shared framework into their evaluations.

Tracking implementation strategies in real-world settings: VA Office of Rural Health enterprise-wide initiative portfolio.

OBJECTIVE: To use a practical approach to examining the use of Expert Recommendations for Implementing Change (ERIC) strategies by Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) dimensions for rural health innovations using annual reports on a diverse array of initiatives. DATA SOURCES AND STUDY SETTING: The Veterans Affairs (VA) Office of Rural Health (ORH) funds initiatives designed to support the implementation and spread of innovations and evidence-based programs and practices to improve the health of rural Veterans. This study draws on the annual evaluation reports submitted for fiscal years 2020-2022 from 30 of these enterprise-wide initiatives (EWIs). STUDY DESIGN: Content analysis was guided by the RE-AIM framework conducted by the Center for the Evaluation of Enterprise-Wide Initiatives (CEEWI), a Quality Enhancement Research Initiative (QUERI)-ORH partnered evaluation initiative. DATA COLLECTION AND EXTRACTION METHODS: CEEWI analysts conducted a content analysis of EWI annual evaluation reports submitted to ORH. Analysis included cataloguing reported implementation strategies by Reach, Adoption, Implementation, and Maintenance (RE-AIM) dimensions (i.e., identifying strategies that were used to support each dimension) and labeling strategies using ERIC taxonomy. Descriptive statistics were conducted to summarize data. PRINCIPAL FINDINGS: A total of 875 implementation strategies were catalogued in 73 reports. Across these strategies, 66 unique ERIC strategies were reported. EWIs applied an average of 12 implementation strategies (range 3-22). The top three ERIC clusters across all 3 years were Develop stakeholder relationships (21%), Use evaluative/iterative strategies (20%), and Train/educate stakeholders (19%). Most strategies were reported within the Implementation dimension. Strategy use among EWIs meeting the rurality benchmark were also compared. CONCLUSIONS: Combining the dimensions from the RE-AIM framework and the ERIC strategies allows for understanding the use of implementation strategies across each RE-AIM dimension. This analysis will support ORH efforts to spread and sustain rural health innovations and evidence-based programs and practices through targeted implementation strategies.

Gaps in communication theory paradigms when conducting implementation science research: qualitative observations from interviews with administrators, implementors, and evaluators of rural health programs.

BACKGROUND: Communication is considered an inherent element of nearly every implementation strategy. Often it is seen as a means for imparting new information between stakeholders, representing a Transaction orientation to communication. From a Process orientation, communication is more than information-exchange and is acknowledged as being shaped by (and shaping) the individuals involved and their relationships with one another. As the field of Implementation Science (IS) works to strengthen theoretical integration, we encourage an interdisciplinary approach that engages communication theory to develop richer understanding of strategies and determinants of practice. METHODS: We interviewed 28 evaluators, 12 implementors, and 12 administrators from 21 Enterprise-Wide Initiatives funded by the Department of Veteran Affairs Office of Rural Health. Semi-structured interviews focused on experiences with implementation and evaluation strategies. We analyzed the interviews using thematic analysis identifying a range of IS constructs. Then we deductively classified those segments based on a Transaction or Process orientation to communication. RESULTS: We organized findings using the two IS constructs most commonly discussed in interviews: Collaboration and Leadership Buy-in. The majority of segments coded as Collaboration (n = 34, 74%) and Leadership Buy-in (n = 31, 70%) discussed communication from a Transaction orientation and referred to communication as synonymous with information exchange, which emphasizes the task over the relationships between the individuals performing the tasks. Conversely, when participants discussed Collaboration and Leadership Buy-in from a Process orientation, they acknowledged both constructs as the result of long-term efforts to develop positive relationships based on trust and respect, and emphasized the time costliness of such strategies. Our findings demonstrate that participants who discussed communication from a Process orientation recognized the nuance and complexity of interpersonal interactions, particularly in the context of IS. CONCLUSIONS: Efficient, reliable information exchange is a critical but often overemphasized element of implementation. Practitioners and researchers must recognize and incorporate the larger role of communication in IS. Two suggestions for engaging a Process orientation to communication are to: (a) use interview probes to learn how communication is enacted, and (b) use process-oriented communication theories to develop interventions and evaluation tools.

Ankle- and Toe-Brachial Index for Peripheral Artery Disease Identification: Unlocking Clinical Data Through Novel Methods.

BACKGROUND: Despite its high prevalence and clinical impact, research on peripheral artery disease (PAD) remains limited due to poor accuracy of billing codes. Ankle-brachial index (ABI) and toe-brachial index can be used to identify PAD patients with high accuracy within electronic health records. METHODS: We developed a novel natural language processing (NLP) algorithm for extracting ABI and toe-brachial index values and laterality (right or left) from ABI reports. A random sample of 800 reports from 94 Veterans Affairs facilities during 2015 to 2017 was selected and annotated by clinical experts. We trained the NLP system using random forest models and optimized it through sequential iterations of 10-fold cross-validation and error analysis on 600 test reports and evaluated its final performance on a separate set of 200 reports. We also assessed the accuracy of NLP-extracted ABI and toe-brachial index values for identifying patients with PAD in a separate cohort undergoing ABI testing. RESULTS: The NLP system had an overall precision (positive predictive value) of 0.85, recall (sensitivity) of 0.93, and F1 measure (accuracy) of 0.89 to correctly identify ABI/toe-brachial index values and laterality. Among 261 patients with ABI testing (49% PAD), the NLP system achieved a positive predictive value of 92.3%, sensitivity of 83.1%, and specificity of 93.1% to identify PAD when compared with a structured chart review. The above findings were consistent in a range of sensitivity analysis. CONCLUSIONS: We successfully developed and validated an NLP system for identifying patients with PAD within the Veterans Affairs electronic health record. Our findings have broad implications for PAD research and quality improvement.

Environmental service workers as potential designers of infection control policy in long-term care settings.

BACKGROUND: Long-term care facility residents are at higher risk of methicillin-resistant Staphylococcus aureus infection and colonization than the general population. In 2009, the Department of Veterans Affairs (VA) implemented the "methicillin-resistant S. aureus prevention initiative" in long-term care facilities (ie, Community Living Centers or "CLCs"). METHODS: Over 4 months, 40 semistructured interviews were conducted with staff in medicine, nursing, and environmental services at 5 geographically dispersed CLCs. Interviews addressed knowledge, attitudes, and beliefs concerning infection prevention and resident-centered care. A modified constant comparative approach was used for data analysis. RESULTS: In CLCs, staff work to prevent and control infections in spaces where residents live. Nurses and Environmental Service Workers daily balance infection prevention conventions with the CLC setting. Infection control team members, who are accustomed to working in acute care settings, struggle to reconcile the CLC context with infection prevention. DISCUSSION: The focus on the resident's room as the locus of care, and thus the main target of infection control, misses opportunities for addressing infection prevention in the spaces beyond the residents' rooms. CONCLUSIONS: Environmental Service Workers' daily work inside the rooms and within the wider facility produces a unique perspective that might help in the design of workable infection control policies in CLCs.