High Impact for Whom? A Qualitative Analysis of Organization Concerns About the Transition to High-Impact Prevention Policy.

HIV represents a significant health burden in the United States. In 2012, the Centers for Disease Control and Prevention (CDC) stopped recommending many once-promoted interventions as part of a shift from one HIV intervention policy, Diffusion of Effective Behavioral Interventions (DEBI), to another, High Impact Prevention (HIP). Twenty-nine staff members from 10 organizations were interviewed to explore how organizations reacted to this shift. Three major themes emerged: (1) Personal experience, community assessment, and epidemiological evidence influenced organizations' perceptions of efficacy and preference for earlier interventions. (2) Organizations were concerned that HIP interventions were not a good fit for their priority populations. (3) Organizations were frustrated with the top-down approach by the CDC prioritizing HIP interventions over earlier interventions. These results indicate that organizations continue to see value in and provide DEBI interventions. In addition, a more participatory process incorporating qualitative evidence and organizations' experiences may be necessary to achieve widespread de-implementation of DEBI interventions.

Model for regional collaboration: Successful strategy to implement a pediatric early warning system in 36 pediatric oncology centers in Latin America.

BACKGROUND: Pediatric early warning systems (PEWS) aid in the early identification of deterioration in hospitalized children with cancer; however, they are under-used in resource-limited settings. The authors use the knowledge-to-action framework to describe the implementation strategy for Proyecto Escala de Valoracion de Alerta Temprana (EVAT), a multicenter quality-improvement collaborative, to scale-up PEWS in pediatric oncology centers in Latin America. METHODS: Proyecto EVAT mentored participating centers through an adaptable implementation strategy to: (1) monitor clinical deterioration in children with cancer, (2) contextually adapt PEWS, (3) assess barriers to using PEWS, (4) pilot and implement PEWS, (5) monitor the use of PEWS, (6) evaluate outcomes, and (7) sustain PEWS. The implementation outcomes assessed included the quality of PEWS use, the time required for implementation, and global program impact. RESULTS: From April 2017 to October 2021, 36 diverse Proyecto EVAT hospitals from 13 countries in Latin America collectively managing more than 4100 annual new pediatric cancer diagnoses successfully implemented PEWS. The time to complete all program phases varied among centers, averaging 7 months (range, 3-13 months) from PEWS pilot to implementation completion. All centers ultimately implemented PEWS and maintained high-quality PEWS use for up to 18 months after implementation. Across the 36 centers, more than 11,100 clinicians were trained in PEWS, and more than 41,000 pediatric hospital admissions had PEWS used in their care. CONCLUSIONS: Evidence-based interventions like PEWS can be successfully scaled-up regionally basis using a systematic approach that includes a collaborative network, an adaptable implementation strategy, and regional mentorship. Lessons learned can guide future programs to promote the widespread adoption of effective interventions and reduce global disparities in childhood cancer outcomes. LAY SUMMARY: Pediatric early warning systems (PEWS) are clinical tools used to identify deterioration in hospitalized children with cancer; however, implementation challenges limit their use in resource-limited settings. Proyecto EVAT is a multicenter quality-improvement collaborative to implement PEWS in 36 pediatric oncology centers in Latin America. This is the first multicenter, multinational study reporting a successful implementation strategy (Proyecto EVAT) to regionally scale-up PEWS. The lessons learned from Proyecto EVAT can inform future programs to promote the adoption of clinical interventions to globally improve childhood cancer outcomes.

The Use of the Consolidated Framework for Implementation Research to Understand Facilitators and Barriers to Sexually Transmitted Infection Screening in Primary Care.

BACKGROUND: Adolescents bear a disproportionate burden of sexually transmitted infections (STIs) and the sequelae of delayed treatment, yet STI screening is infrequently performed in pediatric primary care clinics with many of those at-risk not administered testing. This study aims to understand contextual factors influencing STI screening and testing among adolescents in pediatric primary care. METHODS: We used the Consolidated Framework for Implementation Research (CFIR) as part of a stepwise approach to facilitate a deep understanding the pediatric primary care environment. We conducted semistructured interviews of physicians, nurses, and patient-parent dyads from 4 pediatric primary care practices in the St. Louis metropolitan area about STI screening practices and common concerns regarding STI screening. Qualitative analysis was conducted using a categorical coding technique informed by the CFIR followed by a thematic coding technique. RESULTS: We interviewed 23 physicians/nurses and 12 patient-parent dyads. Individual-level barriers to STI screening and testing included wide variability in clinicians' practice patterns and their perception of STI risk in the patient population. Structural barriers included a lack of capacity to perform testing in clinic and time constraints during patient visits. Confidentiality issues also created significant barriers to screening and testing on both individual and structural levels. Adopting confidential methods for testing and educating providers on patients' recommendations for STI testing were discussed as ways to potentially improve STI care in pediatric patients. CONCLUSIONS: Our use of the CFIR facilitated a systematic approach to identify gaps in STI care for adolescents and identified opportunities to close those gaps. An integrated, systematic approach that enhances patient confidentiality and improves clinicians' knowledge could address gaps in STI care in pediatric primary care settings.

Dissemination and implementation science and antimicrobial stewardship in solid organ transplantation: A perspective.

BACKGROUND: Recipients of solid organ transplants (SOTs) have unique risks for infections, but providers are often hesitant to apply the principles of antimicrobial stewardship to this patient population due to perceived excess risk. The methods of implementation science may move the field forward to simultaneously improve patient outcomes and patient safety. METHODS: Perspective piece on implementation science in SOT patients. RESULTS: Herein, we provide explanation of implementation science as it relates to SOT patients. In addition, we provide examples of how implementation science can be applied to antimicrobial stewardship in SOT patients. CONCLUSION: Implementation science may offer insights and solutions to the challenges of implementing evidence-based antimicrobial stewardship interventions in patients with SOT, including uptake of new practices and the de-implementation of outdated or low-value practices.

Using Implementation Science to Guide the Process of Adapting a Patient Engagement Intervention for Inpatient Spinal Cord Injury/Disorder Rehabilitation.

OBJECTIVES: This study aimed to describe the process of adapting an evidence-based patient engagement intervention, enhanced medical rehabilitation (E-MR), for inpatient spinal cord injury/disease (SCI/D) rehabilitation using an implementation science framework. DESIGN: We applied the collaborative intervention planning framework and included a community advisory board (CAB) in an intervention mapping process. SETTING: A rehabilitation hospital. PARTICIPANTS: Stakeholders from inpatient SCI/D rehabilitation (N=7) serving as a CAB and working with the research team (N=7) to co-adapt E-MR. INTERVENTIONS: E-MR. MAIN OUTCOME MEASURES: Logic model and matrices of change used in CAB meetings to identify areas of intervention adaptation. RESULTS: The CAB and research team implemented adaptations to E-MR, including (1) identifying factors influencing patient engagement in SCI/D rehabilitation (eg, therapist training); (2) revising intervention materials to meet SCI/D rehabilitation needs (eg, modified personal goals interview and therapy trackers to match SCI needs); (3) incorporating E-MR into the rehabilitation hospital's operations (eg, research team coordinated with CAB to store therapy trackers in the hospital system); and (4) retaining fidelity to the original intervention while best meeting the needs of SCI/D rehabilitation (eg, maintained core E-MR principles while adapting). CONCLUSIONS: This study demonstrated that structured processes guided by an implementation science framework can help researchers and clinicians identify adaptation targets and modify the E-MR program for inpatient SCI/D rehabilitation.

Putting the Dissemination and Implementation in Infectious Diseases.

Dissemination and implementation science seeks generalizable knowledge about closing the gap between clinical discovery and actual use in routine practice and public health. The field of infectious diseases enjoys an abundance of highly efficacious interventions (eg, antimicrobial agents, human immunodeficiency virus treatment) which are not adequately used in routine care, thereby missing critical opportunities to improve population health. In this article, we summarize salient features of dissemination and implementation science, reviewing definitions and methodologies for infectious diseases clinicians and researchers. We give examples of the limited use of dissemination and implementation science in infectious diseases thus far, suggest opportunities for application, and provide resources for interested readers to use and apply to their own research and practice.

Better Service by Doing Less: Introducing De-implementation Research in HIV.

The course of HIV research has led to a multitude of interventions to prevent and treat HIV. With the arrival of more effective interventions comes the need to end, or de-implement, less effective interventions. PURPOSE OF REVIEW: To describe the state of de-implementation research in HIV and provide a rationale for expanded research in this area. RECENT FINDINGS: Existing studies have identified a set of HIV-specific interventions appropriate for de-implementing and described the persistence of interventions that should be ended. However, to our knowledge, strategies to successfully promote appropriate de-implementation of HIV-specific interventions have not been examined. De-implementing interventions that are no longer needed is an opportunity to improve the quality and effectiveness of HIV services. Opportunities to expand this field of research abound.

Modelling the impact of menthol sales restrictions and retailer density reduction policies: insights from tobacco town Minnesota.

INTRODUCTION: Tobacco control policies focused on the retail environment have the potential to reduce tobacco use and tobacco-related health disparities through increasing direct and indirect costs. Recently, national and subnational governments have begun to restrict the sale of menthol products and reduce tobacco retailer density. METHODS: We developed an agent-based model to project the impact of menthol cigarette sales restrictions and retailer density reduction policies for six types of communities and three priority populations. During each simulated day, agents smoke cigarettes, travel in the community and make purchase decisions-whether, where and which product type to purchase-based on a combination of their own properties and the current retail environment. RESULTS: Of the policies tested, restricting all cigarette sales or menthol cigarette sales to tobacco specialty shops may have the largest effect on the total (direct and indirect) costs of purchasing cigarettes. Coupling one of these policies with one that establishes a minimum distance between tobacco retailers may enhance the impact. Combining these policies could also make the costs of acquiring cigarettes more equal across communities and populations. DISCUSSION: Our simulations revealed the importance of context, for example, lower income communities in urban areas begin with higher retailer density and may need stronger policies to show impact, as well as the need to focus on differential effects for priority populations, for example, combinations of policies may equalise the average distance travelled to purchase. Adapting and combining policies could enhance the sustainability of policy effects and reduce tobacco use.

Letting Go: Conceptualizing Intervention De-implementation in Public Health and Social Service Settings.

The discontinuation of interventions that should be stopped, or de-implementation, has emerged as a novel line of inquiry within dissemination and implementation science. As this area grows in human services research, like public health and social work, theory is needed to help guide scientific endeavors. Given the infancy of de-implementation, this conceptual narrative provides a definition and criteria for determining if an intervention should be de-implemented. We identify three criteria for identifying interventions appropriate for de-implementation: (a) interventions that are not effective or harmful, (b) interventions that are not the most effective or efficient to provide, and (c) interventions that are no longer necessary. Detailed, well-documented examples illustrate each of the criteria. We describe de-implementation frameworks, but also demonstrate how other existing implementation frameworks might be applied to de-implementation research as a supplement. Finally, we conclude with a discussion of de-implementation in the context of other stages of implementation, like sustainability and adoption; next steps for de-implementation research, especially identifying interventions appropriate for de-implementation in a systematic manner; and highlight special ethical considerations to advance the field of de-implementation research.

Impact of Human Resources on Implementing an Evidence-Based HIV Prevention Intervention.

Evidence-based interventions (EBIs) often require competent staff, or human resources (HR), for implementation. The empirical evidence characterizing the influence of HR fluctuations on EBI delivery is limited and conflicting. Using the Interactive Systems Framework, we explored staff fluctuation and the subsequent influence on RESPECT, an HIV prevention EBI. We conducted interviews with staff in two waves (n = 53, Wave I; n = 37, Wave II) in a national sample of organizations delivering RESPECT (N = 29). We analyzed interviews qualitatively to describe changes among RESPECT staff and explore the subsequent influences on RESPECT implementation. Organizations reported downsizing, turnover, and expansion of staff positions. Staff changes had multiple influences on RESPECT implementation including clients reached, fidelity to specific RESPECT protocols, and overall sustainability of RESPECT over time. HR fluctuations are common, and our analyses provide an initial characterization of the relationship between HR fluctuation and EBI implementation. Given the prominent influence of HR on EBI implementation, the Interactive Systems Framework is a useful guiding tool for future examinations.

ADAPTATIONS TO AN HIV COUNSELING AND TESTING INTERVENTION FROM A COUNSELOR PERSPECTIVE.

Understanding the balance between fidelity and adaptation for evidence-based interventions has the potential to improve their translation from research to practice. The Translation into Practice study explores variation in program implementation within organizations utilizing the RESPECT program, an HIV counseling and testing intervention. Counselors (N = 70) were interviewed using a semistructured interview guide to examine both the influential factors on, and the subsequent adaptations to, RESPECT. Almost all counselors reported making adaptations (N = 69). Adaptations were made both to key characteristics (mean [M] = 2.24, standard deviation [SD] = 1.3) and to core components (M = 0.5, SD = 0.8). Counselors identified the environmental context and factors within the counseling context as common influences leading to adaptation. These findings suggest adaptations were a routine part of program usage. To improve implementation of the RESPECT program, further research is needed to assess the degree to which adaptations can be made to better meet the needs of agencies and clients without compromising fidelity.

Establishing evidence criteria for implementation strategies in the US: a Delphi study for HIV services.

BACKGROUND: There are no criteria specifically for evaluating the quality of implementation research and recommending implementation strategies likely to have impact to practitioners. We describe the development and application of the Best Practices Tool, a set of criteria to evaluate the evidence supporting HIV-specific implementation strategies. METHODS: We developed the Best Practices Tool from 2022-2023 in three phases. (1) We developed a draft tool and criteria based on a literature review and key informant interviews. We purposively selected and recruited by email interview participants representing a mix of expertise in HIV service delivery, quality improvement, and implementation science. (2) The tool was then informed and revised through two e-Delphi rounds using a survey delivered online through Qualtrics. The first and second round Delphi surveys consisted of 71 and 52 open and close-ended questions, respectively, asking participants to evaluate, confirm, and make suggestions on different aspects of the rubric. After each survey round, data were analyzed and synthesized as appropriate; and the tool and criteria were revised. (3) We then applied the tool to a set of research studies assessing implementation strategies designed to promote the adoption and uptake of evidence-based HIV interventions to assess reliable application of the tool and criteria. RESULTS: Our initial literature review yielded existing tools for evaluating intervention-level evidence. For a strategy-level tool, additions emerged from interviews, for example, a need to consider the context and specification of strategies. Revisions were made after both Delphi rounds resulting in the confirmation of five evaluation domains - research design, implementation outcomes, limitations and rigor, strategy specification, and equity - and four evidence levels - best, promising, more evidence needed, and harmful. For most domains, criteria were specified at each evidence level. After an initial pilot round to develop an application process and provide training, we achieved 98% reliability when applying the criteria to 18 implementation strategies. CONCLUSIONS: We developed a tool to evaluate the evidence supporting implementation strategies for HIV services. Although specific to HIV in the US, this tool is adaptable for evaluating strategies in other health areas.

Dissemination and Implementation Science in Cardiothoracic Surgery: A Review and Case Study.

Dissemination and implementation (D&I) science is the practice of taking evidence-based interventions and sustainably incorporating them into routine clinical practice. As a relatively young field, D&I techniques are underutilized in cardiothoracic surgery. This review offers an overview of D&I science from the context of the cardiothoracic surgeon. First, we provide a general introduction to D&I science and basic terminology that is used in the field. Second, to illustrate D&I techniques in a real-world example, we discuss a case study for implementing lung protective management strategies for lung donor optimization nationally. Finally, we discuss challenges to successful implementation that are unique to cardiothoracic surgery and give several examples of evidence-based interventions that have been poorly implemented into surgical practice. We also provide examples of successful D&I interventions-including deimplementation strategies-from other surgical subspecialties. We hope that this review offers additional tools for cardiothoracic surgeons to explore when introducing evidence-based interventions into routine practice.

Market viability: a neglected concept in implementation science.

This debate paper asserts that implementation science needs to incorporate a key concept from entrepreneurship-market demand-and demonstrates how assessing an innovation's potential market viability might advance the pace and success of innovation adoption and sustainment. We describe key concepts, language distinctions, and questions that entrepreneurs pose to implementation scientists-many of which implementation scientists appear ill-equipped to answer. The paper concludes with recommendations about how concepts from entrepreneurship, notably market viability assessment, can enhance the translation of research discoveries into real-world adoption, sustained use, and population health benefits. The paper further proposes activities that can advance implementation science's capacity to draw from the field of entrepreneurship, along with the data foundations required to assess and cultivate market demand.

Persistence of inefficient HIV prevention interventions: a mixed-method analysis of the reasons why.

Growing evidence suggests that public health organizations continue to provide inefficient interventions even when better intervention options may be available. Factors informing an organization's decision to continue providing inefficient interventions are unclear. We present an analysis of HIV service organizations to understand factors influencing organizations to continue or end interventions. Between 2017 and 2019, HIV service organizations were recruited from the Center for Disease Control and Prevention's (CDC) website gettested.org, in the 20 metropolitan areas with the highest HIV incidence. Organizations were eligible to participate if they had provided at least one of 37 HIV prevention interventions identified as inefficient by the CDC. 877 organizations were recruited, with a response rate of 66%, (n = 578). Thirty-eight percent (n = 213) of organizations met the eligibility criteria, and 188 organizations completed the survey asking about reasons for continuing or ending interventions. Funding status (41%, n = 79) and client demand for interventions (60%, n = 116) were reported as the primary driving factors why organizations continued ineffective interventions. Scientific evidence was a rarely reported reason for ending an inefficient intervention (12%, n = 23). Qualitative responses indicated interventions were continued if clients demanded interventions they found useful or if staff perceived interventions as improving client behavior and health outcomes. Conversely, interventions were ended if client demand or retention was low, not relevant to the target population or funding ended. The decision to continue or end inefficient interventions is influenced by a number of factors-most often by funding and client interest but not scientific evidence.

Partnered innovation to implement timely and personalized care: A case study.

BACKGROUND: Understanding how to translate research discoveries into solutions for healthcare improvement is a priority of NIH-funded Clinical and Translational Science Awards (CTSA). This study, supported by one CTSA, aims to capture one process of shaping and implementing innovations to advance the timeliness and patient-centeredness of cardiovascular care. Specifically, we sought to understand a partnership between a private digital health startup company, a university innovation lab, and an academic health system's cardiology program pursuing this goal. FINDINGS: The collaboration proceeded through clear phases to address the questions and challenges: problem definition, exploration and formalization of the partnership, innovation co-creation and pilot test, and scale-up planning. Phases were punctuated by key decisions, such as forming the partnership, negotiating terms of the partnership, iterating form and features of the innovation, and exploring sufficiency of its value-add for scale-up and sustainment. Key implementation concepts were apparent, including implementation strategies (e.g., champions and iterative trialing) and the implementation outcomes of acceptability, sustainment, and scale-up. Participants identified potential risks of collaboration, reflected on their co-creation process, and the value of engaging stakeholders in innovation design. Findings may inform subsequent collaborations between innovators and translational researchers. METHODS: We conducted a case study to understand the partnership; characterize the questions they pursued, their decision points, information and data sources; and identify the challenges and risks. Data were collected through a series of four focus groups with members of each partnering organization. A transdisciplinary research team iteratively worked to condense and synthesize data from audio recorded transcripts into a case narrative.

A scoping review of de-implementation frameworks and models.

BACKGROUND: Reduction or elimination of inappropriate, ineffective, or potentially harmful healthcare services and public health programs can help to ensure limited resources are used effectively. Frameworks and models (FM) are valuable tools in conceptualizing and guiding the study of de-implementation. This scoping review sought to identify and characterize FM that can be used to study de-implementation as a phenomenon and identify gaps in the literature to inform future model development and application for research. METHODS: We searched nine databases and eleven journals from a broad array of disciplines (e.g., healthcare, public health, public policy) for de-implementation studies published between 1990 and June 2020. Two raters independently screened titles and abstracts, and then a pair of raters screened all full text records. We extracted information related to setting, discipline, study design, methodology, and FM characteristics from included studies. RESULTS: The final search yielded 1860 records, from which we screened 126 full text records. We extracted data from 27 articles containing 27 unique FM. Most FM (n = 21) were applicable to two or more levels of the Socio-Ecological Framework, and most commonly assessed constructs were at the organization level (n = 18). Most FM (n = 18) depicted a linear relationship between constructs, few depicted a more complex structure, such as a nested or cyclical relationship. Thirteen studies applied FM in empirical investigations of de-implementation, while 14 articles were commentary or review papers that included FM. CONCLUSION: De-implementation is a process studied in a broad array of disciplines, yet implementation science has thus far been limited in the integration of learnings from other fields. This review offers an overview of visual representations of FM that implementation researchers and practitioners can use to inform their work. Additional work is needed to test and refine existing FM and to determine the extent to which FM developed in one setting or for a particular topic can be applied to other contexts. Given the extensive availability of FM in implementation science, we suggest researchers build from existing FM rather than recreating novel FM. REGISTRATION: Not registered.

Use of effective training and quality assurance strategies is associated with high-fidelity EBI implementation in practice settings: a case analysis.

High-quality implementation of evidence-based interventions is important for program effectiveness and is influenced by training and quality assurance (QA). However, gaps in the literature contribute to a lack of guidance on training and supervision in practice settings, particularly when significant adaptations in programs occur. We examine training and QA in relationship to program fidelity among organizations delivering a widely disseminated HIV counseling and testing EBI in which significant adaptations occurred due to new testing technology. Using a maximum variation case study approach, we examined training and QA in organizations delivering the program with high- and low-fidelity (agencies: 3 = high; 3 = low). We identified themes that distinguished high- and low-fidelity agencies. For example, high-fidelity agencies more often employed a team approach to training; demonstrated use of effective QA strategies; leveraged training and QA to identify and adjust for fit problems, including challenges related to adaptations; and understood the distinctions between RESPECT and other testing programs. The associations between QA and fidelity were strong and straightforward, whereas the relationship between training and fidelity was more complex. Public health needs high-quality training and QA approaches that can address program fit and program adaptations. The study findings reinforced the value of using effective QA strategies. Future work should address methods of increasing program fit through training and QA, identify a set of QA strategies that maximize program fidelity and is feasible to implement, and identify low-cost supplemental training options.

A cluster randomized stepped-wedge trial to de-implement unnecessary post-operative antibiotics in children: the optimizing perioperative antibiotic in children (OPerAtiC) trial.

BACKGROUND: Antibiotic-resistant infections have become a public health crisis that is driven by the inappropriate use of antibiotics. In the USA, antibiotic stewardship programs (ASP) have been established and are required by regulatory agencies to help combat the problem of antibiotic resistance. Post-operative antibiotic use in surgical cases deemed low-risk for infection is an area with significant overuse of antibiotics in children. Consensus among leading public health organizations has led to guidelines eliminating post-operative antibiotics in low-risk surgeries. However, the best strategies to de-implement these inappropriate antibiotics in this setting are unknown. METHODS/DESIGN: A 3-year stepped wedge cluster randomized trial will be conducted at nine US Children's Hospitals to assess the impact of two de-implementation strategies, order set change and facilitation training, on inappropriate post-operative antibiotic prescribing in low risk (i.e., clean and clean-contaminated) surgical cases. The facilitation training will amplify order set changes and will involve a 2-day workshop with antibiotic stewardship teams. This training will be led by an implementation scientist expert (VRM) and a pediatric infectious diseases physician with antibiotic stewardship expertise (JGN). The primary clinical outcome will be the percentage of surgical cases receiving unnecessary post-operative antibiotics. Secondary clinical outcomes will include the rate of surgical site infections and the rate of Clostridioides difficile infections, a common negative consequence of antibiotic use. Monthly semi-structured interviews at each hospital will assess the implementation process of the two strategies. The primary implementation outcome is penetration, which will be defined as the number of order sets changed or developed by each hospital during the study. Additional implementation outcomes will include the ASP team members' assessment of the acceptability, appropriateness, and feasibility of each strategy while they are being implemented. DISCUSSION: This study will provide important information on the impact of two potential strategies to de-implement unnecessary post-operative antibiotic use in children while assessing important clinical outcomes. As more unnecessary medical practices are identified, de-implementation strategies, including facilitation, need to be rigorously evaluated. Along with this study, other rigorously designed studies evaluating additional strategies are needed to further advance the burgeoning field of de-implementation. TRIAL REGISTRATION: NCT04366440. Registered April 28, 2020, https://clinicaltrials.gov/ct2/show/NCT04366440 .

The de-implementation and persistence of low-value HIV prevention interventions in the United States: a cross-sectional study.

BACKGROUND: As more effective or efficient interventions emerge out of scientific advancement to address a particular public health issue, it may be appropriate to de-implement low-value interventions, or interventions that are less effective or efficient. Furthermore, factors that contribute to appropriate de-implementation are not well identified. We examined the extent to which low-value interventions were de-implemented among public health organizations providing HIV prevention services, as well as explored socio-economic, organizational, and intervention characteristics associated with de-implementation. METHODS: We conducted an online cross-sectional survey from the fall of 2017 to the spring of 2019 with organizations (N = 188) providing HIV prevention services in the USA. Organizations were recruited from the Center for Disease Control and Prevention's (CDC) website gettested.org from 20 metropolitan statistical areas with the highest HIV incidence. An organization was eligible to participate if the organization had provided at least one of the HIV prevention interventions identified as inefficient by the CDC in the last ten years, and one administrator familiar with HIV prevention programming at the organization was recruited to respond. Complete responses were analyzed to describe intervention de-implementation and identify organizational and intervention characteristics associated with de-implementation using logistic regression. RESULTS: Organizations reported 359 instances of implementing low-value interventions. Out of the low-value interventions implemented, approximately 57% were group, 34% were individual, and 5% were community interventions. Of interventions implemented, 46% had been de-implemented. Although we examined a number of intervention and organizational factors thought to be associated with de-implementation, the only factor statistically associated with de-implementation was organization size, with larger organizations-those with 50+ FTEs-being 3.1 times more likely to de-implement than smaller organizations (95% CI 1.3-7.5). CONCLUSIONS: While low-value interventions are frequently de-implemented among HIV prevention organizations, many persisted representing substantial inefficiency in HIV prevention service delivery. Further exploration is needed to understand why organizations may opt to continue low-value interventions and the factors that lead to de-implementation.