Maintaining Program Fidelity in a Changing World: National Implementation of a School-Based HIV Prevention Program.

Large-scale, evidence-based interventions face challenges to program fidelity of implementation. We developed implementation strategies to support teachers implementing an evidence-based HIV prevention program in schools, Focus on Youth in The Caribbean (FOYC) and Caribbean Informed Parents and Children Together (CImPACT) in The Bahamas. We examined the effects of these implementation strategies on teachers' implementation in the subsequent year after the initial implementation during the COVID-19 pandemic. Data were collected from 79 Grade 6 teachers in 24 government elementary schools. Teachers completed training workshops and a pre-implementation questionnaire to record their characteristics and perceptions that might affect their program fidelity. School coordinators and peer mentors provided teachers with monitoring, feedback, and mentoring. In Year 1, teachers on average taught 79.3% of the sessions and 80.8% of core activities; teachers in Year 2 covered 84.2% of sessions and 72.9% of the core activities. Teachers with "good" or "excellent" school coordinators in the second year taught significantly more sessions on average (7.8 vs. 7.0, t = 2.04, P < 0.05) and more core activities (26.3 vs. 23.0, t = 2.41, P < 0.05) than teachers with "satisfactory" coordinators. Teachers who had a "good" or "satisfactory" mentor taught more sessions than teachers who did not have a mentor (7.9 vs. 7.3; t = 2.22; P = 0.03). Two-level mixed-effects model analysis indicated that teachers' program fidelity in Year 1, confidence in the execution of core activities, and school coordinators' performance were significantly associated with Year 2 implementation dose. Implementation of FOYC + CImPACT was significantly associated with improved student outcomes. Teachers maintained high fidelity to a comprehensive HIV prevention program over 2 years during the COVID-19 pandemic. Future program implementers should consider additional implementation support to improve the implementation of school-based programs.

Implementation of a Treatment Program for Individuals Imprisoned for Sex Offenses in Uruguay: Achievements, Problems and Challenges.

Treatment for individuals convicted of sex offenses has substantially improved in developed countries in recent decades, providing practitioners with an extensive literature to guide the implementation of effective programs to reduce sexual reoffending. Nevertheless, sexual offending rehabilitation is still in its infancy in Latin American countries such as Uruguay, so little is known about the transference and implementation of evidence-based programs. The current study examines the strengths, barriers, and challenges of implementing a sex offenses treatment program in Uruguay. The findings suggest some achievements of the program, but also several problems with implementation. Some problems are universal among different countries (e.g., scarce resources and facilities, insufficiently trained staff, and unexpected changes in the organization), but others were particularly relevant in the Uruguayan context (e.g., government policy alien to a rehabilitation approach, lack of appropriate prison facilities, lack of training for therapists from a cognitive-behavioral perspective). All these difficulties must be anticipated and solved for successful generalizability of rehabilitation programs to different correctional systems.

Does knowing translate to doing? Practitioners' experiences of Circle of Security-Parenting (COS-P) training and implementation.

This study sought to understand how practitioners perceive and experience the Circle of Security-Parenting (COS-P) training, and further, how they integrate and implement it into practice, and how these experiences influence their use and understanding of the program and its underlying model. A thematic analysis of semistructured interviews at two time points (shortly after training and 3-6 months after training) was used to explore 12 practitioners' experiences of COS-P training and subsequent implementation. Three main themes were identified; clinical salience, personal salience, and partial use of the program. The findings reflected participants' common perception that the model is relevant and generalizable to a wide variety of contexts. It also highlighted potential barriers to implementation, particularly practitioners' experiences using only components of the COS-P program in isolation. The results suggest practitioners' assumptions about client complexities, vulnerabilities, and/or incapacities, can prompt practitioners to withhold the use of COS-P (in part or whole), thereby potentially neglecting key components required for client change. The only participants who implemented the COS-P training in full had additional training in Circle of Security.

Este estudio buscaba comprender cómo los profesionales de la práctica perciben y experimentan el entrenamiento Círculo de Seguridad-Crianza (COS-P), y aún más, cómo ellos lo integran e implementan en la práctica y cómo estas experiencias influyen en su uso y comprensión del programa y el modelo que el mismo enfatiza. Se usó un análisis temático de entrevistas semiestructuradas en dos momentos temporales (poco después del entrenamiento y 3-6 meses después del entrenamiento) para explorar doce experiencias de profesionales de la práctica en cuanto al entrenamiento de COS-P y la subsecuente implementación. Se identificaron tres temas principales; relevancia clínica, relevancia personal y el uso parcial del programa. Los resultados reflejan la percepción común de los participantes de que el modelo es relevante y se puede generalizar a una variedad amplia de contextos. También destacó barreras potenciales para la implementación, particularmente las experiencias de los profesionales de la práctica usando sólo componentes aislados del COS-P. Los resultados sugieren que las suposiciones de los profesionales de la práctica acerca de las complejidades, vulnerabilidades y/o incapacidades del cliente pueden dar pie para que tales profesionales no revelen el uso de COS-P (en parte o totalmente), de modo que potencialmente desatiendan componentes claves requeridos para el cambio en el cliente. Los únicos participantes que implementaron el entrenamiento de COS-P en su totalidad recibieron entrenamiento adicional en cuanto a Círculo de Seguridad.

Cette étude s'est donné pour but de comprendre comment les praticiens perçoivent et font l'expérience de la formation du Cercle de Sécurité-Parentage (COS-P en anglais), et ensuite comment ils l'intègrent et la mettent en place dans leur pratique, ainsi que la manière dont ces expériences influencent leur utilisation et compréhension du programme, tout comme son modèle sous-jacent. Une analyse thématique d'entretiens semi-structurés en deux temps (pas longtemps du Cercle de Sécurité-Parentage (COS-P) après la formation et 3 à 6 mois après la formation) a été utilisée afin d'explorer à la fois les expériences du COS-P faites par douze praticiens et leur mise en place subséquente. Trois thèmes principaux ont été identifiés, la saillance clinique, la saillance personnelle et l'utilisation partielle du programme. Les résultats ont reflété la perception commune des participants selon laquelle le modèle est pertinent et généralisable à une grande variété de contextes. L'étude a aussi mis en lumière les barrières potentielles pour la mise en place, en particulier les expériences des praticiens utilisant uniquement des composantes du programme COS-P en isolation. Les résultats suggèrent que les suppositions des praticiens sur les complexités des clients, leurs vulnérabilités et / ou leurs incapacités, peuvent pousser les praticiens à ne pas utiliser le COS-P (en partie ou totalement), négligeant ainsi potentiellement des composantes clés exigées pour que le client change. Les seuls participants qui ont mis en place la formation COS-P en totalité avaient la formation supplémentaire de Cercle de Sécurité.

Higher compliance to a neuromuscular injury prevention program improves overall injury rate in male football players.

PURPOSE: The 11+ injury prevention program has been shown to decrease injury rate. However, few studies have investigated compliance and if it is correlated to time loss. The purpose of this study was to (1) analyze how differences in compliance may impact injury rate and (2) if compliance may impact time loss due to injury. METHODS: This study was a Level 1 prospective cluster randomized controlled trial conducted in NCAA men's football (soccer) teams that examined the efficacy of the 11+ injury prevention program. The two outcome variables examined were number of injuries and number of days missed from competition. Twenty-seven teams (n = 675 players) used the 11+ program. Compliance, injuries and time loss were recorded. There were three compliance categories, low (LC, 1-19 doses/season), moderate (MC, 20-39 doses/season), and high (HC, > 40 doses/season). RESULTS: There was a significant difference among the groups for injuries, p = 0.04, pη2 = 0.23. The LC group [mean (M) = 13.25, 95% confidence interval (CI) 9.82-16.68, injury rate (IR) = 10.35 ± 2.21] had a significantly higher injury rate than the HC group (M = 8.33, 95%CI 6.05-10.62, IR = 10.35 ± 2.21), p = 0.02. The MC group (M = 11.21, 95%CI 9.38-13.05, IR = 8.55 ± 2.46) was not significantly different than the LC group, p = 0.29, but was significantly greater than the HC group, p = 0.05. When examined as a continuous variable, compliance was significantly negatively related to injury rate (p = 0.004). It was also significantly negatively related to number of days missed (p = 0.012). CONCLUSIONS: When compliance was high, there was a significant reduction in injury and time loss. This evidence reinforces the importance of consistent injury prevention program utilization. Clinically, these findings have important implications when discussing the importance of consistent utilization of an injury prevention protocol in sport. LEVEL OF EVIDENCE: Level 1-Randomized controlled trial (RCT).

Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions.

As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers' ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane's (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.

Replicating or franchising a STEM afterschool program model: core elements of programmatic integrity.

BACKGROUND: Designed in 2012 with a first implementation in 2013, NE STEM 4U is a professional development program for post-secondary students/undergraduates, and serves as a source of outreach, content knowledge generation, and STEM literacy for youth in grades kindergarten through 8th grade (ages 5-14). The model empowers post-secondary students as facilitators of inquiry-based learning within the context of an out-of-school time program. This study investigated the potential for replicating or 'franchising' this model by evaluating on the following: (1) Is the model replicable? And, if so, (2) what core elements are necessary for program fidelity? And (3) is there a dependency on a particular setting/participant type (e.g., a more rural or urban setting)? RESULTS: Strategic expansion of the program to different institutional types (i.e., Research 1, Research II, and a predominantly undergraduate institution), different geographical locations (i.e., rural and urban), and with various school district partners (i.e., large and small) determined that program fidelity and replicability required 4 core elements or criteria: (i) intentional programming, (ii) staff quality, (iii) effective partnerships, and (iv) program evaluation and continuous improvement. Importantly, we examined emergent themes by each site, as well as in combination (n = 16 focus group participants, n = 12 reflection surveys). These data indicated that Flexibility (21.22%), Student Engagement (i.e., Youth) (19.53%), Classroom Management (i.e., also pertaining to youth) (19.31%), and Communication (15.71%) were the themes most referenced by the post-secondary student mentors in the NE STEM 4U program, regardless of site. Finally, the YPQA results demonstrate general replication of program quality in a "franchise" location. CONCLUSIONS: These results highlight the core elements of the NE STEM 4U program for consideration of expansion (through strategic replication or 'franchising') as a possible international model. The findings and voices highlight the program's trajectory toward success into environments that expand professional development for post-secondary students, and for delivering STEM opportunities for youth. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s40594-021-00320-0.

Using Process Evaluation Results to Compare Peer and Adult Leader Delivery of the PAWS (Peer-Education about Weight Steadiness) Club Program.

To date, there is limited published literature on process evaluation of adolescent health promotion programs. In this paper, we describe the methods and results of PAWS Club process evaluation over 2 years of implementation to compare the effectiveness of delivery by peer and adult leaders. PAWS (Peer-education About Weight Steadiness) Club was a 12-week healthy lifestyle program, delivered to 6th and 7th graders by peer and adult educators, using cluster randomized controlled design. Peer educators were 8th graders in the program schools and adult educators were staff/teachers in the program schools. Trained university students filled out fidelity logs at each session led by peer and adult educators to assess program delivery. The fidelity logs included questions to collect information about the number of participants, duration of the session, percent of activities completed, and if lessons started on time, lesson objectives were clearly stated, lesson objectives were emphasized, demonstrations were visible to participants, all activities were completed, the leader was familiar with lessons, the leader maintained an appropriate pace, the leader kept participants on track, and the leader asked if participants had any questions. Adult educators had a higher mean performance for all questions compared to peer leaders. Significant differences were observed for emphasizing lesson objectives (p = 0.005), making demonstrations visible to participants (p = 0.031), being familiar with the lesson plan (p = 0.000), maintaining an appropriate pace (p = 0.000), keeping participants on track (p = 0.000), and asking if participants had any questions (p = 0.000). Significance was set at p < 0.05. Findings from the current study have implications for designing and conducting a process evaluation of complex healthy lifestyle programs with adolescents in schools. Additional training of peer educators may be needed to enhance program delivery.

Evaluating Delivery of a CBT-Based Group Intervention for Schoolchildren With Emotional Problems: Examining the Reliability and Applicability of a Video-Based Adherence and Competence Measure.

Adherence and competence are essential parts of program fidelity and having adequate measures to assess these constructs is important. The Competence and Adherence Scale for Cognitive Behavioral Therapy (CAS CBT) was developed to evaluate the delivery of cognitive therapies for children with clinical anxiety. The present study is an assessment of the slightly adapted version of the CAS CBT evaluating the delivery of a Cognitive Behavioral Therapy (CBT)-based preventive group intervention: EMOTION: Kids Coping with Anxiety and Depression. This study was part of a Norwegian cluster randomized controlled trial (cRCT) investigating the effectiveness of a transdiagnostic intervention, the EMOTION program-an indicated prevention program targeting anxious and depressive symptoms. The applicability and psychometric properties of the CAS CBT were explored. Results are based on six raters evaluating 239 video-recorded sessions of the EMOTION program being delivered by 68 trained group leaders from different municipal services. Interrater reliability (intraclass correlation coefficients, ICC [3, 1]) indicated fair to good agreement between raters. Internal consistency of the instrument's key domains was calculated using the Omega coefficient which ranged between 0.70 to 0.94. There was a strong association between the two scales Adherence and Competence, and inter-item correlations were high across the items, except for the items rating the adherence to the session goals. Competence and Adherence Scale for Cognitive Behavioral Therapy is a brief measure for use in first-line services, with some promising features for easily assessing program fidelity, but some of the results indicated that the instrument should be improved. Future attention should also be made to adapt the instrument to fit better within a group setting, especially regarding evaluation of session goals. More research on how to adequately evaluate fidelity measures are also warranted. Clinical Trial Registration: www.ClinicalTrials.gov, identifier: NCT02340637.

A Qualitative Analysis of Program Fidelity and Perspectives of Educators and Parents after Two Years of the Girls' Iron-Folate Tablet Supplementation (GIFTS) Program in Ghanaian Secondary Schools.

BACKGROUND: To address the burden of anemia in adolescent girls in Ghana, the Girls' Iron-Folate Tablet Supplementation (GIFTS) program was established in 2017. An evaluation found that although iron and folic acid (IFA) supplementation reached nearly all adolescent girls in schools during year 1, most girls received fewer than the minimum effective number of tablets over the school year. Barrier analyses highlighted schools as drivers of adherence, though information was incomplete on the reasons for the disparities among schools. Information was also lacking on the implementation of health and nutrition education. OBJECTIVES: At the start of year 3 of an integrated adolescent anemia prevention program with IFA supplementation, the present study sought to illuminate differences in program fidelity among schools and across time and potential factors that drive such differences. METHODS: After stratifying by school level, size, geographic location, and intake adherence during year 1, 16 schools were purposively selected. For each school, semistructured key informant interviews were conducted with 1 coordinator at the district level, 3 educators at the school level, and 1 parent leader. Following thematic analysis methods, recorded and transcribed interviews were coded and organized into deductive and inductive themes. RESULTS: Limited training, challenges during distribution of IFA, lack of incentives, and inconsistent health and nutrition education diminished program fidelity. Strong supply chain, widespread awareness promotion, improved acceptability, and intrinsically motivated educators improved program fidelity. After 2 y of implementation, schools had made program adaptations, and widespread changes in attitudes and beliefs about the IFA tablets had improved their acceptability. However, limitations remained related to supply chain, program ownership, communication between health and education sectors, training, motivation, and resources. CONCLUSIONS: The fidelity of Ghana's GIFTS program is strengthened by its supply chain, acceptability, and motivated stakeholders; however, training, curricula, clear communication, and incentives could improve it.

Strategies to Avoid Replication Failure With Evidence-Based Prevention Interventions: Case Examples From the Strengthening Families Program.

Research has found disturbing long-term effects of poor parenting on children's behavioral health including addiction, delinquency, depression/anxiety, and poorer health as adults. Poor parenting practices thus contribute substantially to the health crisis in America. However, skilled, nurturing parents, or caretakers can help youth avoid these developmental problems. A number of family and parenting evidence-based interventions (EBIs) that teach parenting skills are now available for dissemination. Unfortunately, replications of EBIs do not always produce the original positive results. Organizations that seek to use family EBIs to improve parenting and family skills need to avoid practices that create replication failure. We examine several possible factors that contribute to replication failure using examples from five replications of the EBI "Iowa Strengthening Families Program for ages 10-14." We then share six strategies conducive to avoid replication failures including (1) choosing the right program and implementation strategy for the population, (2) administering the right "dosage," (3) choosing and properly training implementers, (4) maintaining program integrity and adherence, (5) ensuring cultural sensitivity, and (6) ensuring accurate and complete reporting of evaluation results. These guidelines can advance prevention science to meet the demands of a growing public health agenda.

Process Evaluation of a Large-Scale Community-Based Nutrition Program in Malawi.

BACKGROUND: Global attention to the study of nutrition program implementation has been inadequate yet is critical for effective delivery and impact at scale. OBJECTIVES: The objective of this mixed-methods process evaluation study was to measure the recruitment, fidelity, and reach of a large-scale, community-based nutrition program in Malawi. METHODS: The nutrition program delivered a small-quantity lipid-based nutrient supplement (SQ-LNS) and social and behavior change communication (SBCC) to improve infant and young child feeding (IYCF) and water, sanitation, and hygiene (WASH) practices in households with children aged 6-23 mo. Program monitoring and evaluation data were used to measure program recruitment, reach, and fidelity. Structured direct observations and knowledge questionnaires with program volunteers measured quality aspects of program fidelity. The number of times activities were done correctly was used to tabulate proportions used to represent program functioning. RESULTS: Half (49.5%) of eligible children redeemed program benefits by 8 mo of age during the first 4 y of program implementation. Implementation of training activities for SBCC cadres exceeded program targets (100.6%), but the completion of certain modules (breastfeeding and complementary feeding) was lower (22.9% and 18.6%, respectively). Knowledge of IYCF, WASH, and SQ-LNS messages by volunteers was >85% for most messages, except ability to list the 6 food groups (35.7%). Structured direct observations of SQ-LNS distributions indicated high fidelity to program design, whereas those of household-level counseling sessions revealed lack of age-appropriate messaging. Program reach showed participation in monthly distribution sessions of 81.0%, group counseling of 93.3%, and individual-level counseling of 36.9%. CONCLUSIONS: This community-based nutrition program was implemented with high fidelity and quality, with specific interventions requiring further attention. The documentation of implementation contributes to our understanding about how program impacts were achieved.

A comparison of sampling approaches for monitoring schoolwide inclusion program fidelity.

Including students with disabilities requires schoolwide interventions that are implemented with fidelity (adherence). Collection of fidelity data may become problematic when multiple evidence-based treatments exist in one setting. To address concerns around efficiency of data collection, this study hypothesized that the three sampling approaches (proportional, consensus, stratified random) provided similar levels of agreement with an expert rater, thus warranting the differentiation in their applications based on the needs of the evaluators. Three high schools were randomly assigned to one of three sampling approaches (i.e. proportional, consensus, stratified random) to complete a fidelity measure for the implementation of a schoolwide inclusion approach. Based on the Median Test, each sampling approach did not differ significantly from the scores of an expert rater. The results indicate that schools may have some choice in the methods they use to sample their staff around the measurement of fidelity of implementation.

Lessons learned: Evaluating the program fidelity of UNWomen Partnership for Peace domestic violence diversion program in the Eastern Caribbean.

To date, there have been a plethora of punitive and diversion programs to address domestic violence around the world. However, the evaluative scholarship of such programs overwhelmingly reflects studies in developed countries while barely showcasing the realities of addressing domestic violence in developing countries. This paper features a multi-year (2008-2011) evaluation study that measured the fidelity of the United Nations Partnership for Peace (PfP) domestic violence diversion program in the Eastern Caribbean country of Grenada. Our findings illuminate organic engagement strategies that were built within existing multi-sectoral partnerships that included magistrate court judges, law enforcement officials, and social service agencies. Furthermore, we documented how the locally-devised implementation strategies ensured the program's fidelity within a resource-limited context. This paper contributes to the global evaluative scholarship, highlighting the lessons learned about implementing culturally-adapted and theoretically-driven domestic violence diversion within a developing country.

Pilot Study of a Group-Based Psychosocial Trauma Recovery Program in Secure Accommodation in Scotland.

The current study is the first to implement and evaluate a group-based trauma-specific program for adolescents in a secure accommodation facility in Scotland. A randomized control and qualitative pilot study compared an intervention group (n = 10), who received Teaching Recovery Techniques, to a waitlist control group (n = 7). Measures included subjective units of disturbance (SUDs), standardized trauma symptom questionnaires, and analysis of behavior monitoring logs. Adolescent interviews (n = 10) and a presenter focus group (n = 4) assessed program experience and views on future development. Sessions were videoed and analyzed for program adherence. Analysis involved MANOVA, and a quasi-qualitative thematic approach for participant views. Adolescents reported high SUDs and a range of trauma symptoms. A large effect size was found for reduced SUDs (d = 1.10) and positive trends were identified for symptoms and behavior change in the intervention group. Program adaptations included smaller groups, the use of visual materials and liaison with care staff to facilitate generalization. Recommendations are made for program development and large scale evaluation.

Sustaining program effectiveness after implementation: The case of the self-management of well-being group intervention for older adults.

OBJECTIVE: The Self-Management of Well-being (SMW) group intervention for older women was implemented in health and social care. Our aim was to assess whether effects of the SMW intervention were comparable with the original randomized controlled trial (RCT). Furthermore, we investigated threats to effectiveness, such as participant adherence, group reached, and program fidelity. METHODS: In the implementation study (IMP) 287 and RCT 142 women participated. We compared scores on self-management ability and well-being of the IMP and RCT. For adherence, drop-out rates and session attendance were compared. Regarding reach, we compared participants' baseline characteristics. Professionals completed questions regarding program fidelity. RESULTS: No significant differences were found on effect outcomes and adherence between IMP and RCT (all p≥0.135). Intervention effect sizes were equal (0.47-0.59). IMP participants were significantly less lonely and more likely to be married, but had lower well-being. Most professionals followed the protocol, with only minimal deviations. CONCLUSION: The effectiveness of the SMW group intervention was reproduced after implementation, with similar participant adherence, minimal changes in the group reached, and high program fidelity. PRACTICE IMPLICATIONS: The SMW group intervention can be transferred to health and social care without loss of effectiveness. Implementation at a larger scale is warranted.