Inconsistent Counselor Fidelity in Delivering an Evidence-Based Adherence Intervention During a PrEP Trial.

Evidence-based adherence counseling interventions must be delivered with fidelity to ensure that their effectiveness is retained, but little is known regarding how counselors in biomedical HIV trials deliver these interventions. Forty-two counselors from the MTN-025/HOPE Study, which was conducted in 14 sites in sub-Saharan Africa, participated. They completed a quantitative assessment and consented for their HOPE counseling session ratings to be analyzed. Twenty-two (52%) self-identified as research nurses and 20 (48%) as counselors. Of 928 session ratings, 609 (66%) were classified as Good, 188 (20%) as Fair, and 131 (14%) as Poor, based on pre-established criteria. Overall mean ratings for session tasks and global components (each rated from 1 to 5) were 4.12 (SD = 0.45; range 2.46-4.73) and 4.02 (SD = 0.64; range 1.75-4.79), respectively. Twenty-six (62%) counselors attained Good or Fair ratings on at least 85% of their sessions, but 33% of counselors had more than 25% of their sessions rated as Poor; three counselors (7%) never met criteria for a Good session. Even after extensive training, counselors' fidelity to the intervention varied. Our findings highlight the value of fidelity monitoring using session audio-recordings, the importance of ongoing coaching and support, and the need to plan for counselors with consistently poor fidelity.

RESUMEN: Las intervenciones de consejería de adherencia basadas en la evidencia deben ser realizadas con fidelidad para asegurar que retengan su efectividad, pero se sabe poco sobre cómo los consejeros en ensayos biomédicos de VIH realizan estas intervenciones. Cuarenta y dos consejeros participaron del Estudio MTN-025/HOPE, el cual se llevó a cabo en 14 sitios en África subsahariana. Completaron un cuestionario cuantitativo y dieron su consentimiento para el análisis de las calificaciones de sus sesiones de consejería para HOPE. Veintidós (52%) se identificaron como enfermeras investigadoras y 20 (48%) como consejeros. De 928 sesiones calificadas, 609 (66%) fueron clasificadas como Buenas, 188 (20%) como Suficientes, y 131 (14%) como Mediocres, basado en criterios preestablecidos. Las calificaciones promedias de las tareas de las sesiones y de los componentes globales (calificados de 1 a 5) fueron de 4.12 (SD = 0.45; rango 2.46­4.73) y de 4.02 (SD = 0.64; rango 1.75­4.79), respectivamente. Veintiséis (62%) consejeros lograron calificaciones Buenas o Suficientes en al menos el 85% de sus sesiones, pero para el 33% de los consejeros, más del 25% de sus sesiones fueron calificadas como Mediocres; tres consejeros (7%) nunca cumplieron con los criterios de una Buena sesión. Aún después de una capacitación intensiva, la fidelidad a la intervención variaba. Nuestros hallazgos destacan el valor de monitorear para la fidelidad usando audio-grabaciones de las sesiones, la importancia de la instrucción y el apoyo continuo, y la necesidad de planear qué medidas tomar cuando hay consejeros que no son fieles a la intervención.

Counselors' acceptability of adherence counseling session recording, fidelity monitoring, and feedback in a multi-site HIV prevention study in four African countries.

Biomedical HIV prevention trials increasingly include evidence-based adherence counseling to encourage product use. To retain effectiveness, interventions must contain key components. Monitoring counseling fidelity ensures inclusion of components but is challenging in multinational contexts with different languages and scarce local supervision. The MTN-025/HOPE Study, a Phase 3b open-label trial to assess continued safety of and adherence to the dapivirine vaginal ring for HIV prevention, was the largest such trial to integrate fidelity monitoring using audio recordings of counseling sessions. We describe the monitoring process, along with counselor and participants' perceptions of it, which were collected via quantitative online survey (counselors only N = 42) and in-depth interviews with a subset of counselors (N = 22) and participants (N = 10). Sessions were conducted in five languages across 14 study sites in four countries. In total, 1238 sessions (23% of submitted sessions) were randomly selected and rated. Assessment of interrater reliability was essential to address drift in ratings. Counselors were apprehensive about being monitored, but appreciated clear guidance and found ratings very helpful (mean = 6.64 out of 7). Some participants perceived sessions as time-consuming; others found monitoring improved counseling quality. Fidelity monitoring of counseling sessions in mult-isite biomedical HIV studies is feasible and supportive for counselors.

Implementing evidence-based programs: a four-step protocol for assuring replication with fidelity.

Health care providers are challenged to replicate evidence-based programs in their communities. These programs may be adapted to fit new communities, but the key components must be delivered with fidelity. This article describes a four-step fidelity assurance protocol developed by the Hawai'i's Healthy Aging Partnership as it adapted and replicated evidence-based health promotion programs for Hawai'i's older adults. The four steps are the following: (a) deconstruct the program into its components and prepare a step-by-step plan for program replication; (b) identify agencies ready to replicate the program, and sponsor excellent training to local staff who will deliver and coordinate it; (c) monitor the fidelity of program delivery using standardized checklists; and (d) track participant outcomes to assure achievement of expected outcomes. The protocol is illustrated with examples from Hawai'i's Healthy Aging Partnership's experience replicating EnhanceFitness, a senior exercise program. This protocol is transferrable to other communities wanting to adapt and replicate evidence-based, public health programs.

Monitoring unplanned sexual health curricula adaptations: Using results to improve fidelity and support implementation.

Maintaining fidelity to an evidence-based curriculum is important, yet educators may need to adapt to unexpected situations or particular contexts. The purpose of this study is to identify the reasons for unplanned adaptations during implementation of an evidence-based sexual health education program in California. Evaluators reviewed fidelity checklists from the implementation of 571 cohorts for activities with reported unplanned adaptations. Reasons were qualitatively coded and compared across two phases of implementation and by setting. Educators reported 319 unplanned adaptations, affecting 21.5% of the 571 cohorts and 2.9% of 13,782 activities. The most common reasons for unplanned adaptations were due to time management issues, site logistic issues, and to increase participant engagement. Over time, health educators reported fewer unplanned adaptations, particularly those due to time management, resulting in a decrease in the cohorts and activities affected. Adaptations to evidence-based curricula are necessary and often occur during implementation to fit local conditions and populations. Ongoing review of adaptation data provides an opportunity to refine training and technical assistance efforts. Guidance about the types of permitted adaptations and how to anticipate and plan for adaptations for future implementation can ensure fidelity to the core curriculum components and responsiveness to youth participants.

Iterative development and pilot testing of an intervention fidelity monitoring plan for the enhanced, electronic health record-facilitated pragmatic clinical trial: Implications for training and protocol integrity.

Fidelity monitoring is the degree to which a clinical trial intervention is implemented as intended by a research protocol. Consistent implementation of research protocols supported with extant fidelity monitoring plans contribute rigor and validity of study results. Fidelity monitoring plans should be comprehensive yet practical to accommodate the realities of conducting research, particularly a pragmatic clinical trial, in dynamic settings with heterogeneous patient populations. The purposes of this paper are to describe the (1) iterative development and implementation of protocols for intervention fidelity monitoring, (2) pilot testing of the fidelity monitoring plan, (3) the identification of interventionist training deficiencies, and (4) opportunities to enhance protocol rigor for a cancer symptom management intervention delivered through the electronic health record patient portal and telephone as part of a complex, multi-component pragmatic clinical trial to uncover training deficits and bolster protocol integrity. The intervention focuses on prominent symptoms reported among medical oncology patients including sleep disturbance, pain, anxiety, depression, low energy (fatigue) and physical function. In this pragmatic trial, the role of interventionist is a registered nurse symptom care manager (RN SCM). A three-part fidelity monitoring plan with checklists audit: Part-1 RN SCM role training activities in research components, clinical training components, and protocol simulation training; Part-2 RN SCM adherence to the intervention core components delivered over the telephone; and Part-3 maintenance of adherence to core intervention components. The goal is ≥ 80% adherence to components of each of the three checklists. An initial pilot test of the fidelity monitoring plan was conducted to evaluate the checklists and the RN SCM adherence to core protocol components. RN SCM skills and training deficits were identified during the pilot phase, as were opportunities to improve protocol integrity. Overall, approximately 50% of the audited RN SCM telephone calls had ≥80% fidelity to the core components. There remains on-going need for RN SCM training and skill building in action planning. The content presented in this paper is intended to begin to fill the gap of fidelity monitoring plans for complex interventions tested in pragmatic clinical trials and delivered remotely in an effort to strengthen protocol integrity.

Intervention fidelity monitoring of Urban Zen Integrative Therapy (UZIT) for persons with pulmonary hypertension.

BACKGROUND: Systematic and consistent dose delivery is critical in intervention research. Few studies testing complementary health approach (CHA) interventions describe intervention fidelity monitoring (IFM) and measurement. OBJECTIVE: To describe methodological processes in establishing and measuring consistent dose, delivery, and duration of a multi-component CHA intervention. METHODS: Adults with pulmonary hypertension received six weekly, 1-hour Urban Zen Integrative Therapy (UZIT) sessions. A total of 78 sessions were delivered and 33% of these sessions were audited. Intervention dose (time allocated to each component), intervention consistency (protocol adherence audits), and intervention delivery (performance and sequence of components) were captured using remote video observation and review of the recorded video. IFM audits were performed at the beginning (n = 16), middle (n = 5), and end (n = 5) of the study. RESULTS: UZIT interventionists adhered to the intervention protocol (99.3%) throughout the study period. Interventionists delivered UZIT components within the prescribed timeframe: 1) Beginning: gentle body movement (18.9 ± 5.8 min.), restorative pose with guided body awareness meditation (21.3 ± 2.7 min.), and Reiki (22.8 ± 3.1 min.); 2) Middle: gentle body movement (15.9 ± 1.5 min.), pose/body awareness meditation (30.1 ± 6.5 min.), and Reiki (30.1 ± 7.0 min.); 3) End: gentle body movement (18.1 ± 3.6 min.), pose/body awareness meditation (35.3 ± 6.4 min.), and Reiki (34.5 ± 7.0 min.). Essential oil inhalation was delivered during UZIT sessions 100% of the time. Interventionists adhered to treatment delivery behaviors throughout the study period: beginning (98.86%), middle (100%), and end (100%). DISCUSSION: In this pilot study, we demonstrated that the dose, consistency, and delivery of multi-component CHA therapy can be standardized and monitored to ensure intervention fidelity.

Using data to improve fidelity when implementing evidence-based programs.

In fall 2011, the South Carolina Campaign to Prevent Teen Pregnancy (SC Campaign), with funding from Office of Adolescent Health, began replicating an evidence-based curriculum, It's Your Game, Keep It Real in 12 middle schools across South Carolina. Fidelity of the curriculum was monitored by the use of lesson fidelity logs completed by curriculum facilitators and lesson observation logs submitted by independent classroom observers. These data were monitored weekly to identify possible threats to fidelity. The innovative model Fidelity Through Informed Technical Assistance and Training was developed by SC Campaign to react to possible fidelity threats in real time, through a variety of technical assistance modalities. Fidelity Through Informed Technical Assistance and Training guided the 55 hours of technical assistance delivered by the SC Campaign during the first year of It's Your Game, Keep It Real implementation to 18 facilitators across 12 SC middle schools, and achieved 98.4% curriculum adherence and a high quality of implementation scores.