Pilot Trial of Online Measurement Training and Feedback in Family Therapy for Adolescent Behavior Problems.

OBJECTIVE: Pragmatic procedures for sustaining high-fidelity delivery of evidence-based interventions are needed to support implementation in usual care. This study tested an online therapist training system, featuring observational coder training and self-report fidelity feedback, to promote self-report acumen and routine use of family therapy (FT) techniques for adolescent behavior problems. METHOD: Therapists (N = 84) from nine substance use and mental health treatment sites reported on 185 adolescent clients. Therapists submitted baseline data on FT technique use with clients, completed a workshop introducing the 32-week training system, and were randomly assigned by site to Core Training versus Core Training + Consultation. Core Training included a therapist coder training course (didactic instruction and mock session coding exercises in 13 FT techniques) and fidelity feedback procedures depicting therapist-report data on FT use. Consultation convened therapists and supervisors for one-hour monthly sessions with an external FT expert. During the 32 weeks of training, therapists submitted self-report data on FT use along with companion session audiotapes subsequently coded by observational raters. RESULTS: Therapist self-report reliability and accuracy both increased substantially during training. Observers reported no increase over time in FT use; therapists self-reported a decrease in FT use, likely an artifact of their improved self-report accuracy. Consultation did not enhance therapist self-report acumen or increase FT use. CONCLUSIONS: Online training methods that improve therapist-report reliability and accuracy for FT use may confer important advantages for treatment planning and fidelity monitoring. More intensive and/or different training interventions appear needed to increase routine FT delivery.

Therapist Self-Report of Fidelity to Core Elements of Family Therapy for Adolescent Behavior Problems: Psychometrics of a Pragmatic Quality Indicator Tool.

Therapist-report measures of evidence-based interventions have enormous potential utility as quality indicators in routine care; yet, few such tools have shown strong psychometric properties. This study describes reliability and validity characteristics of a therapist-report measure of family therapy techniques for treating adolescent conduct and substance use problems: Inventory of Therapy Techniques for Core Elements of Family Therapy (ITT-CEFT). Study participants included 31 staff therapists treating 68 adolescent clients in eight community-based mental health and substance use clinics. Therapists submitted ITT-CEFT checklists and companion audio recordings for 189 sessions. The ITT-CEFT contains 13 techniques identified as core elements of three manualized family therapy models that are empirically supported for the target group. Therapists also reported on their use of three motivational interventions, and independent observers coded the submitted recordings. ITT-CEFT factor validity was shown via confirmatory factor analyses of the tool's theoretical structure. Derived modules were: Family Engagement (four items; Cronbach's α = .72); Relational Orientation (five items; α = .74); and Interactional Change (four items; α = .66). Concurrent validity analyses showed fair-to-excellent therapist reliability compared to observer ratings (ICCs range .64-.75); they showed moderate therapist accuracy compared to observer mean scores, reflecting a tendency to overestimate delivery of the techniques. Discriminant validity analyses showed tool differentiation from motivational interventions. Results offer provisional evidence for the feasibility of using the therapist-report ITT-CEFT to anchor quality procedures for family therapy interventions in real-world settings.Trial Registration: The parent clinical trial is registered at www.ClinicalTrials.gov , ID: NCT03342872 (registration date: 11.10.17).

Online Training of Community Therapists in Observational Coding of Family Therapy Techniques: Reliability and Accuracy.

A foundational strategy to promote implementation of evidence-based interventions (EBIs) is providing EBI training to therapists. This study tested an online training system in which therapists practiced observational coding of mock video vignettes demonstrating family therapy techniques for adolescent behavior problems. The study compared therapists ratings to gold-standard scores to measure therapist reliability (consistency across vignettes) and accuracy (approximation to gold scores); tested whether reliability and accuracy improved during training; and tested therapist-level predictors of overall accuracy and change in accuracy over time. Participants were 48 therapists working in nine community behavioral health clinics. The 32-exercise training course provided online instruction (about 15 min/week) in 13 core family therapy techniques representing three modules: Family Engagement, Relational Orientation, Interactional Change. Therapist reliability in rating technique presence (i.e., technique recognition) remained moderate across training; reliability in rating extensiveness of technique delivery (i.e., technique judgment) improved sharply over time, from poor to good. Whereas therapists on average overestimated extensiveness for almost every technique, their tendency to give low-accuracy scores decreased. Therapist accuracy improved significantly over time only for Interactional Change techniques. Baseline digital literacy and submission of self-report checklists on use of the techniques in their own sessions predicted coding accuracy. Training therapists to be more reliable and accurate coders of EBI techniques can potentially yield benefits in increased EBI self-report acumen and EBI use in daily practice. However, training effects may need to improve from those reported here to avail meaningful impact on EBI implementation.Trial Registration: The parent clinical trial is registered at www.ClinicalTrials.gov , ID: NCT03342872 (registration date: 11.10.17).