The Content of Usual Clinical Care for Youth with Primary Anxiety Problems.

The current study was designed to describe usual clinical care for youth with primary anxiety problems in community mental health centers. The observer-rated Therapy Process Observational Coding System for Child Psychotherapy - Revised Strategies scale (TPOCS-RS), designed to assess therapeutic techniques from five theory-based domains, was used to code sessions (N = 403) from the usual clinical care group of two randomized effectiveness trials: (a) Youth Anxiety Study (YAS) with 21 youth (M age = 10.44 years, SD = 1.91; 49.2% Latinx; 46.6%, 53.4% male) and 16 clinicians (77.5% female; 43.8% White), and (b) Child STEPS Multisite Trial with 17 youth (M age = 10.00 years, SD = 1.87; 58.8% male; 41.2% White) and 13 clinicians (M age = 40.00 years; SD = 9.18; 76.9% female; 61.5% White). The average number of TPOCS-RS items observed per treatment session was more than 10, and multiple techniques were used together in each session. All TPOCS-RS items were observed at least once throughout a clinical case, and most items reoccurred (i.e., observed in two or more sessions). The dosage of TPOCS-RS in all items was below 5 on a 7-point scale. In conclusion, clinicians in both usual care samples used a wide range of techniques from several theory-based domains at a low to medium dose. However, the type and dosage of the techniques used did vary across the two samples.

Using Adherence and Competence Measures Based on Practice Elements to Evaluate Treatment Fidelity for Two CBT Programs for Youth Anxiety.

Measures designed to assess the quantity and quality of practices found across treatment programs for specific youth emotional or behavioral problems may be a good fit for evaluating treatment fidelity in effectiveness and implementation research. Treatment fidelity measures must demonstrate certain reliability and validity characteristics to realize this potential. This study examines the extent to which two observational measures, the Cognitive-Behavioral Treatment for Anxiety in Youth Adherence Scale (CBAY-A) and the CBAY Competence Scale (CBAY-C), can assess the quantity (the degree to which prescribed therapeutic techniques are delivered as intended) or quality (the competence with which prescribed techniques are delivered) of practices found in two distinct treatment programs for youth anxiety. Treatment sessions (N = 796) from 55 youth participants (M age = 9.89 years, SD = 1.71; 46% female; 55% White) with primary anxiety problems who participated in an effectiveness study were independently coded by raters who coded quantity, quality, and the youth-clinician alliance. Youth received one of three treatments: (a) standard (i.e., cognitive-behavioral therapy program), (b) modular (i.e., a cognitive-behavioral and parent-training program), and (c) usual clinical care. Interrater reliability for the CBAY-A items was good across the standard and modular conditions but mixed for the CBAY-C items. Across the standard and modular conditions, the CBAY-A Model subscale scores demonstrated evidence of construct validity, but the CBAY-C Model subscale scores showed mixed evidence. The results provide preliminary evidence that the CBAY-A can be used across different treatment programs but raise concerns about the generalizability of the CBAY-C.

A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers.

Introduction: Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms. Method: Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools-Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure. Results: Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested. Conclusion: The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.

Collecting observational data in schools is challenging, so developing teacher self-report measures and involving teachers in the design process is important to help make them easier to use. This paper reports on the development of a teacher self-report measure designed to collect information about the instructional practices teachers deliver to promote positive student behavior.

Practitioner Adherence and Competence in MEYA, a Free Online Self-Instruction Program in Modular Psychotherapy and Counseling for Children's Autism-Related Clinical Needs.

The quality of care in public schools and other community settings for school-aged youths on the autism spectrum is variable and often not evidence-based. Training practitioners in these settings to deliver evidence-based practices (EBPs) may improve the quality of care. We developed a free internet-based training and clinical guidance system synthesizing multiple EBPs for youth on the autism spectrum addressing a range of mental health needs and autism-related behaviors, entitled Modular EBPs for Youth on the Autism Spectrum (MEYA; meya.ucla.edu). A multiple baseline study was conducted with seven practitioners recruited from mental health practice settings across the United States who were providing services to children on the autism spectrum (aged 6 to 17 years). Practitioners were randomly assigned to undergo baseline conditions of 2 to 8 weeks. Once online training in MEYA commenced, practitioners engaged in algorithm-guided self-instruction in EBPs for autism. Participants video-recorded sessions. Independent coders used the MEYA Fidelity Scale (MEYA-FS) to rate adherence and competence in EBPs for autism. Practitioners also completed measures pertaining to implementation outcomes and parents rated youth outcomes on personalized target behaviors. Five of seven practitioners increased their adherence to MEYA practices (i.e., MEYA-FS scores) following MEYA training. Findings for competence were similar, though somewhat less robust. Practitioners generally viewed MEYA as feasible, understandable, and acceptable. Most youth outcomes improved during MEYA. A randomized, controlled trial of MEYA would be helpful in characterizing its effectiveness for supporting practitioner EBP implementation and youth outcomes in school and community service settings.

Assessing the effectiveness and implementation of a universal classroom-based set of educator practices to improve preschool children's social-emotional outcomes: Protocol for a cluster randomized controlled type 2 hybrid trial in Singapore.

BACKGROUND: Providing high-quality early childhood care and education is understood as key to maximizing children's potential to succeed later in life, as it stimulates young children's development of skills and competencies needed to promote optimal outcomes and success later in life. Despite the government's efforts to support the early childhood sector, educators in Singapore continue to report difficulties in implementing practices in classrooms that promote children's social, emotional, and cognitive development. To enhance educators' skills in these domains, we developed the Enhancing And Supporting Early development to better children's Lives (EASEL) Approach, a set of universal, educator-delivered practices for use with 3-6-year-old children in early childhood settings to improve social, emotional, behavioral, and executive functioning (SEB+EF) outcomes. METHODS: This study will evaluate the effectiveness and implementation of the EASEL Approach in improving early childhood educators' teaching practices and, in turn, children's SEB+EF outcomes. We will conduct a cluster randomized controlled trial with a type 2 hybrid effectiveness-implementation study in 16 childcare centers. The EPIS (Explore, Prepare, Implement, Sustain) Framework will be used to inform the implementation of the EASEL Approach. Implementation strategies include training, educator self-assessments, practice-based coaching, and data monitoring. Our primary outcome is educators' teaching practices. Secondary outcomes include educators' adoption of the EASEL Approach in everyday practice, the acceptability and feasibility of the EASEL Approach, and children's SEB+EF outcomes. Quantitative and qualitative data will be collected at baseline, six months, and after implementation. CONCLUSION: Findings from this study will provide significant evidence on the effectiveness of the EASEL Approach in improving educators' teaching practices and its impact on children's SEB+EF outcomes and the implementation of the EASEL Approach in early childhood classrooms in Singapore. TRIAL REGISTRATION: This study was prospectively registered on ClinicalTrials.gov, Identifier: NCT05445947 on 6th July 2022.

Problem-solving interventions and depression among adolescents and young adults: A systematic review of the effectiveness of problem-solving interventions in preventing or treating depression.

Problem-solving (PS) has been identified as a therapeutic technique found in multiple evidence-based treatments for depression. To further understand for whom and how this intervention works, we undertook a systematic review of the evidence for PS's effectiveness in preventing and treating depression among adolescents and young adults. We searched electronic databases (PsycINFO, Medline, and Cochrane Library) for studies published between 2000 and 2022. Studies meeting the following criteria were included: (a) the intervention was described by authors as a PS intervention or including PS; (b) the intervention was used to treat or prevent depression; (c) mean or median age between 13-25 years; (d) at least one depression outcome was reported. Risk of bias of included studies was assessed using the Cochrane Risk of Bias 2.0 tool. A narrative synthesis was undertaken given the high level of heterogeneity in study variables. Twenty-five out of 874 studies met inclusion criteria. The interventions studied were heterogeneous in population, intervention, modality, comparison condition, study design, and outcome. Twelve studies focused purely on PS; 13 used PS as part of a more comprehensive intervention. Eleven studies found positive effects in reducing depressive symptoms and two in reducing suicidality. There was little evidence that the intervention impacted PS skills or that PS skills acted as a mediator or moderator of effects on depression. There is mixed evidence about the effectiveness of PS as a prevention and treatment of depression among AYA. Our findings indicate that pure PS interventions to treat clinical depression have the strongest evidence, while pure PS interventions used to prevent or treat sub-clinical depression and PS as part of a more comprehensive intervention show mixed results. Possible explanations for limited effectiveness are discussed, including missing outcome bias, variability in quality, dosage, and fidelity monitoring; small sample sizes and short follow-up periods.

Examining the Role of Student Responsiveness in Treatment Effects of a Tier 2 Program Targeting Reductions in Problem Behavior.

Student responsiveness's role in promoting intervention outcomes for students who exhibit problem behavior is understudied. Due to the relational nature of many interventions delivered by teachers that target social, emotional, or behavioral outcomes of students in classrooms, it is essential to assess how responsive students are to teachers' attempts to engage them in the intervention, particularly for students with problem behaviors that may impede teachers' attempts to engage these students in intervention effectively. In the current study, we combine samples from four randomized controlled trials to examine the relationship between student outcomes and teacher attempts to deliver BEST in CLASS, a Tier 2 intervention, via student responsiveness. Delivery of BEST in CLASS and student responsiveness were assessed through direct observations and teachers' reported measures. Results suggest that teacher adherence and competence in delivering BEST in CLASS practices was associated with reductions in problem behavior from pretest to post-test via student responsiveness. Limitations of the current study and implications for future research are discussed.

Study Protocol: Multi-level Determinants of Implementation and Sustainment in the Education Sector.

Evidence-based programs (EBPs) delivered in elementary schools show great promise in reducing risk for emotional and behavioral disorders (EBDs). However, efforts to sustain EBPs in school face barriers. Improving EBP sustainment thus represents a priority, but little research exists to inform the development of sustainment strategies. To address this gap, the Sustaining Evidenced-Based Innovations through Multi-level Implementation Constructs (SEISMIC) project will: (a) Determine if malleable individual, intervention, and organizational factors predict EBP treatment fidelity and modifications during implementation, sustainment, or both; (b) Assess the impact of EBP fidelity and modifications on child outcomes during implementation and sustainment; and (c) Explore the mechanisms through which individual, intervention, and organizational factors influence sustainment outcomes. This protocol paper describes SEISMIC, which builds upon a federally-funded RCT evaluating BEST in CLASS, a teacher-delivered program for K-3rd grade children at risk for EBDs. The sample will include 96 teachers, 384 children, and 12 elementary schools. A multi-level, interrupted time series design will be used to examine the relationship between baseline factors, treatment fidelity, modifications, and child outcomes, followed by a mixed-method approach to elucidate the mechanisms that influence sustainment outcomes. Findings will be used to create a strategy to improve EBP sustainment in schools.

The Therapy Process Observational Coding System: Group cohesion scale in youth anxiety treatment: Psychometric properties.

BACKGROUND: There are no well-established measures of group cohesion, defined as the collaborative bond between group members, in group cognitive behavioral therapy (GCBT) with youth. We therefore examined the Therapy Process Observational Coding System for Child Psychotherapy-Group Cohesion Scale (TPOCS-GC), which has previously only been used with adult samples, in a youth sample. METHODS: Observers coded 32 sessions from 16 groups with 83 youth aged 8 to 15 years (90.7% European White). Youth had anxiety disorders and received manualized GCBT in community clinics. We examined psychometric properties of the TPOCS-GC and its' construct validity in terms of relations with pretreatment variables, alliance and fidelity during treatment, and post-treatment variables. Group cohesion was measured twice during treatment (early and late). RESULTS: The TPOCS-GC was internally consistent (α = 0.72) and was reliably coded (M ICC = 0.61). Higher clinical severity at pretreatment predicted lower early group cohesion. Higher youth age, higher clinical severity at pretreatment, and higher youth-rated early alliance predicted lower late group cohesion. Higher therapist-rated early alliance predicted higher early group cohesion. Higher therapist-rated late alliance predicted higher late group cohesion. Higher late group cohesion predicted lower clinical severity and higher client treatment satisfaction at post-treatment. Early group cohesion did not predict any post-treatment variables. CONCLUSIONS: A four-item version of the TPOCS-GC can be reliably used in youth GCBT. The TPOCS-GC is distinct from, but associated with, multiple clinical variables.

Core Elements of Family Therapy for Adolescent Behavioral Health Problems: Validity Generalization in Community Settings.

OBJECTIVE: The core elements of family therapy for adolescent mental health and substance use problems, originally distilled from high-fidelity sessions conducted by expert clinicians, were tested for validity generalization when delivered by community therapists in routine settings. METHOD: The study sampled recorded sessions from 161 cases participating in one of three treatment pools: implementation trial of Functional Family Therapy (98 sessions/50 cases/22 therapists), adaptation trial of Multisystemic Therapy (115 sessions/59 cases/2 therapists), and naturalistic trial of non-manualized family therapy in usual care (107 sessions/52 cases/21 therapists). Adolescents were identified as 60% male and 40% female with an average age of 15.4 years; 49% were Latinx, 27% White Non-Latinx, 15% African American, 3% another race/ethnicity, 6% race/ethnicity unknown. Session recordings (n = 320) were randomly selected for each case and coded for 21 discrete family therapy techniques. Archived data of one-year clinical outcomes were gathered. RESULTS: Confirmatory factor analyses replicated the factor structure from the original distillation study, retaining all four clinically coherent treatment modules comprised of all 21 techniques: Interactional Change (ICC = .77, Cronbach's α = .81); Relational Reframe (ICC = .75, α = .81); Adolescent Engagement (ICC = .72, α = .78); Relational Emphasis (ICC = .76, α = .80). Exploratory analyses found that greater use of core techniques predicted symptom improvements in one treatment pool. CONCLUSIONS: Core techniques of family therapy distilled from manualized treatments for adolescent behavioral health problems showed strong evidence of validity generalization, and initial evidence of links to client outcomes, in community settings.

What is the Status of Multi-Informant Treatment Fidelity Research?

OBJECTIVE: The precise measurement of treatment fidelity (quantity and quality in the delivery of treatment strategies in an intervention) is essential for intervention development, evaluation, and implementation. Various informants are used in fidelity assessment (e.g., observers, practitioners [clinicians, teachers], clients), but these informants often do not agree on ratings. This scoping review aims to ascertain the state of science around multi-informant assessment of treatment fidelity. METHOD: A literature search of articles published through December 2021 identified 673 articles. Screening reduced the number of articles to 44, and the final study set included 35 articles. RESULTS: There was substantial variability across studies regarding study design, how fidelity was operationalized, and how reliability was defined and assessed. Most studies evaluated the agreement between independent observers and practitioner-report, though several other informant pairs were assessed. Overall, findings suggest that concordance across fidelity informants was low to moderate, with a few key exceptions. CONCLUSIONS: It is difficult to draw clear conclusions about the degree to which single versus multiple informant assessment is needed to produce an accurate and complete picture of treatment fidelity. The field needs to take steps to determine how to leverage multi-informant assessment to accurately assess treatment fidelity.

A Randomized Trial to Identify Accurate Measurement Methods for Adherence to Cognitive-Behavioral Therapy.

Clinician fidelity to cognitive behavioral therapy (CBT) is an important mechanism by which desired clinical outcomes are achieved and is an indicator of care quality. Despite its importance, there are few fidelity measurement methods that are efficient and have demonstrated reliability and validity. Using a randomized trial design, we compared three methods of assessing CBT adherence-a core component of fidelity-to direct observation, the gold standard. Clinicians recruited from 27 community mental health agencies (n = 126; M age = 37.69 years, SD = 12.84; 75.7% female) were randomized 1:1:1 to one of three fidelity conditions: self-report (n = 41), chart-stimulated recall (semistructured interviews with the chart available; n = 42), or behavioral rehearsal (simulated role-plays; n = 43). All participating clinicians completed fidelity assessments for up to three sessions with three different clients that were recruited from clinicians' caseloads (n = 288; M age = 13.39 years SD = 3.89; 41.7% female); sessions were also audio-recorded and coded for comparison to determine the most accurate method. All fidelity measures had parallel scales that yielded an adherence maximum score (i.e., the highest-rated intervention in a session), a mean of techniques observed, and a count total of observed techniques. Results of three-level mixed effects regression models indicated that behavioral rehearsal produced comparable scores to observation for all adherence scores (all ps > .01), indicating no difference between behavioral rehearsal and observation. Self-report and chart-stimulated recall overestimated adherence compared to observation (ps < .01). Overall, findings suggested that behavioral rehearsal indexed CBT adherence comparably to direct observation, the gold-standard, in pediatric populations. Behavioral rehearsal may at times be able to replace the need for resource-intensive direct observation in implementation research and practice.

Treatment Fidelity and Outcome in CBT for Youth with Autism: The MEYA Fidelity Scale.

OBJECTIVE: Assessing treatment fidelity in effectiveness research is critical to interpreting study findings. This paper details the development and initial psychometric evaluation of the Modular Evidence-Based Practices for Youth with Autism Fidelity Scale (MEYA-FS) designed to support the assessment of cognitive-behavioral treatments for youth with autism in effectiveness research. METHOD: Recorded treatment sessions (N = 338) were randomly selected from 77 youth (M age = 9.65 years, SD = 1.87; 50.67% White; 85.33% male) who received the Schema, Emotion, and Behavior-Focused Therapy for Children (SEBASTIEN) (n = 51) or Coping Cat (n = 24) program. RESULTS: The MEYA-FS Adherence items displayed acceptable interrater reliability, but more than half of the MEYA-FS Competence items did not. The magnitude and pattern of correlations supported the score validity of the MEYA-FS Adherence and Competence items and subscales. However, some corresponding Adherence and Competence items displayed significant overlap. Scores on each Adherence subscale distinguished between the SEBASTIEN and Coping Cat programs, providing support for discriminant validity. Finally, higher Adherence and Competence subscales predicted significant improvements in youth clinical outcomes (adjustment problems in the school setting, social-communication difficulties, restrictive/repetitive behaviors, and externalizing problems), providing initial evidence for predictive validity. CONCLUSIONS: The psychometric properties of the MEYA-FS make it appropriate for supporting efforts to evaluate cognitive-behavioral interventions for youth with autism in effectiveness and implementation research.

To Utility and Beyond! Specifying and Advancing the Utility of Measurement-Based Care for Youth.

Mental health organizations that serve youth are under pressure to adopt measurement-based care (MBC), defined as the continuous collection of client-report data used to support clinical decision-making as part of standard care. However, few frameworks exist to help leadership ascertain how to select an MBC approach for a clinical setting. This paper seeks to define how an MBC approach can display clinical utility to provide such a framework. Broadly, we define clinical utility as evidence that an MBC approach assists stakeholders in fulfilling clinical goals related to care quality (i.e., improve client-clinician alliance and clinical outcomes) at the client (i.e., youth and caregiver), clinician, supervisor, and administrator levels. More specifically, our definition of clinical utility is divided into two categories relevant to the usability and usefulness of an MBC approach for a specific setting: (a) implementability (i.e., evidence indicating ease of use in a clinical setting) and (b) usefulness in aiding clinical activities (i.e., evidence indicating the potential to improve communication and make clinical activities related to care quality easier or more effective). These categories provide valuable information about how easy an MBC approach is to use and the potential benefits that the MBC data will confer. To detail how we arrived at this definition, we review prior definitions of clinical utility, discuss how previous definitions inform our definition of clinical utility for MBC, and provide examples of how the concept of clinical utility can be applied to MBC. We finish with a discussion of future research directions.

Can a Single Measure Estimate Protocol Adherence for Two Psychosocial Treatments for Youth Anxiety Delivered in Community Mental Health Settings?

Treatment adherence measurement can be time and resource-intensive in clinical trials, so the ability to measure protocol adherence for two distinct treatment programs with a single measure may benefit the field. The present study sought to determine if the Therapy Process Observational Coding System - Revised Strategies Scale (TPOCS-RS) could assess protocol adherence to two youth treatment programs. Treatment sessions (N = 796) from 55 youth (M age = 9.89 years, SD = 1.71; range 7-15 years; 55.0% White; 46.0% female) with primary anxiety problems treatment by 39 clinicians (M age = 40.54 years, SD = 9.56; 50.0% White; 80.0% female) were independently scored by coders using observational treatment adherence and alliance measures. The youth received one of three treatments: (a) Standard (i.e., cognitive-behavioral treatment program), (b) Modular (i.e., a program with cognitive-behavioral and parent training components), or (c) Usual Care. Consultants filled out a self-report measure of protocol adherence within the Standard and Modular conditions. Interrater reliability, ICC(2,2) for the various items for the full sample ranged from .17 to .92 (M ICC = .67; SD = .17). Scores from a TPOCS-RS subscale that mapped onto the specific content of the treatment protocols used in the Standard and Modular conditions evidenced convergent validity with the consultant-report adherence measure and discriminant validity with the alliance measure. The model-specific TPOCS-RS subscales also discriminated between the Standard and Modular treatments and Usual Care. This study provides initial evidence that (a) the TPOCS-RS has utility in estimating protocol adherence in different treatment programs and (b) support the score validity of the self-report consultation records.

Stakeholder intention to engage in fidelity measurement methods in community mental health settings: A mixed methods study.

Background: The current gold standard for measuring fidelity (specifically, adherence) to cognitive behavioral therapy (CBT) is direct observation, a costly, resource-intensive practice that is not feasible for many community organizations to implement regularly. Recent research indicates that behavioral rehearsal (i.e., role-play between clinician and individual with regard to session delivery) and chart-stimulated recall (i.e., brief structured interview between clinician and individual about what they did in session; clinicians use the client chart to prompt memory) may provide accurate and affordable alternatives for measuring adherence to CBT in such settings, with behavioral rehearsal yielding greater correspondence with direct observation. Methods: Drawing on established causal theories from social psychology and leading implementation science frameworks, this study evaluates stakeholders' intention to use behavioral rehearsal and chart-stimulated recall. Specifically, we measured attitudes, self-efficacy, and subjective norms toward using each, and compared these factors across the two methods. We also examined the relationship between attitudes, self-efficacy, subjective norms, and intention to use each method. Finally, using an integrated approach we asked stakeholders to discuss their perception of contextual factors that may influence beliefs about using each method. These data were collected from community-based supervisors (n = 17) and clinicians (n = 66). Results: Quantitative analyses suggest moderately strong intention to use both methods across stakeholders. There were no differences in supervisors' or clinicians' attitudes, self-efficacy, subjective norms, or intention across methods. More positive attitudes and greater reported subjective norms were associated with greater reported intention to use either measure. Qualitative analyses identified participants' specific beliefs about using each fidelity measure in their organization, and results were organized using the Consolidated Framework for Implementation Research. Conclusions: Strategies are warranted to overcome or minimize potential barriers to using fidelity measurement methods and to further increase the strength of intention to use them.Plain Language Summary: The best way to measure fidelity, or how closely a clinician follows the protocol, to Cognitive Behavioral Therapy (CBT) is watching the session. This is an expensive practice that is not feasible for many community organizations to do regularly. Recent research indicates that behavioral rehearsal, or a role-play between the clinician and individual with regard to session delivery, and chart-stimulated recall, or a brief discussion between an individual and the clinician about what they did in session with the clinician having access to the chart to help them remember, may provide accurate and affordable alternatives for measuring fidelity to CBT. We just completed a study demonstrating that both methods are promising, with behavioral rehearsal offering scores that are the most similar to watching the session. Drawing on established theories from social psychology and leading implementation science frameworks, this study evaluates future supervisor and clinician motivation to use these fidelity measurement methods. Specifically, we measured supervisor (n = 17) and clinician (n = 66) attitudes, norms, self-efficacy, intentions, and anticipated barriers and facilitators to using each of these fidelity measurement tools. Quantitative and qualitative analyses suggest similar intention to use both methods, and concerns about barriers to using each method. Further research is warranted to minimize the burden associated with implementing fidelity measurement methods and deploying strategies to increase use.

Development and Initial Psychometrics of a Generic Treatment Integrity Measure Designed to Assess Practice Elements Targeting Social, Emotional, and Behavioral Outcomes in Early Childhood Settings.

Though treatment integrity measurement is important for research intended to promote social and behavioral outcomes of children at risk for emotional and behavioral disorders (EBDs) in early childhood settings, measurement gaps exist in the field. This paper reports on the development and preliminary psychometric assessment of the treatment integrity measure for early childhood settings (TIMECS), an observational measure designed to address existing measurement gaps related to treatment integrity with tier 2 interventions in the early childhood field. To assess the preliminary score reliability (interrater) and validity (construct, discriminant) of the TIMECS, live observations (N = 650) in early childhood classrooms from 54 teachers (92.6% female, 7.4% male; 61.1% White) and 91 children (M age = 4.53 years, SD = .44; 45.1% female, 54.9% male; 45.1% Black) at risk for EBDs were scored by 12 coders using the TIMECS and an observational measure designed to assess teacher-child interactions. Teachers also self-reported on the quality of the teacher-child relationship. Interrater reliability (intraclass correlation coefficients, ICC [2,2]) for the quantity (i.e., adherence) item scores had a mean of .81 (SD = .07; range from .68 to .95), and the quality (i.e., competence) item scores had a mean of .69 (SD = .08; range from .52 to .80). Scores on the TIMECS Quantity and Quality items and scales showed evidence of construct validity, with the magnitude of the correlations suggesting that the quantity and quality items assess distinct components of treatment integrity. A TIMECS quantity scale also showed promise for intervention evaluation research by discriminating between teachers who had and had not been trained in a specific evidence-based intervention targeting social and behavioral skills in early childhood. The findings support the potential of the TIMECS to assess treatment integrity of teacher-delivered practices designed to address child social and behavioral outcomes of children at risk for EBDs in early childhood settings.

Examining the Relation Between Technical and Global Competence in Two Treatments for Youth Anxiety.

Although technical (quality of delivering techniques from a specific treatment) and global (general clinical expertise) competence are believed to be important ingredients of successful psychosocial treatment with youth, there have been few empirical efforts to measure both dimensions. Efforts to understand the role that each competence dimension plays in the process and outcome of youth treatment starts with determining whether the dimensions can be measured separately. This study examined whether scores from measures designed to assess technical and global competence were distinct. Treatment sessions (N = 603) from 38 youths (M age = 9.84 years, SD = 1.65; 60.5% White; 52.6% male) treated for primary anxiety problems within a randomized effectiveness trial were coded. Four coders used observational measures designed to assess technical competence, global competence, protocol adherence, and the alliance. Mean item interrater reliability was .70 (SD = .09) for technical competence and .66 (SD = .05) for global competence. While most components of global competence were distinct from technical competence scores, two components showed redundancy (r > .70). Scores on both competence measures were empirically distinct (r < .70) from scores on measures of protocol adherence and the alliance. Although the measures did not fully distinguish between technical and global competence, our findings do indicate that some components of technical and global competence may provide unique information about competence.

Reliability, factor structure, and validity of an observer-rated alliance scale with youth.

Measurement limitations lessen the conclusions about the role of the alliance in youth psychosocial treatment. This article examined the score reliability, factor structure, and validity of the 9-item Therapy Process Observational Coding System for Child Psychotherapy-Alliance scale (TPOCS-A). The sample, 51 youth aged 7-15 years (Mage = 10.36 years, SD = 1.90; 86.3% White; 60.8% male, 39.2% female), met diagnostic criteria for a principal anxiety disorder and received cognitive-behavioral therapy. Treatment sessions (N = 463) were coded by independent coders using the TPOCS-A along with observational measures of treatment adherence and therapist competence. Youth and therapists also completed self-report alliance measures at the end of each session. Reliability estimates, ICC(2,2), at the item level indicated a mean interrater reliability of .68 (SD = .10) and a mean coder stability of .64 (SD = .11). An exploratory factor analysis identified a one-factor solution with five items. TPOCS-A scores evidenced convergent validity with the therapist and adolescent reports of alliance but did not converge with the child-report alliance measure. TPOCS-A scores evidenced discriminant validity when compared with scores on adherence and competence measures. The reliability and validity profile of the 9- and 5-item versions of the TPOCS-A were similar. Overall, findings support the reliability and validity of the TPOCS-A scores, but questions remain about how best to assess the alliance with children. (PsycInfo Database Record (c) 2021 APA, all rights reserved).