Influence of Classroom-Level Factors on Implementation Fidelity During Scale-up of Evidence-Based Interventions.

As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementation (FOI) often wanes. This study explores the association between FOI and malleable variables within classrooms that could be targeted to optimize resources without compromising FOI as school-based EBIs are disseminated across real-world settings. We utilized process evaluation data from a national dissemination project of the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention shown to reduce substance use. The sample included 1,626 teachers in 371 schools across 14 states. Hierarchical linear models examined the relationship between observational measures of implementation factors and three domains of fidelity (e.g., adherence, student responsiveness, and quality of delivery). Findings suggest that curriculum modifications, student misbehavior, and shortage of time to implement the LST middle school program were factors most associated with lower FOI. Class size, access to program materials, and whether LST was delivered in a traditional classroom setting that is well-suited for instruction (versus in a less structured environment such as the school cafeteria) are less predictive. In scale-up of classroom-based universal interventions targeting behavioral health outcomes, our findings indicate that carefully vetting modifications, supporting classroom management strategies, and ensuring sufficient class time for implementation of highly interactive EBIs such as LST are important considerations. Since changes to EBIs are inevitable, efforts are needed to guide facilitators in making adjustments that improve program fit without compromising the essential intervention activities deemed necessary to produce desired outcomes.

Partnering for Success: Factors Impacting Implementation of a Cross-Systems Collaborative model Between Behavioral Health and Child Welfare.

Cross-system implementation efforts can support needed mental health (MH) service utilization among children involved in the child welfare (CW) system. The Partnering for Success (PfS) initiative is one such effort that promotes greater collaboration between the CW and MH providers by building capacity within and across each system. Frontline CW providers learn to accurately identify child MH treatment targets, link families to locally-provided evidence-based treatments (EBTs), and monitor treatment progress. Concurrently, local MH providers are trained along with CW workers to utilize Cognitive Behavioral Therapy plus Trauma-Focused CBT (CBT +), a common elements training and consultation approach focusing on typical MH issues for CW-involved children: Anxiety, Depression, Behavioral Problems, and Traumatic Stress. Finally, agency leadership receive support around promoting implementation and sustainment. This paper examines factors identified by participating CW and MH staff which impacted PfS implementation. Twenty-nine frontline, supervisory, and executive CW and MH providers were interviewed via audio-recorded web-based calls in six focus groups and 10 individual interviews. Factors facilitating implementation success included training/consultation, support from supervisors and agency leadership, improved referral processes, high quality relationships and communication between CW and MH frontline staff, PfS tools and resources, opportunities to use PfS, as well as buy-in from providers and families. Implementation barriers included poor communication between CW and MH providers, conflicts over role expectations, workload and turnover challenges, lack of buy-in, as well as provider (e.g., not aligned with CBT +) and client characteristics (e.g., frequent crises).

Measurement Based Care in Child Welfare-Involved Children and Youth: Reliability and Validity of the PSC-17.

This study evaluates whether the psychometric properties of the Pediatric Symptoms Checklist-17 (PSC-17), a common behavioral health measure typically used as a dichotomous screening tool for mental health needs, support its use as a continuous measure for tracking behavioral health over time. A total of 6492 foster parents of children and youth aged 5.5-17 completed the PSC-17. Convergent and discriminant validity was assessed by comparing raw PSC-17 subscale scores with associated outcomes (e.g. psychiatric diagnoses). Long-term test-retest reliability was assessed over 6 months. Scores on the PSC-17 demonstrated good convergent and divergent validity. PSC-17 subscale scores were most strongly associated with analogous diagnoses. Test-retest reliability was moderate, as expected for a time window of this length. This study provides moderate support for the psychometric qualities of the PSC-17 when used with children and youth in the child welfare system as a continuous measure of psychosocial functioning over time.

Identifying Psychosocial Problems Among Children and Youth in the Child Welfare System Using the PSC-17: Exploring Convergent and Discriminant Validity with Multiple Informants.

Youth who enter foster care are at risk of mental health need, but questions arise as to the validity of their self-reported symptomatology. This study examines the screening validity of the youth-report version of the Pediatric Symptom Checklist-17 (PSC-17) in a child welfare population. Data come from 2389 youth who completed a version of the PSC-17 adapted for youth report, and their biological and foster parents who completed the parent-report version. Youth also completed a shortened version of the Screen for Child Anxiety Related Disorders (SCARED). Convergent and discriminant validity of the PSC-17 was assessed using multi-trait multi-method matrices. The PSC-17's internalizing subscale was strongly correlated, attention subscale was moderately correlated, and externalizing subscale was weakly correlated with the SCARED's anxiety and PTSD subscales. Comparing youth and foster parent scores, the PSC-17 had moderate convergent validity and weak/fair discriminant validity. Comparing youth, foster parent, and biological parent scores, the PSC-17 had moderate convergent validity and weak/fair discriminant validity. The current study provides some support for the validity of the PSC-17 for the population of youth in foster care.

Agreement and Discrepancy Between Supervisor and Clinician Alliance: Associations with Clinicians' Perceptions of Psychological Climate and Emotional Exhaustion.

Despite increasing interest in supervision as a leverage point for bolstering public mental health services, the potential influence of supervisory alliance on organizations and direct service providers remains understudied, particularly in the context of supporting evidence-based treatment (EBT) use. This study examined agreement and discrepancy between supervisor and clinician ratings of alliance associated with clinicians' perceptions of psychological climate and emotional exhaustion. Results indicated that discrepancies in alliance ratings were common and associated with clinicians' perceptions of psychological climate. These findings have important implications for collaboration among supervisors and clinicians within a community mental health organizational context and the provision of EBTs.

Evidence Base Update for Psychosocial Treatments for Children and Adolescents Exposed to Traumatic Events.

Child and adolescent trauma exposure is prevalent, with trauma exposure-related symptoms, including posttraumatic stress, depressive, and anxiety symptoms often causing substantial impairment. This article updates the evidence base on psychosocial treatments for child and adolescent trauma exposure completed for this journal by Silverman et al. (2008). For this review, we focus on 37 studies conducted during the seven years since the last review. Treatments are grouped by overall treatment family (e.g., cognitive behavioral therapy), treatment modality (e.g., individual vs. group), and treatment participants (e.g., child only vs. child and parent). All studies were evaluated for methodological rigor according to Journal of Clinical Child & Adolescent Psychology evidence-based treatment evaluation criteria (Southam-Gerow & Prinstein, 2014), with cumulative designations for level of support for each treatment family. Individual CBT with parent involvement, individual CBT, and group CBT were deemed well-established; group CBT with parent involvement and eye movement desensitization and reprocessing (EMDR) were deemed probably efficacious; individual integrated therapy for complex trauma and group mind-body skills were deemed possibly efficacious; individual client-centered play therapy, individual mind-body skills, and individual psychoanalysis were deemed experimental; and group creative expressive + CBT was deemed questionable efficacy. Advances in the evidence base, with comparisons to the state of the science at the time of the Silverman et al. (2008) review, are discussed. Finally, we present dissemination and implementation challenges and areas for future research.

Developing a Quality Assurance System for Multiple Evidence Based Practices in a Statewide Service Improvement Initiative.

Efforts to implement evidence based practices (EBP) are increasingly common in child-serving systems. However, public systems undertaking comprehensive improvement efforts that aim to increase availability of multiple practices at the same time may struggle to build comprehensive and user-friendly strategies to develop the workforce and encourage adoption, faithful implementation, and sustainability of selected EBPs. Given that research shows model adherence predicts positive outcomes, one critical EBP implementation support is systematic quality, fidelity, and compliance monitoring. This paper describes the development and initial implementation of a quality assurance framework for a statewide EBP initiative within child welfare. This initiative aimed to improve provider practice and monitor provider competence and compliance across four different EBPs, and to inform funding and policy decisions. The paper presents preliminary data as an illustration of lessons learned during the quality monitoring process and concludes with a discussion of the promise and challenges of developing and applying a multi-EBP quality assurance framework for use in public systems.

The Juggling Act of Supervision in Community Mental Health: Implications for Supporting Evidence-Based Treatment.

Supervisors are an underutilized resource for supporting evidence-based treatments (EBTs) in community mental health. Little is known about how EBT-trained supervisors use supervision time. Primary aims were to describe supervision (e.g., modality, frequency), examine functions of individual supervision, and examine factors associated with time allocation to supervision functions. Results from 56 supervisors and 207 clinicians from 25 organizations indicate high prevalence of individual supervision, often alongside group and informal supervision. Individual supervision serves a wide range of functions, with substantial variation at the supervisor-level. Implementation climate was the strongest predictor of time allocation to clinical and EBT-relevant functions.

An Equity-Focused Assessment of Evidence-Based Parenting Intervention Research.

Evidence-based parenting interventions (EBPI) support children and families to promote resilience, address emotional and behavioral concerns, and prevent or address issues related to child maltreatment. Critiques of EBPIs include concerns about their relevance and effectiveness for diverse populations when they are implemented at population scale. Research methods that center racial equity and include community-based participatory approaches have the potential to address some of these concerns. The purpose of the present review was to document the extent to which methods associated with promoting racial equity in research have been used in studies that contribute to the evidence base for programs that meet evidentiary standards for a clearinghouse that was developed to support the Family First Prevention Services Act in the United States. We developed a coding system largely based on the Culturally Responsive Evaluation model. A sample of 47 papers that are part of the evidence base for ten in-home parent skill-based programs were reviewed and coded. Only three of 28 possible codes were observed to occur in over half of the studies (including race/ethnicity demographic characteristics, conducting measure reliability for the study sample, and including information on socioeconomic status). Although the overall presence of equity-informed methods was low, a positive trend was observed over time. This review highlights ways in which rigorous research can incorporate racial equity into the planning, design, execution, and interpretation and dissemination of programs of study. We posit that doing so improves the external validity of studies while maintaining high-quality research that can contribute to an evidence base.

Developing the mental health workforce: review and application of training approaches from multiple disciplines.

Strategies specifically designed to facilitate the training of mental health practitioners in evidence-based practices (EBPs) have lagged behind the development of the interventions themselves. The current paper draws from an interdisciplinary literature (including medical training, adult education, and teacher training) to identify useful training and support approaches as well as important conceptual frameworks that may be applied to training in mental health. Theory and research findings are reviewed, which highlight the importance of continued consultation/support following training workshops, congruence between the training content and practitioner experience, and focus on motivational issues. In addition, six individual approaches are presented with careful attention to their empirical foundations and potential applications. Common techniques are highlighted and applications and future directions for mental health workforce training and research are discussed.

Translating economic evaluations into financing strategies for implementing evidence-based practices.

BACKGROUND: Implementation researchers are increasingly using economic evaluation to explore the benefits produced by implementing evidence-based practices (EBPs) in healthcare settings. However, the findings of typical economic evaluations (e.g., based on clinical trials) are rarely sufficient to inform decisions about how health service organizations and policymakers should finance investments in EBPs. This paper describes how economic evaluations can be translated into policy and practice through complementary research on financing strategies that support EBP implementation and sustainment. MAIN BODY: We provide an overview of EBP implementation financing, which outlines key financing and health service delivery system stakeholders and their points of decision-making. We then illustrate how economic evaluations have informed decisions about EBP implementation and sustainment with three case examples: (1) use of Pay-for-Success financing to implement multisystemic therapy in underserved areas of Colorado, USA, based in part on the strength of evidence from economic evaluations; (2) an alternative payment model to sustain evidence-based oncology care, developed by the US Centers for Medicare and Medicaid Services through simulations of economic impact; and (3) use of a recently developed fiscal mapping process to collaboratively match financing strategies and needs during a pragmatic clinical trial for a newly adapted family support intervention for opioid use disorder. CONCLUSIONS: EBP financing strategies can help overcome cost-related barriers to implementing and sustaining EBPs by translating economic evaluation results into policy and practice. We present a research agenda to advance understanding of financing strategies in five key areas raised by our case examples: (1) maximize the relevance of economic evaluations for real-world EBP implementation; (2) study ongoing changes in financing systems as part of economic evaluations; (3) identify the conditions under which a given financing strategy is most beneficial; (4) explore the use and impacts of financing strategies across pre-implementation, active implementation, and sustainment phases; and (5) advance research efforts through strong partnerships with stakeholder groups while attending to issues of power imbalance and transparency. Attention to these research areas will develop a robust body of scholarship around EBP financing strategies and, ultimately, enable greater public health impacts of EBPs.

Objective coding of content and techniques in workplace-based supervision of an EBT in public mental health.

BACKGROUND: Workplace-based clinical supervision as an implementation strategy to support evidence-based treatment (EBT) in public mental health has received limited research attention. A commonly provided infrastructure support, it may offer a relatively cost-neutral implementation strategy for organizations. However, research has not objectively examined workplace-based supervision of EBT and specifically how it might differ from EBT supervision provided in efficacy and effectiveness trials. METHODS: Data come from a descriptive study of supervision in the context of a state-funded EBT implementation effort. Verbal interactions from audio recordings of 438 supervision sessions between 28 supervisors and 70 clinicians from 17 public mental health organizations (in 23 offices) were objectively coded for presence and intensity coverage of 29 supervision strategies (16 content and 13 technique items), duration, and temporal focus. Random effects mixed models estimated proportion of variance in content and techniques attributable to the supervisor and clinician levels. RESULTS: Interrater reliability among coders was excellent. EBT cases averaged 12.4 min of supervision per session. Intensity of coverage for EBT content varied, with some discussed frequently at medium or high intensity (exposure) and others infrequently discussed or discussed only at low intensity (behavior management; assigning/reviewing client homework). Other than fidelity assessment, supervision techniques common in treatment trials (e.g., reviewing actual practice, behavioral rehearsal) were used rarely or primarily at low intensity. In general, EBT content clustered more at the clinician level; different techniques clustered at either the clinician or supervisor level. CONCLUSIONS: Workplace-based clinical supervision may be a feasible implementation strategy for supporting EBT implementation, yet it differs from supervision in treatment trials. Time allotted per case is limited, compressing time for EBT coverage. Techniques that involve observation of clinician skills are rarely used. Workplace-based supervision content appears to be tailored to individual clinicians and driven to some degree by the individual supervisor. Our findings point to areas for intervention to enhance the potential of workplace-based supervision for implementation effectiveness. TRIAL REGISTRATION: NCT01800266 , Clinical Trials, Retrospectively Registered (for this descriptive study; registration prior to any intervention [part of phase II RCT, this manuscript is only phase I descriptive results]).

Economic Impact of the Statewide Implementation of an Evidence-Based Treatment: Multisystemic Therapy in New Mexico.

Several states have made considerable investments into large-scale implementation of evidence-based treatments (EBTs), yet little is known about key success indicators for these implementation efforts such as cost and sustainability. To that end, the present study examined the economic impact of statewide implementation of multisystemic therapy (MST; Henggeler, Schoenwald, Borduin, Rowland, & Cuningham, 2009), a family- and community-based behavioral EBT for serious juvenile offenders in New Mexico. Participants were 1,869 youth who received MST across 23 teams during the study period. We conducted a cost-benefit analysis using metrics from state data sources that compared the cost of MST to its benefits (i.e., avoided expenses from pre- to posttreatment) in two domains: (a) behavioral health services (i.e., Medicaid claims) and (b) juvenile crime (i.e., taxpayer expenses, tangible and intangible expenses to crime victims). MST costs were based on Medicaid claims, which were reimbursed at an enhanced billing rate that was intended to cover expenses for both clinical and implementation (e.g., training, quality assurance) activities. Results suggest that implementation of MST in New Mexico over the 7-year study period may have produced net benefits, through 2 years posttreatment, of more than $4,643 per youth in avoided behavioral health claims and $15,019 per youth through reductions in juvenile crime. Stated differently, every dollar that New Mexico spent on MST appeared to have returned $3.34 for a total benefit of $64.2 million over the course of the study. We discuss implications of these findings for policymakers, administrators, and researchers who are interested in increasing the sustainability of complex EBTs in community settings.

Development and Implementation of a Child Welfare Workforce Strategy to Build a Trauma-Informed System of Support for Foster Care.

Effective strategies that increase the extent to which child welfare professionals engage in trauma-informed case planning are needed. This study evaluated two approaches to increase trauma symptom identification and use of screening results to inform case planning. The first study evaluated the impact of training on trauma-informed screening tools for 44 child welfare professionals who screen all children upon placement into foster care. The second study evaluated a two-stage approach to training child welfare workers on case planning for children's mental health. Participants included (a) 71 newly hired child welfare professionals who received a 3-hr training and (b) 55 child welfare professionals who participated in a full-day training. Results from the first study indicate that training effectively increased knowledge and skills in administering screening tools, though there was variability in comfort with screening. In the second study, participants self-reported significant gains in their competency in identifying mental health needs (including traumatic stress) and linking children with evidence-based services. These findings provide preliminary evidence for the viability of this approach to increase the extent to which child welfare professionals are trauma informed, aware of symptoms, and able to link children and youth with effective services designed to meet their specific needs.

Research, Data, and Evidence-Based Treatment Use in State Behavioral Health Systems, 2001-2012.

OBJECTIVE: Empirical study of public behavioral health systems' use of data and their investment in evidence-based treatments (EBTs) is limited. This study describes trends in state-level EBT investment and research supports from 2001 to 2012. METHODS: Data were from National Association for State Mental Health Program Directors Research Institute (NRI) surveys, which were completed by representatives of state mental health authorities (SMHAs). Multilevel models examined change over time related to state adoption of EBTs, numbers of clients served, and penetration rates for six behavioral health EBTs for adults and children: supported housing, supported employment, assertive community treatment, therapeutic foster care, multisystemic therapy, and functional family therapy. State supports related to research, evaluation, and information management were also examined. RESULTS: Increasing percentages of states reported funding an external research center, promoting the adoption of EBTs through provider contracts, and providing financial incentives for EBTs. Decreasing percentages of states reported promoting EBT adoption through stakeholder mobilization, monitoring fidelity, and specific budget requests. There was greater reported use of adult-focused EBTs (65%-80%) compared with youth-focused EBTs (25%-50%). Overall penetration rates of EBTs were low (1%-3%) and EBT adoption by states showed flat or declining trends. SMHAs' investment in data systems and use of research showed little change. CONCLUSIONS: SMHA investment in EBTs, implementation infrastructure, and use of research has declined. More systematic measurement and examination of these metrics may provide a useful approach for setting priorities, evaluating success of health reform efforts, and making future investments.

Child welfare caseworkers as brokers of mental health services: a pilot evaluation of Project Focus Colorado.

Youth in the child welfare system (CWS) have substantially higher rates of mental health needs compared to the general population, yet they rarely receive targeted, evidence-based practices (EBPs). Caseworkers play the critically important role of "service broker" for CWS youth and families. This study examines preliminary caseworker-level outcomes of Project Focus Colorado (PF-C), a training and consultation program designed to improve access to EBPs for CWS youth. PF-C evaluation occurred in four child welfare offices (two intervention [n = 16 caseworkers] vs. two practice-as-usual, wait-list control [WLC; n = 12 caseworkers]). Receipt of PF-C was associated with significantly increased caseworker knowledge of (a) EBPs, (b) child mental health problems, (c) evidence-based treatment components targeting mental health problem areas, and (d) mental health screening instruments, compared to WLC. Dose of training and consultation was associated with greater ability to correctly classify mental health problems and match them to EBPs. These preliminary results suggest that targeted training and consultation help to improve caseworker knowledge of children's mental health needs, EBPs for mental health, and mental health screening instruments.