Disparities in Youth and Family Experiences of System-of-Care Principles by Level of Youth Need.

The extent to which mental health services for youths embody system-of-care (SOC) principles is an important quality indicator. This study tested whether youth and family experiences of SOC principles varied depending on youths' level of need after adjusting for sociodemographic and treatment factors. The relationship to caregiver-reported clinical outcomes was also examined. Using administrative data and cross-sectional surveys from a stratified random sample of 1124 caregivers of youths ages 5-20 within a statewide system, adjusted analyses indicated caregivers of youths with the most intensive needs were significantly less likely to report receiving care that embodied SOC principles, with deficits on six of nine items. Youths whose services embodied SOC principles experienced significantly greater improvement in caregiver-reported functioning even after adjusting for level of need. Results highlight disparities in SOC principles for youths with intensive needs and the need for policy and intervention development to improve care for this population.

In Search of the Common Elements of Clinical Supervision: A Systematic Review.

The importance of clinical supervision for supporting effective implementation of evidence-based treatments (EBTs) is widely accepted; however, very little is known about which supervision practice elements contribute to implementation effectiveness. This systematic review aimed to generate a taxonomy of empirically-supported supervision practice elements that have been used in treatment trials and shown to independently predict improved EBT implementation. Supervision practice elements were identified using a two-phase, empirically-validated distillation process. In Phase I, a systematic review identified supervision protocols that had evidence of effectiveness based on (a) inclusion in one or more EBT trials, and (b) independent association with improved EBT implementation in one or more secondary studies. In Phase II, a hybrid deductive-inductive coding process was applied to the supervision protocols to characterize the nature and frequency of supervision practice elements across EBTs. Twenty-one of the 876 identified articles assessed the associations of supervision protocols with implementation or clinical outcomes, representing 13 separate studies. Coding and distillation of the supervision protocols resulted in a taxonomy of 21 supervision practice elements. The most frequently used elements were: reviewing supervisees' practice (92%; n = 12), clinical suggestions (85%; n = 11), behavioral rehearsal (77%; n = 10), elicitation (77%; n = 10), and fidelity assessment (77%; n = 10). This review identified supervision practice elements that could be targets for future research testing which elements are necessary and sufficient to support effective EBT implementation. Discrepancies between supervision practice elements observed in trials as compared to routine practice highlights the importance of research addressing supervision-focused implementation strategies.

Organization-level variation in therapists' attitudes toward and use of measurement-based care.

PURPOSE: Despite significant interest in improving behavioral health therapists' implementation of measurement-based care (MBC)-and widespread acknowledgment of the potential importance of organization-level determinants-little is known about the extent to which therapists' use of, and attitudes toward, MBC vary across and within provider organizations or the multilevel factors that predict this variation. METHODS: Data were collected from 177 therapists delivering psychotherapy to youth in 21 specialty outpatient clinics in the USA. Primary outcomes were use of MBC for progress monitoring and treatment modification, measured by the nationally-normed Current Assessment of Practice Evaluation-Revised. Secondary outcomes were therapist attitudes towards MBC. Linear multilevel regression models tested the association of theory-informed clinic and therapist characteristics with these outcomes. RESULTS: Use of MBC varied significantly across clinics, with means on progress monitoring ranging from values at the 25th to 93rd percentiles and means on treatment modification ranging from the 18th to 71st percentiles. At the clinic level, the most robust predictor of both outcomes was clinic climate for evidence-based practice implementation; at the therapist level, the most robust predictors were: attitudes regarding practicality, exposure to MBC in graduate training, and prior experience with MBC. Attitudes were most consistently related to clinic climate for evidence-based practice implementation, exposure to MBC in graduate training, and prior experience with MBC. CONCLUSIONS: There is important variation in therapists' attitudes toward and use of MBC across clinics. Implementation strategies that target clinic climate for evidence-based practice implementation, graduate training, and practicality may enhance MBC implementation in behavioral health.

Community stakeholder preferences for evidence-based practice implementation strategies in behavioral health: a best-worst scaling choice experiment.

BACKGROUND: Community behavioral health clinicians, supervisors, and administrators play an essential role in implementing new psychosocial evidence-based practices (EBP) for patients receiving psychiatric care; however, little is known about these stakeholders' values and preferences for implementation strategies that support EBP use, nor how best to elicit, quantify, or segment their preferences. This study sought to quantify these stakeholders' preferences for implementation strategies and to identify segments of stakeholders with distinct preferences using a rigorous choice experiment method called best-worst scaling. METHODS: A total of 240 clinicians, 74 clinical supervisors, and 29 administrators employed within clinics delivering publicly-funded behavioral health services in a large metropolitan behavioral health system participated in a best-worst scaling choice experiment. Participants evaluated 14 implementation strategies developed through extensive elicitation and pilot work within the target system. Preference weights were generated for each strategy using hierarchical Bayesian estimation. Latent class analysis identified segments of stakeholders with unique preference profiles. RESULTS: On average, stakeholders preferred two strategies significantly more than all others-compensation for use of EBP per session and compensation for preparation time to use the EBP (P < .05); two strategies were preferred significantly less than all others-performance feedback via email and performance feedback via leaderboard (P < .05). However, latent class analysis identified four distinct segments of stakeholders with unique preferences: Segment 1 (n = 121, 35%) strongly preferred financial incentives over all other approaches and included more administrators; Segment 2 (n = 80, 23%) preferred technology-based strategies and was younger, on average; Segment 3 (n = 52, 15%) preferred an improved waiting room to enhance client readiness, strongly disliked any type of clinical consultation, and had the lowest participation in local EBP training initiatives; Segment 4 (n = 90, 26%) strongly preferred clinical consultation strategies and included more clinicians in substance use clinics. CONCLUSIONS: The presence of four heterogeneous subpopulations within this large group of clinicians, supervisors, and administrators suggests optimal implementation may be achieved through targeted strategies derived via elicitation of stakeholder preferences. Best-worst scaling is a feasible and rigorous method for eliciting stakeholders' implementation preferences and identifying subpopulations with unique preferences in behavioral health settings.

Applying NUDGE to Inform Design of EBP Implementation Strategies in Community Mental Health Settings.

We demonstrate the application of NUDGE (Narrow, Understand, Discover, Generate, Evaluate), a behavioral economics approach to systematically identifying behavioral barriers that impede behavior enactment, to the challenge of evidence-based practice (EBP) use in community behavioral health. Drawing on 65 clinician responses to a system-wide crowdsourcing challenge about EBP underutilization, we applied NUDGE to discover, synthesize and validate specific behavioral barriers to EBP utilization that directly inform the design of tailored implementation strategies. To our knowledge, this is the first study to apply behavioral economic insights to clinician-proposed solutions to implementation challenges in order to design implementation strategies. The study demonstrates the successful application of NUDGE to implementation strategy design and provides novel targets for intervention.

The Organizational Financial Context of Publicly-Funded Mental Health Clinics: Development and Preliminary Psychometric Evaluation of the Agency Financial Status Scales.

Funding is a major barrier to implementation of evidence-based practices (EBPs) in publicly-funded community mental health clinics (CMHCs). Understanding how best to deploy implementation strategies that address this barrier requires greater clarity on the financial context within agencies. We developed the Agency Financial Status Scales (AFSS) to assess employee perceptions of the level of three hypothesized and theoretical funding related constructs in organizations: (a) perceptions of financial health, (b) financial attitudes toward EBPs, and (c) strategic financial climate. This investigation serves as a preliminary evaluation of this measure. Participants were 239 therapists and 40 supervisors from 25 publicly-funded CMHCs providing outpatient mental health services for young people. Confirmatory factor analysis was used to investigate the latent trait structure of the items. Internal consistency, interrater agreement, concordance between therapists and supervisors, and convergent validity were also examined. A two-factor model measuring perceptions of financial health and strategic financial climate best fit the data. For both of these scales, alpha reliability was acceptable and agreement statistics provided moderate support for aggregation at the organizational level. Analyses supported the convergent validity of the scales. The development and preliminary evaluation of the AFSS is an important first step in understanding the financial context of publicly-funded CMHCs. Though findings from this investigation are promising, additional development and testing are needed to develop a more thorough understanding of the constructs and to improve the validity and reliability of this measure.

Identifying the organizational innovation-specific capacity needed for exposure therapy.

BACKGROUND: Current approaches to increasing the rates of clinician use of exposure therapy for anxiety disorders in community settings are limited. Research underscores the importance of addressing contextual variables to facilitate clinician use of evidence-based practices; however, no studies have identified the innovation-specific organizational capacity necessary to implement exposure therapy. Such work is critical to ensure that treatment-seeking individuals with anxiety receive effective care. METHODS: We used a two-step process to identify the innovation-specific organizational capacity necessary to deliver exposure. First, 24 leaders of specialty anxiety clinics in the United States (50% female, mean [M]age = 47.7 years) completed a survey about the organizational innovation-specific capacity (e.g., policies and procedures) they employ to support their providers in delivering exposure therapy. Second, 19 community clinicians (79% female, M age = 42.9 years) reported on the extent to which these characteristics were present in their settings. RESULTS: In Step 1, specialty clinic leaders unanimously endorsed six organizational characteristics as essential and five as important within the areas of organizational policies, supervisory support, and peer clinician support. These characteristics were present in more than 90% of specialty clinics. In Step 2, therapists in community clinics reported these characteristics were minimally present in their organizations. CONCLUSIONS: Specialty clinic leaders exhibited consensus on the innovation-specific organizational capacity necessary to implement exposure therapy. Identified characteristics were largely absent from community clinics. Developing fiscal, policy, or organizational strategies that enhance the organizational capacity within community settings may improve the patients' access to effective treatment for anxiety disorders.

Annual Research Review: The state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field.

BACKGROUND: Scientists have developed evidence-based interventions that improve the symptoms and functioning of youth with psychiatric disorders; however, these interventions are rarely used in community settings. Eliminating this research-to-practice gap is the purview of implementation science, the discipline devoted to the study of methods to promote the use of evidence-based practices in routine care. METHODS: We review studies that have tested factors associated with implementation in child psychology and psychiatry, explore applications of social science theories to implementation, and conclude with recommendations to advance implementation science through the development and testing of novel, multilevel, causal theories. RESULTS: During its brief history, implementation science in child psychology and psychiatry has documented the implementation gap in routine care, tested training approaches and found them to be insufficient for behavior change, explored the relationships between variables and implementation outcomes, and initiated randomized controlled trials to test implementation strategies. This research has identified targets related to implementation (e.g., clinician motivation, organizational culture) and demonstrated the feasibility of activating these targets through implementation strategies. However, the dominant methodological approach has been atheoretical and predictive, relying heavily on a set of variables from heuristic frameworks. CONCLUSIONS: Optimizing the implementation of effective treatments in community care for youth with psychiatric disorders is a defining challenge of our time. This review proposes a new direction focused on developing and testing integrated causal theories. We recommend implementation scientists: (a) move from observational studies of implementation barriers and facilitators to trials that include causal theory; (b) identify a core set of implementation determinants; (c) conduct trials of implementation strategies with clear targets, mechanisms, and outcomes; (d) ensure that behaviors that are core to EBPs are clearly defined; and (e) agree upon standard measures. This agenda will help fulfill the promise of evidence-based practice for improving youth behavioral health.

Interactions Between Clinician and Organizational Characteristics to Predict Cognitive-Behavioral and Psychodynamic Therapy Use.

Conceptual models of implementation posit contextual factors and their associations with evidence-based practice (EBP) use at multiple levels and suggest these factors exhibit complex cross-level interactions. Little empirical work has examined these interactions, which is critical to advancing causal implementation theory and optimizing implementation strategy design. Mixed effects regression examined cross-level interactions between clinician (knowledge, attitudes) and organizational characteristics (culture, climate) to predict cognitive-behavioral and psychodynamic therapy use with youth (N = 247 clinicians across 28 agencies). Results indicated several interactions, highlighting the importance of attending to interactions between variables at multiple levels to advance multilevel implementation theory and strategies.

Therapist Financial Strain and Turnover: Interactions with System-Level Implementation of Evidence-Based Practices.

Therapist turnover is a major problem in community mental health. Financial strain, which is composed of cognitive, emotional, and behavioral responses to the experience of economic hardship, is an understudied antecedent of therapist turnover given the tumultuous financial environment in community mental health. We prospectively examined the relationship between therapist financial strain and turnover in 247 therapists in 28 community mental health agencies. We expected greater therapist financial strain to predict higher turnover and participation in a system-funded evidence-based practice (EBP) training initiative to alleviate this effect. Controlling for covariates, financial strain predicted therapist turnover (OR 1.12, p = .045), but not for therapists who participated in an EBP training initiative. Reducing financial strain and/or promoting EBP implementation may be levers to reduce turnover.

Concordance Between Administrator and Clinician Ratings of Organizational Culture and Climate.

Organizational culture and climate are important determinants of behavioral health service delivery for youth. The Organizational Social Context measure is a well validated assessment of organizational culture and climate that has been developed and extensively used in public sector behavioral health service settings. The degree of concordance between administrators and clinicians in their reports of organizational culture and climate may have implications for research design, inferences, and organizational intervention. However, the extent to which administrators' and clinicians' reports demonstrate concordance is just beginning to garner attention in public behavioral health settings in the United States. We investigated the concordance between 73 administrators (i.e., supervisors, clinical directors, and executive directors) and 247 clinicians in 28 child-serving programs in a public behavioral health system. Findings suggest that administrators, compared to clinicians, reported more positive cultures and climates. Organizational size moderated this relationship such that administrators in small programs (<466 youth clients served annually) provided more congruent reports of culture and climate in contrast to administrators in large programs (≥466 youth clients served annually) who reported more positive cultures and climates than clinicians. We propose a research agenda that examines the effect of concordance between administrators and clinicians on organizational outcomes in public behavioral health service settings.

Mechanisms of Change in the ARC Organizational Strategy: Increasing Mental Health Clinicians' EBP Adoption Through Improved Organizational Culture and Capacity.

The development of efficient and scalable implementation strategies in mental health is restricted by poor understanding of the change mechanisms that increase clinicians' evidence-based practice (EBP) adoption. This study tests the cross-level change mechanisms that link an empirically-supported organizational strategy for supporting implementation (labeled ARC for Availability, Responsiveness, and Continuity) to mental health clinicians' EBP adoption and use. Four hundred seventy-five mental health clinicians in 14 children's mental health agencies were randomly assigned to the ARC intervention or a control condition. Measures of organizational culture, clinicians' intentions to adopt EBPs, and job-related EBP barriers were collected before, during, and upon completion of the three-year ARC intervention. EBP adoption and use were assessed at 12-month follow-up. Multilevel mediation analyses tested changes in organizational culture, clinicians' intentions to adopt EBPs, and job-related EBP barriers as linking mechanisms explaining the effects of ARC on clinicians' EBP adoption and use. ARC increased clinicians' EBP adoption (OR = 3.19, p = .003) and use (81 vs. 56 %, d = .79, p = .003) at 12-month follow-up. These effects were mediated by improvement in organizational proficiency culture leading to increased clinician intentions to adopt EBPs and by reduced job-related EBP barriers. A combined mediation analysis indicated the organizational culture-EBP intentions mechanism was the primary carrier of ARC's effects on clinicians' EBP adoption and use. ARC increases clinicians' EBP adoption and use by creating proficient organizational cultures that increase clinicians' intentions to adopt EBPs.

Multilevel Mechanisms of Implementation Strategies in Mental Health: Integrating Theory, Research, and Practice.

A step toward the development of optimally effective, efficient, and feasible implementation strategies that increase evidence-based treatment integration in mental health services involves identification of the multilevel mechanisms through which these strategies influence implementation outcomes. This article (a) provides an orientation to, and rationale for, consideration of multilevel mediating mechanisms in implementation trials, and (b) systematically reviews randomized controlled trials that examined mediators of implementation strategies in mental health. Nine trials were located. Mediation-related methodological deficiencies were prevalent and no trials supported a hypothesized mediator. The most common reason was failure to engage the mediation target. Discussion focuses on directions to accelerate implementation strategy development in mental health.

Assessing and changing organizational social contexts for effective mental health services.

Culture and climate are critical dimensions of a mental health service organization's social context that affect the quality and outcomes of the services it provides and the implementation of innovations such as evidence-based treatments (EBTs). We describe a measure of culture and climate labeled Organizational Social Context (OSC), which has been associated with innovation, service quality, and outcomes in national samples and randomized controlled trials (RCTs) of mental health and social service organizations. The article also describes an empirically supported organizational intervention model labeled Availability, Responsiveness, and Continuity (ARC), which has improved organizational social context, innovation, and effectiveness in five RCTs. Finally, the article outlines a research agenda for developing more efficient and scalable organizational strategies to improve mental health services by identifying the mechanisms that link organizational interventions and social context to individual-level service provider intentions and behaviors associated with innovation and effectiveness.

The organizational social context of mental health medicaid waiver programs with family support services: implications for research and practice.

Peer family support specialists (FSS) are parents with practical experience in navigating children's mental health care systems who provide support, advocacy, and guidance to the families of children who need mental health services. Their experience and training differ from those of formally trained mental health clinicians, creating potential conflicts in priorities and values between FSS and clinicians. We hypothesized that these differences could negatively affect the organizational cultures and climates of mental health clinics that employ both FSS and mental health clinicians, and lower the job satisfaction and organizational commitment of FSS. The Organizational Social Context measure was administered on site to 209 FSS and clinicians in 21 mental health programs in New York State. The study compared the organizational-level culture and climate profiles of mental health clinics that employ both FSS and formally trained clinicians to national norms for child mental health clinics, assessed individual-level job satisfaction and organizational commitment as a function of job (FSS vs. clinician) and other individual-level and organizational-level characteristics, and tested whether FSS and clinicians job attitudes were differentially associated with organizational culture and climate. The programs organizational culture and climate profiles were not significantly different from national norms. Individual-level job satisfaction and organizational commitment were unrelated to position (FSS vs. clinician) or other individual-level and organizational-level characteristics except for culture and climate. Both FSS' and clinicians' individual-level work attitudes were associated similarly with organizational culture and climate.

Improving measurement-based care implementation in youth mental health through organizational leadership and climate: a mechanistic analysis within a randomized trial.

BACKGROUND: Theory and correlational research indicate organizational leadership and climate are important for successful implementation of evidence-based practices (EBPs) in healthcare settings; however, experimental evidence is lacking. We addressed this gap using data from the WISDOM (Working to Implement and Sustain Digital Outcome Measures) hybrid type III effectiveness-implementation trial. Primary outcomes from WISDOM indicated the Leadership and Organizational Change for Implementation (LOCI) strategy improved fidelity to measurement-based care (MBC) in youth mental health services. In this study, we tested LOCI's hypothesized mechanisms of change, namely: (1) LOCI will improve implementation and transformational leadership, which in turn will (2) mediate LOCI's effect on implementation climate, which in turn will (3) mediate LOCI's effect on MBC fidelity. METHODS: Twenty-one outpatient mental health clinics serving youth were randomly assigned to LOCI plus MBC training and technical assistance or MBC training and technical assistance only. Clinicians rated their leaders' implementation leadership, transformational leadership, and clinic implementation climate for MBC at five time points (baseline, 4-, 8-, 12-, and 18-months post-baseline). MBC fidelity was assessed using electronic metadata for youth outpatients who initiated treatment in the 12 months following MBC training. Hypotheses were tested using longitudinal mixed-effects models and multilevel mediation analyses. RESULTS: LOCI significantly improved implementation leadership and implementation climate from baseline to follow-up at 4-, 8-, 12-, and 18-month post-baseline (all ps < .01), producing large effects (range of ds = 0.76 to 1.34). LOCI's effects on transformational leadership were small at 4 months (d = 0.31, p = .019) and nonsignificant thereafter (ps > .05). LOCI's improvement of clinic implementation climate from baseline to 12 months was mediated by improvement in implementation leadership from baseline to 4 months (proportion mediated [pm] = 0.82, p = .004). Transformational leadership did not mediate LOCI's effect on implementation climate (p = 0.136). Improvement in clinic implementation climate from baseline to 12 months mediated LOCI's effect on MBC fidelity during the same period (pm = 0.71, p = .045). CONCLUSIONS: LOCI improved MBC fidelity in youth mental health services by improving clinic implementation climate, which was itself improved by increased implementation leadership. Fidelity to EBPs in healthcare settings can be improved by developing organizational leaders and strong implementation climates. TRIAL REGISTRATION: ClinicalTrials.gov identifier: NCT04096274. Registered September 18, 2019.

Implementation leadership and implementation climate in context: A single organization intrinsic case study for implementation of digital measurement-based care.

Background: Although studies have demonstrated that implementation leadership and climate are important constructs in predicting evidence-based practice (EBP) implementation, concrete descriptions of how they operate during organizational implementation efforts are lacking. This case study fills that gap through an in-depth description of an organization with effective implementation leadership that successfully built a strong implementation climate. This case study provides an illustration of implementation leadership and climate in tangible, replicable terms to assist managers, practitioners, and researchers in addressing the organizational context in their own implementation projects. Method: A single organization, intrinsic case study was employed to paint a multifaceted picture of how one organization leveraged implementation leadership to strengthen a climate for the successful implementation of digital measurement-based care. The case was drawn from a cluster-randomized trial designed to test the effects of a leadership-focused implementation strategy on youth-level fidelity and clinical outcomes of digital measurement-based care. Following the completion of the trial, case study activities commenced. Descriptive summaries of multiple data sources (including quantitative data on implementation leadership and climate, coaching call and organizational alignment meeting recordings and notes, and development plans) were produced and revised iteratively until consensus was reached. Leadership actions were analyzed for corresponding dimensions of implementation leadership and climate. Results: Specific actions organizational leaders took, as well as the timing specific strategies were enacted, to create a climate for implementation are presented, along with lessons learned from this experience. Conclusion: This case study offers concrete steps organizational leaders took to create a consistent and aligned message that the implementation of a specific EBP was a top priority in the agency. The general approach taken to create an implementation climate provides several lessons for leaders, especially for EBPs that have broad implications across an organization.

Using treatments with known positive impact in community-based mental health programs is challenging. Many studies suggest leaders of these programs can help. Similarly, certain features of community-based programs can also be helpful. This case study of an outpatient mental health clinic provides rich descriptions of actions leaders took that shaped the environment in their program and helped improve the use of a treatment with known positive impact. This case study can serve as a practical guide for leaders to reference when aiming to improve the use of treatments with known impact in their own programs.

Reducing turnover is not enough: The need for proficient organizational cultures to support positive youth outcomes in child welfare.

OBJECTIVE: High caseworker turnover has been identified as a factor in the poor outcomes of child welfare services. However, almost no empirical research has examined the relationship between caseworker turnover and youth outcomes in child welfare systems and there is an important knowledge gap regarding whether, and how, caseworker turnover relates to outcomes for youth. We hypothesized that the effects of caseworker turnover are moderated by organizational culture such that reduced caseworker turnover is only associated with improved youth outcomes in organizations with proficient cultures. METHODS: The study applied hierarchical linear models (HLM) analysis to the second National Survey of Child and Adolescent Well-being (NSCAW II) with a U.S. nationwide sample of 2,346 youth aged 1.5- to 18-years-old and 1,544 caseworkers in 73 child welfare agencies. Proficient organizational culture was measured by caseworkers' responses to the Organizational Social Context (OSC) measure; staff turnover was reported by the agencies' directors; and youth outcomes were measured as total problems in psychosocial functioning with the Child Behavior Checklist (CBCL) completed by the youths' caregivers at intake and at 18 month follow-up. RESULTS: The association between caseworker turnover and youth outcomes was moderated by organizational culture. Youth outcomes were improved with lower staff turnover in proficient organizational cultures and the best outcomes occurred in organizations with low turnover and high proficiency. CONCLUSIONS: To be successful, efforts to improve child welfare services by lowering staff turnover must also create proficient cultures that expect caseworkers to be competent and responsive to the needs of the youth and families they serve.

Psychometric evaluation of a pragmatic measure of clinical supervision as an implementation strategy.

BACKGROUND: Valid and reliable measurement of implementation strategies is essential to advancing implementation science; however, this area lags behind the measurement of implementation outcomes and determinants. Clinical supervision is a promising and highly feasible implementation strategy in behavioral healthcare for which pragmatic measures are lacking. This research aimed to develop and psychometrically evaluate a pragmatic measure of clinical supervision conceptualized in terms of two broadly applicable, discrete clinical supervision techniques shown to improve providers' implementation of evidence-based psychosocial interventions-(1) audit and feedback and (2) active learning. METHODS: Items were generated based on a systematic review of the literature and administered to a sample of 154 outpatient mental health clinicians serving youth and 181 community-based mental health providers serving adults. Scores were evaluated for evidence of reliability, structural validity, construct-related validity, and measurement invariance across the two samples. RESULTS: In sample 1, confirmatory factor analysis (CFA) supported the hypothesized two-factor structure of scores on the Evidence-Based Clinical Supervision Strategies (EBCSS) scale (χ2=5.89, df=4, p=0.208; RMSEA=0.055, CFI=0.988, SRMR=0.033). In sample 2, CFA replicated the EBCSS factor structure and provided discriminant validity evidence relative to an established supervisory alliance measure (χ2=36.12, df=30, p=0.204; RMSEA=0.034; CFI=0.990; SRMR=0.031). Construct-related validity evidence was provided by theoretically concordant associations between EBCSS subscale scores and agency climate for evidence-based practice implementation in sample 1 (d= .47 and .55) as well as measures of the supervision process in sample 2. Multiple group CFA supported the configural, metric, and partial scalar invariance of scores on the EBCSS across the two samples. CONCLUSIONS: Scores on the EBCSS provide a valid basis for inferences regarding the extent to which behavioral health providers experience audit and feedback and active learning as part of their clinical supervision in both clinic- and community-based behavioral health settings. TRIAL REGISTRATION: ClinicalTrials.gov NCT04096274 . Registered on 19 September 2019.