After-School Programs and Children's Mental Health: Organizational Social Context, Program Quality, and Children's Social Behavior.

OBJECTIVE: The current study examined associations among organizational social context, after-school program (ASP) quality, and children's social behavior in a large urban park district. METHOD: Thirty-two park-based ASPs are included in the final sample, including 141 staff and 593 children. Staff reported on organizational culture (rigidity, proficiency, resistance) and climate (engagement, functionality, stress), and children's social skills and problem behaviors. Children and their parents reported on program quality indicators (e.g., activities, routines, relationships). Parents also completed a children's mental health screener. RESULTS: A series of Hierarchical Linear Models revealed that proficiency and stress were the only organizational predictors of program quality; associations between stress and program quality were moderated by program enrollment and aggregated children's mental health need. Higher child- and parent-perceived program quality related to fewer staff-reported problem behaviors, while overall higher enrollment and higher aggregated mental health need were associated with fewer staff-reported social skills. CONCLUSIONS: Data are informing ongoing efforts to improve organizational capacity of urban after-school programs to support children's positive social and behavior trajectories.

Evaluation of the Implementation Research Institute: Trainees' Publications and Grant Productivity.

With growth in the field of dissemination and implementation (D&I) research, there has been growth in capacity building, with many training opportunities. As such, it is important to continue to evaluate D&I research training programs. This paper reports the results of an evaluation of the Implementation Research Institute (IRI), a R25 funded by the National Institute of Mental Health with additional funding by the Department of Veterans Affairs (VA). The fourth cohort also had a supplement from the National Institute on Drug Abuse. Using bibliometrics data, we report on a quasi-experimental retrospective cohort study assessing whether the rates of scholarly productivity in D&I science of IRI fellows (those who applied and were accepted to the training) were greater than those who applied but were not accepted to IRI. Our findings show that Selected Applicants' odds of publishing in implementation science were higher for earlier alumni, starting at 12% 1 year out and increasing to 94% for those who were 4 years out from starting training. Chances for Non-Selected Applicants remained relatively stable, starting at 47% at 1 year and going to 33% at 4 years since their application, a pattern that was stable even after controlling for demographic characteristics. These results support the hypothesis that IRI is increasing the D&I research productivity of those selected to the program, and that our fellows are advancing the field of D&I compared to those investigators not selected to our institute. Our finding also indicates the importance of a 2-year training.

Mechanisms of Change in the ARC Organizational Strategy: Increasing Mental Health Clinicians' EBP Adoption Through Improved Organizational Culture and Capacity.

The development of efficient and scalable implementation strategies in mental health is restricted by poor understanding of the change mechanisms that increase clinicians' evidence-based practice (EBP) adoption. This study tests the cross-level change mechanisms that link an empirically-supported organizational strategy for supporting implementation (labeled ARC for Availability, Responsiveness, and Continuity) to mental health clinicians' EBP adoption and use. Four hundred seventy-five mental health clinicians in 14 children's mental health agencies were randomly assigned to the ARC intervention or a control condition. Measures of organizational culture, clinicians' intentions to adopt EBPs, and job-related EBP barriers were collected before, during, and upon completion of the three-year ARC intervention. EBP adoption and use were assessed at 12-month follow-up. Multilevel mediation analyses tested changes in organizational culture, clinicians' intentions to adopt EBPs, and job-related EBP barriers as linking mechanisms explaining the effects of ARC on clinicians' EBP adoption and use. ARC increased clinicians' EBP adoption (OR = 3.19, p = .003) and use (81 vs. 56 %, d = .79, p = .003) at 12-month follow-up. These effects were mediated by improvement in organizational proficiency culture leading to increased clinician intentions to adopt EBPs and by reduced job-related EBP barriers. A combined mediation analysis indicated the organizational culture-EBP intentions mechanism was the primary carrier of ARC's effects on clinicians' EBP adoption and use. ARC increases clinicians' EBP adoption and use by creating proficient organizational cultures that increase clinicians' intentions to adopt EBPs.

Aligning organizational priorities with ARC to improve youth mental health service outcomes.

OBJECTIVE: The Availability, Responsiveness, and Continuity (ARC) organizational intervention is designed to improve community-based youth mental health services by aligning organizational priorities with 5 principles of effective service organizations (i.e., mission driven, results oriented, improvement directed, relationship centered, participation based). This study assessed the effect of the ARC intervention on youth outcomes and the mediating role of organizational priorities as a mechanism linking the ARC intervention to outcomes. METHOD: Fourteen community-based mental health agencies in a midwestern metropolis along with 475 clinicians and 605 youth (ages 5-18) served by those agencies were randomly assigned to the 3-year ARC intervention or control condition. The agencies' priorities were measured with the ARC Principles Questionnaire (APQ) completed by clinicians at the end of the intervention. Youth outcomes were measured as total problems in psychosocial functioning described by their caregivers using the Shortform Assessment for Children (SAC) at 6 monthly intervals. RESULTS: The rate of improvement in youths' psychosocial functioning in agencies assigned to the ARC condition was 1.6 times the rate of improvement in agencies assigned to the control condition, creating a standardized difference in functioning of d = .23 between the 2 groups at the 6-month follow-up. The effect on youth outcomes was fully mediated by the alignment of organizational priorities described in the 5 ARC principles (d = .21). CONCLUSION: The ARC organizational intervention improves youth outcomes by aligning organizational priorities with the 5 ARC principles. The findings suggest that organizational priorities explain why some community mental health agencies are more effective than others. (PsycINFO Database Record

Increasing clinicians' EBT exploration and preparation behavior in youth mental health services by changing organizational culture with ARC.

OBJECTIVE: Clinician EBT exploration and preparation behavior is essential to the ongoing implementation of new EBTs. This study examined the effect of the ARC organizational intervention on clinician EBT exploration and preparation behavior and assessed the mediating role of organizational culture as a linking mechanism. METHOD: Fourteen community mental health agencies that serve youth in a major Midwestern metropolis along with 475 clinicians who worked in those agencies, were randomly assigned to either the three-year ARC intervention or control condition. Organizational culture was assessed with the Organizational Social Context (OSC) measure at baseline and follow-up. EBT exploration and preparation behavior was measured as clinician participation in nine separate community EBT workshops held over a three-year period. RESULTS: There was 69 percent greater odds (OR = 1.69, p < .003) of clinicians in the ARC condition (versus control) attending each subsequent workshop. Workshop attendance in the control group remained under two percent and declined over three years while attendance in the ARC condition grew from 3.6 percent in the first workshop to 12 percent in the ninth and final workshop. Improvement in proficient organizational culture mediated the positive effect of the ARC intervention on clinicians' workshop attendance (a × b = .21; 95% CI:LL = .05, UL = .40). CONCLUSIONS: Organizational interventions that create proficient mental health agency cultures can increase clinician EBT exploration and preparation behavior that is essential to the ongoing implementation of new EBTs in community youth mental health settings.

Assessing and changing organizational social contexts for effective mental health services.

Culture and climate are critical dimensions of a mental health service organization's social context that affect the quality and outcomes of the services it provides and the implementation of innovations such as evidence-based treatments (EBTs). We describe a measure of culture and climate labeled Organizational Social Context (OSC), which has been associated with innovation, service quality, and outcomes in national samples and randomized controlled trials (RCTs) of mental health and social service organizations. The article also describes an empirically supported organizational intervention model labeled Availability, Responsiveness, and Continuity (ARC), which has improved organizational social context, innovation, and effectiveness in five RCTs. Finally, the article outlines a research agenda for developing more efficient and scalable organizational strategies to improve mental health services by identifying the mechanisms that link organizational interventions and social context to individual-level service provider intentions and behaviors associated with innovation and effectiveness.

Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: a United States national study.

Theories of organizational culture and climate (OCC) applied to child welfare systems hypothesize that strategic dimensions of organizational culture influence organizational climate and that OCC explains system variance in youth outcomes. This study provides the first structural test of the direct and indirect effects of culture and climate on youth outcomes in a national sample of child welfare systems and isolates specific culture and climate dimensions most associated with youth outcomes. The study applies multilevel path analysis (ML-PA) to a U.S. nationwide sample of 2,380 youth in 73 child welfare systems participating in the second National Survey of Child and Adolescent Well-being. Youths were selected in a national, two-stage, stratified random sample design. Youths' psychosocial functioning was assessed by caregivers' responses to the Child Behavior Checklist at intake and at 18-month follow-up. OCC was assessed by front-line caseworkers' (N=1,740) aggregated responses to the Organizational Social Context measure. Comparison of the a priori and subsequent trimmed models confirmed a reduced model that excluded rigid organizational culture and explained 70% of the system variance in youth outcomes. Controlling for youth- and system-level covariates, systems with more proficient and less resistant organizational cultures exhibited more functional, more engaged, and less stressful climates. Systems with more proficient cultures and more engaged, more functional, and more stressful climates exhibited superior youth outcomes. Findings suggest child welfare administrators can support service effectiveness with interventions that improve specific dimensions of culture and climate.

The organizational social context of mental health medicaid waiver programs with family support services: implications for research and practice.

Peer family support specialists (FSS) are parents with practical experience in navigating children's mental health care systems who provide support, advocacy, and guidance to the families of children who need mental health services. Their experience and training differ from those of formally trained mental health clinicians, creating potential conflicts in priorities and values between FSS and clinicians. We hypothesized that these differences could negatively affect the organizational cultures and climates of mental health clinics that employ both FSS and mental health clinicians, and lower the job satisfaction and organizational commitment of FSS. The Organizational Social Context measure was administered on site to 209 FSS and clinicians in 21 mental health programs in New York State. The study compared the organizational-level culture and climate profiles of mental health clinics that employ both FSS and formally trained clinicians to national norms for child mental health clinics, assessed individual-level job satisfaction and organizational commitment as a function of job (FSS vs. clinician) and other individual-level and organizational-level characteristics, and tested whether FSS and clinicians job attitudes were differentially associated with organizational culture and climate. The programs organizational culture and climate profiles were not significantly different from national norms. Individual-level job satisfaction and organizational commitment were unrelated to position (FSS vs. clinician) or other individual-level and organizational-level characteristics except for culture and climate. Both FSS' and clinicians' individual-level work attitudes were associated similarly with organizational culture and climate.

Quality indicators for family support services and their relationship to organizational social context.

Quality measurement is an important component of healthcare reform. The relationship of quality indicators (QIs) for parent-delivered family support services to organizational social contexts known to improve quality is unexamined. This study employs data collected from 21 child mental health programs that deliver team-based family support services. Performance on two levels of QIs-those targeting the program and staff-were significantly associated with organizational social context profiles and dimensions. High quality program policies are associated with positive organizational cultures and engaging climates. Inappropriate staff practices are associated with resistant cultures. Implications for organizational strategies to improve service quality are discussed.

Reducing turnover is not enough: The need for proficient organizational cultures to support positive youth outcomes in child welfare.

OBJECTIVE: High caseworker turnover has been identified as a factor in the poor outcomes of child welfare services. However, almost no empirical research has examined the relationship between caseworker turnover and youth outcomes in child welfare systems and there is an important knowledge gap regarding whether, and how, caseworker turnover relates to outcomes for youth. We hypothesized that the effects of caseworker turnover are moderated by organizational culture such that reduced caseworker turnover is only associated with improved youth outcomes in organizations with proficient cultures. METHODS: The study applied hierarchical linear models (HLM) analysis to the second National Survey of Child and Adolescent Well-being (NSCAW II) with a U.S. nationwide sample of 2,346 youth aged 1.5- to 18-years-old and 1,544 caseworkers in 73 child welfare agencies. Proficient organizational culture was measured by caseworkers' responses to the Organizational Social Context (OSC) measure; staff turnover was reported by the agencies' directors; and youth outcomes were measured as total problems in psychosocial functioning with the Child Behavior Checklist (CBCL) completed by the youths' caregivers at intake and at 18 month follow-up. RESULTS: The association between caseworker turnover and youth outcomes was moderated by organizational culture. Youth outcomes were improved with lower staff turnover in proficient organizational cultures and the best outcomes occurred in organizations with low turnover and high proficiency. CONCLUSIONS: To be successful, efforts to improve child welfare services by lowering staff turnover must also create proficient cultures that expect caseworkers to be competent and responsive to the needs of the youth and families they serve.

The implementation research institute: training mental health implementation researchers in the United States.

BACKGROUND: The Implementation Research Institute (IRI) provides two years of training in mental health implementation science for 10 new fellows each year. The IRI is supported by a National Institute of Mental Health (NIMH) R25 grant and the Department of Veterans Affairs (VA). Fellows attend two annual week-long trainings at Washington University in St. Louis. Training is provided through a rigorous curriculum, local and national mentoring, a 'learning site visit' to a federally funded implementation research project, pilot research, and grant writing. METHODS: This paper describes the rationale, components, outcomes to date, and participant experiences with IRI. RESULTS: IRI outcomes include 31 newly trained implementation researchers, their new grant proposals, contributions to other national dissemination and implementation research training, and publications in implementation science authored by the Core Faculty and fellows. Former fellows have obtained independent research funding in implementation science and are beginning to serve as mentors for more junior investigators. CONCLUSIONS: Based on the number of implementation research grant proposals and papers produced by fellows to date, the IRI is proving successful in preparing new researchers who can inform the process of making evidence-based mental healthcare more available through real-world settings of care and who are advancing the field of implementation science.

A mixed methods multiple case study of implementation as usual in children's social service organizations: study protocol.

BACKGROUND: Improving quality in children's mental health and social service settings will require implementation strategies capable of moving effective treatments and other innovations (e.g., assessment tools) into routine care. It is likely that efforts to identify, develop, and refine implementation strategies will be more successful if they are informed by relevant stakeholders and are responsive to the strengths and limitations of the contexts and implementation processes identified in usual care settings. This study will describe: the types of implementation strategies used; how organizational leaders make decisions about what to implement and how to approach the implementation process; organizational stakeholders' perceptions of different implementation strategies; and the potential influence of organizational culture and climate on implementation strategy selection, implementation decision-making, and stakeholders' perceptions of implementation strategies. METHODS/DESIGN: This study is a mixed methods multiple case study of seven children's social service organizations in one Midwestern city in the United States that compose the control group of a larger randomized controlled trial. Qualitative data will include semi-structured interviews with organizational leaders (e.g., CEOs/directors, clinical directors, program managers) and a review of documents (e.g., implementation and quality improvement plans, program manuals, etc.) that will shed light on implementation decision-making and specific implementation strategies that are used to implement new programs and practices. Additionally, focus groups with clinicians will explore their perceptions of a range of implementation strategies. This qualitative work will inform the development of a Web-based survey that will assess the perceived effectiveness, relative importance, acceptability, feasibility, and appropriateness of implementation strategies from the perspective of both clinicians and organizational leaders. Finally, the Organizational Social Context measure will be used to assess organizational culture and climate. Qualitative, quantitative, and mixed methods data will be analyzed and interpreted at the case level as well as across cases in order to highlight meaningful similarities, differences, and site-specific experiences. DISCUSSION: This study is designed to inform efforts to develop more effective implementation strategies by fully describing the implementation experiences of a sample of community-based organizations that provide mental health services to youth in one Midwestern city.

Randomized trial of the Availability, Responsiveness and Continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs.

OBJECTIVES: The primary objective of the study was to assess whether the Availability, Responsiveness and Continuity (ARC) organizational intervention improved youth outcomes in community based mental health programs. The second objective was to assess whether programs with more improved organizational social contexts following the 18-month ARC intervention had better youth outcomes than programs with less improved social contexts. METHOD: Eighteen community mental health programs that serve youth between the ages of 5 and 18 were randomly assigned to ARC or control conditions. Clinicians (n = 154) in the participating programs completed the Organizational Social Context (OSC) measure at baseline and following the 18-month ARC organizational intervention. Caregivers of 393 youth who were served by the 18 programs (9 in ARC and 9 in control) completed the Shortform Assessment for Children (SAC) once a month for six months beginning at intake. RESULTS: Hierarchical linear models (HLM) analyses indicated that youth outcomes were significantly better in the programs that completed the 18 month ARC intervention. HLM analyses also showed that youth outcomes were best in the programs with the most improved organizational social contexts following the 18 month ARC intervention. CONCLUSIONS: Youth outcomes in community mental health programs can be improved with the ARC organizational intervention and outcomes are best in programs that make the most improvements in organizational social context. The relationships linking ARC, organizational social context, and youth outcomes suggest that service improvement efforts will be more successful if those efforts include strategies to improve the organizational social contexts in which the services are embedded.

Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice.

OBJECTIVE: The study: (1) provides the first assessment of the a priori measurement model and psychometric properties of the Organizational Social Context (OSC) measurement system in a US nationwide probability sample of child welfare systems; (2) illustrates the use of the OSC in constructing norm-based organizational culture and climate profiles for child welfare systems; and (3) estimates the association of child welfare system-level organizational culture and climate profiles with individual caseworker-level job satisfaction and organizational commitment. METHODS: The study applies confirmatory factor analysis (CFA) and hierarchical linear models (HLM) analysis to a US nationwide sample of 1,740 caseworkers from 81 child welfare systems participating in the second National Survey of Child and Adolescent Wellbeing (NSCAW II). The participating child welfare systems were selected using a national probability procedure reflecting the number of children served by child welfare systems nationwide. RESULTS: The a priori OSC measurement model is confirmed in this nationwide sample of child welfare systems. In addition, caseworker responses to the OSC scales generate acceptable to high scale reliabilities, moderate to high within-system agreement, and significant between-system differences. Caseworkers in the child welfare systems with the best organizational culture and climate profiles report higher levels of job satisfaction and organizational commitment. Organizational climates characterized by high engagement and functionality, and organizational cultures characterized by low rigidity are associated with the most positive work attitudes. CONCLUSIONS: The OSC is the first valid and reliable measure of organizational culture and climate with US national norms for child welfare systems. The OSC provides a useful measure of Organizational Social Context for child welfare service improvement and implementation research efforts which include a focus on child welfare system culture and climate.

Randomized trial of the Availability, Responsiveness, and Continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth.

OBJECTIVE: Evidence-based Practice (EBP) implementation is likely to be most efficient and effective in organizations with positive social contexts (i.e., organizational culture, climate, and work attitudes of clinicians). The study objective was to test whether an organizational intervention labeled Availability, Responsiveness and Continuity (ARC) could improve the organizational social contexts of community-based mental health programs for youth. METHOD: The study randomly assigned 26 community-based mental health programs for youth to ARC or control conditions. The organizational cultures, climates, and work attitudes of clinicians (n = 197) in the programs were assessed with the Organizational Social Context (OSC) measure for mental health services at baseline and following the 18-month ARC intervention. RESULTS: Hierarchical linear models (HLM) analyses indicated that organizational culture, climate, and work attitudes were significantly improved in the ARC condition after 18 months. Clinicians in programs assigned to ARC reported less rigid, less centralized and less apathetic organizational cultures, more engaged and functional organizational climates with less role conflict, and work attitudes with improved morale, job satisfaction, and organizational commitment. CONCLUSIONS: ARC improved the organizational social contexts of clinicians in community-based mental health programs for youth. Results suggest that organizational intervention strategies can be used to create the types of organizational social contexts that are believed to be necessary for EBP implementation and other service innovations in mental health programs.

The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study.

UNLABELLED: ABSTBACKGROUND: Evidence-based practices have not been routinely adopted in community mental health organizations despite the support of scientific evidence and in some cases even legislative or regulatory action. We examined the association of clinician attitudes toward evidence-based practice with organizational culture, climate, and other characteristics in a nationally representative sample of mental health organizations in the United States. METHODS: In-person, group-administered surveys were conducted with a sample of 1,112 mental health service providers in a nationwide sample of 100 mental health service institutions in 26 states in the United States. The study examines these associations with a two-level Hierarchical Linear Modeling (HLM) analysis of responses to the Evidence-Based Practice Attitude Scale (EBPAS) at the individual clinician level as a function of the Organizational Social Context (OSC) measure at the organizational level, controlling for other organization and clinician characteristics. RESULTS: We found that more proficient organizational cultures and more engaged and less stressful organizational climates were associated with positive clinician attitudes toward adopting evidence-based practice. CONCLUSIONS: The findings suggest that organizational intervention strategies for improving the organizational social context of mental health services may contribute to the success of evidence-based practice dissemination and implementation efforts by influencing clinician attitudes.

Organizational climate, services, and outcomes in child welfare systems.

OBJECTIVE: This study examines the association of organizational climate, casework services, and youth outcomes in child welfare systems. Building on preliminary findings linking organizational climate to youth outcomes over a 3-year follow-up period, the current study extends the follow-up period to 7 years and tests main, moderating and mediating effects of organizational climate and casework services on outcomes. METHODS: The study applies hierarchical linear models (HLMs) analyses to all 5 waves of the National Survey of Child and Adolescent Well-being (NSCAW) with a US nationwide sample of 1,678 maltreated youth aged 4-16 years and 1,696 caseworkers from 88 child welfare systems. Organizational climate is assessed on 2 dimensions, Engagement and Stress, with scales from the well established measure, Organizational Social Context (OSC); youth outcomes are measured as problems in psychosocial functioning with the Child Behavior Checklist (CBCL); and casework services are assessed with original scales developed for the study and completed by the maltreated youths' primary caregivers and caseworkers. RESULTS: Maltreated youth served by child welfare systems with more engaged organizational climates have significantly better outcomes. Moreover, the quantity and quality of casework services neither mediate nor interact with the effects of organizational climate on youth outcomes. CONCLUSIONS: Organizational climate is associated with youth outcomes in child welfare systems, but a better understanding is needed of the mechanisms that link organizational climate to outcomes. In addition, there is a need for evidence-based organizational interventions that can improve the organizational climates and effectiveness of child welfare systems.

Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy.

OBJECTIVE: A randomized trial assessed the effectiveness of a 2-level strategy for implementing evidence-based mental health treatments for delinquent youth. METHOD: A 2 x 2 design encompassing 14 rural Appalachian counties included 2 factors: (a) the random assignment of delinquent youth within each county to a multisystemic therapy (MST) program or usual services and (b) the random assignment of counties to the ARC (for availability, responsiveness, and continuity) organizational intervention for implementing effective community-based mental health services. The design created 4 treatment conditions (MST plus ARC, MST only, ARC only, control). Outcome measures for 615 youth who were 69% male, 91% Caucasian, and aged 9-17 years included the Child Behavior Checklist and out-of-home placements. RESULTS: A multilevel, mixed-effects, regression analysis of 6-month treatment outcomes found that youth total problem behavior in the MST plus ARC condition was at a nonclinical level and significantly lower than in other conditions. Total problem behavior was equivalent and at nonclinical levels in all conditions by the 18-month follow-up, but youth in the MST plus ARC condition entered out-of-home placements at a significantly lower rate (16%) than youth in the control condition (34%). CONCLUSIONS: Two-level strategies that combine an organizational intervention such as ARC and an evidence-based treatment such as MST are promising approaches to implementing effective community-based mental health services. More research is needed to understand how such strategies can be used effectively in a variety of organizational contexts and with other types of evidence-based treatments.

Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS).

The Evidence-Based Practice Attitude Scale (EBPAS) assesses mental health and social service provider attitudes toward adopting evidence-based practices. Scores on the EBPAS derive from 4 subscales (i.e., Appeal, Requirements, Openness, and Divergence) as well as the total scale, and preliminary studies have linked EBPAS scores to clinic structure and policies, organizational culture and climate, and first-level leadership. EBPAS scores are also related to service provider characteristics, including age, education level, and level of professional development. The present study examined the factor structure, reliability, and norms of EBPAS scores in a sample of 1,089 mental health service providers from a nationwide sample drawn from 100 service institutions in 26 states in the United States. The study also examined associations of provider demographic characteristics with EBPAS subscale and total scores. Confirmatory factor analysis supported a second-order factor model, and reliability coefficients for the subscales ranged from .91 to .67 (total scale = .74). The study establishes national norms for the EBPAS so that comparisons can be drawn for U.S. local as well as international studies of attitudes toward evidence-based practices. The results suggest that the factor structure and reliability are likely generalizable to a variety of service provider contexts and different service settings and that the EBPAS subscales are associated with provider characteristics. Directions for future research are discussed.