The Role of Implementation Climate in Moderating Educator Use of Evidence-Based Practices and Outcomes for Autistic Students.

Ensuring effective use of evidence-based practice (EBP) for autism in schools is imperative due to the significantly increasing number of autistic students receiving school services each year. High-quality EBP use has proven challenging in schools. Research indicates implementation climate, or how EBP are supported, rewarded, and valued, and EBP resources are related to successful implementation. However, limited understanding of system-level contextual factors that impact EBP implementation for school-based providers makes development of appropriate implementation supports challenging. Understanding these factors is crucial for selecting and tailoring implementation strategies to support EBP scale up. In this observational study, California school-based providers (n = 1084) completed surveys related to implementation climate, leadership, autism experience and EBP implementation (use, competence, knowledge). Student outcomes included state level academic and behavioral indicators. Using an implementation science framework (Aarons et al., in Administration and Policy in Mental Health and Mental Health Services Research 38:4-23, 2011) and multilevel modeling, we examined the relationship between EBP Implementation and student outcomes and the moderation effects of provider and district level factors. Higher implementation climate predicted better EBP implementation outcomes, and proved more impactful when provider hands-on autism experience was low. Greater EBP resources predicted a higher percentage of students who met math standards only when district poverty level was high. Our findings suggested moderating effects on EBP implementation from both provider and system level factors. Implementation climate and resources may be especially key in addressing equity issues related to high poverty schools in which teachers often have less autism experience.

Implementation readiness for evidence-based autism practices in school systems.

Background: The increase in the number of autistic children being identified has led to increased demand on public schools to provide high-quality services. Effectively scaling up evidence-based practice (EBP) use for autistic students is challenging, given the complicated organization of special education. Teachers have significant challenges implementing autism EBP with fidelity. Factors such as implementation leadership and climate and attitudes toward EBP are linked to successful EBP use and may vary at different levels of the education system. Examining mechanisms of successful implementation is a critical step to support scale-up. Method: In this observational study, conducted from September 2018 to March 2020, California school personnel (n = 2273) at multiple levels of the system completed surveys related to implementation climate, leadership, and attitudes toward EBP. Data were collected throughout California at the Special Education Local Plan Areas, County Office of Education, and district and school levels from educators and administrators working in public schools supporting autistic students. Multi-level modeling was conducted to characterize implementation readiness. Results: Overall, implementation climate and leadership scores are low across levels with regional levels rated more positively than districts or schools. Attitudes toward EBP were moderate, with those working in schools having the poorest ratings and specialists/trainers and related service providers (e.g., speech-language pathologists) having the highest ratings. Conclusions: Outcomes provide a unique opportunity to compare implementation factors across organizational levels with a large, statewide sample. These data provide guidance for developing implementation interventions at multiple levels of the education system to increase readiness for effective scale-up of autism EBP in schools. Personnel and leaders at different organizational levels may need differentiated training targeting improved implementation climate and leadership. Personnel within districts and schools may experience a particular benefit from leadership support for EBP implementation.

The increase in the number of autistic children being identified in schools is increasing. To address this, schools are trying to do a better job of using high-quality practices based on research. However, teachers have had difficulty using research-based strategies for autistic students the way the manuals indicate they should be used. This might be due to the complexity of the strategies or limited support from special education leadership and infrastructure. Research shows that leaders can be very important in helping teachers use effective strategies. Over 2200 school personnel in California, including administrators, professional development providers, teachers, and paraprofessionals completed surveys asking about how their leaders, schools, districts, and regions supported the use of research practices for autistic students. Overall, limited support is provided in special education, with regional agencies providing more support than districts or schools. These data suggest that school and district leaders need training in how to support educators in using autism-specific strategies.

A Waitlist Randomized Implementation Trial of Classroom Pivotal Response Teaching for Students With Autism.

Classroom Pivotal Response Teaching (CPRT) is a community-partnered adaptation of a naturalistic developmental behavioral intervention identified as an evidence-based practice for autistic children. The current study evaluated student outcomes in a randomized, wait-list controlled implementation trial across classrooms. Participants included teachers (n = 126) and students with autism (n = 308). Teachers participated in 12 hours of didactic, interactive training and additional in-classroom coaching. Generalized Estimating Equations accounted for clustering. Adjusted models evaluated the relative effects of training group, CPRT fidelity, and classroom quality on student outcomes. Results indicate higher CPRT fidelity was associated with greater increases in student learning. Having received CPRT training predicted increased student engagement and greater decreases in reported approach/withdrawal problems. These differences may be linked to the theoretical foundations of CPRT of increasing student motivation and engagement and collaborative adaptation to increase feasibility in schools. Overall, results suggest CPRT may be a beneficial approach for supporting autistic students.

Practice-driven research for statewide scale up: Implementation outcomes of the California Autism Professional Training and Information Network.

LAY ABSTRACT: Supporting use of evidence-based practice in public service programs for autistic individuals is critical. The California Autism Professional Training and Information Network (CAPTAIN) brings together best practices from intervention and implementation research to support scale up of autism services. The current study was designed to evaluate the impact of CAPTAIN on provider-level outcomes including attitude toward, knowledge, fidelity, and use of autism EBPs and overall classroom quality. Overall, results indicated variability across measures, with some significant differences between CAPTAIN-trained and non-CAPTAIN-trained providers. These preliminary findings show promise for the efficacy of the CAPTAIN model to increase dissemination and implementation of EBP at the classroom level.

Proactive versus Reactive: Strategies in the Implementation of School-based Services for Students with ASD.

Autism spectrum disorder (ASD) is increasingly prevalent, intervention costs are high, and long-term outcomes are poor. Proactive implementation of evidence-based practices (EBPs; Steinbrenner et al., Evidence-Based Practices for Children, Youth, and Young Adults with Autism, 2020) through an assessment or planning process can lead to more effective services (Rubin et al., Admin Policy Mental Health Mental Health Serv Res 43(6), 1023-1028, 2016). The objective of this study is (a) to identify factors associated with the use of proactive versus reactive strategies for school-based services for students with ASD and (b) to examine school personnel perceptions of the use of proactive versus reactive strategies. Data were from a larger project in which 6 semi-structured focus groups were conducted to understand school personnel perceptions of how school districts implement new programs for ASD. Transcripts were coded using an iterative coding and review process. In the present study, primary themes were identified and classified as proactive or reactive. Participants reported both proactive and reactive allocation of district resources for intervention implementation. Reactive implementation was associated with litigation or due process, escalated student behavior, and non-public school placement, whereas proactive implementation was associated with ASD-specific programming, grants, personnel experience with EBPs, and prospective needs assessment. Participants perceived the reactive strategies as disadvantageous, and yet these strategies were sometimes still employed. Understanding the role of proactive and reactive strategies and the factors that influence their use could enable more effective planning for EBP use to improve both cost savings and student outcomes. This study is an important first step to explore resource allocation for school-based services for students with ASD.

Concordance between Community Supervisor and Provider Ratings of Fidelity: Examination of Multi-Level Predictors and Outcomes.

Fidelity is conceptualized as a key mechanism impacting treatment outcomes, yet community examination of fidelity is limited, partly due to limited feasible training methods and tools. Provider-report tools represent a feasible method for improving routine fidelity monitoring. Additional work examining the effectiveness of these methods and concordance with other fidelity sources is needed. Further, provider and client factors impact both intervention implementation and concordance, but their impact on fidelity measurement is poorly understood. Using a train-the-trainer methodology, we examine concordance between three methods of assessing fidelity (trained independent coders, supervisor evaluation and provider self-report) using a fidelity assessment tool adapted for community use. Multilevel models examined predictors (provider- and child- level factors) and outcomes (provider average fidelity) of concordance. Results suggest supervisors and providers are able to use the fidelity tool, but demonstrated variable concordance, with higher concordance with trained coders for supervisors than providers. Provider credentials and child language were marginally predictive of rater concordance. Higher supervisor accuracy was associated with higher provider fidelity. Findings inform broader efforts to improve the development and implementation of feasible fidelity tools for routine use in community settings and highlight multi-level factors impacting the adoption and implementation of such tools.

Implementation of school-based services for students with autism: Barriers and facilitators across urban and rural districts and phases of implementation.

LAY ABSTRACT: The law requires that schools use evidence-based practices to educate students with autism spectrum disorder. However, these practices are often not used, or are not used correctly in school programs. Understanding barriers and facilitators of use of evidence-based practices in schools will help improve the implementation process. This study uses focus groups to characterize how school-based providers representing urban or rural school districts perceive barriers and facilitators for implementing new practices for students with autism spectrum disorder. Guiding questions include the following: (1) Are contextual factors perceived as barriers or facilitators and how do these vary by district location? and (2) What are the key factors impacting implementation across the Exploration, Preparation, Implementation, and Sustainment phases? Focus group participants (n = 33) were service providers to children with autism spectrum disorder from urban- and rural-located school districts. Several personnel-related themes (attitudes and buy-in, knowledge and skills, staffing, and burnout) were shared by participants representing both urban and rural districts. However, some personnel-related themes and organizational factors were unique to rural or urban districts. For example, themes related to system and organizational factors (leadership approval, support and expectations, district structure, competing priorities, time for effective professional development, litigation and due process, and materials and resources) differed between the district locations. This project serves as an initial step in identifying implementation strategies that may improve the use of evidence-based practices in schools.

Implementation Science, Behavior Analysis, and Supporting Evidence-based Practices for Individuals with Autism.

Although applied behavior analysis researchers have created efficacious treatment and intervention practices for children and youth with autism spectrum disorder (ASD), there is a gap between research and practice. Implementation Science (IS) and Organizational Behavior Management (OBM), based with Applied Behavior Analysis, are two parallel fields that could close this gap. This paper provides descriptions of both IS and OBM, highlighting their commonalities and unique featuers. The paper concludes with examples of how researchers have used IS and OBM to promote practitioners' use of evidence-based practices and services for children and youth with ASD.

Fidelity Assessment in Community Programs: An Approach to Validating Simplified Methodology.

Fidelity to intervention protocol is linked to best outcomes for individuals with autism spectrum disorder (ASD; see Boyd & Corley [Autism 5(4):430-441, 2001]; Pellecchia et al. [J Autism Dev Disord 45(9):2917-2927, 2015]); however, fidelity measurement tools that are both accurate and feasible for community use are often not available. In this paper we explore methods for validated simplification of fidelity assessment procedures toward the goal of increased use in clinical practice. Video recordings (n = 36) of therapists working with children with ASD were coded using three variations of fidelity assessment methodology (trial-by-trial, 5-point Likert Scale, and 3-point Likert Scale), and the results were compared for exact agreement, mastery criterion agreement, and overall reliability. The results indicated overall a very high percentage of exact agreement (mean 99.44%, range 94.4-100%) and excellent reliability (mean Krippendorff's alpha [Kα] 1.0) between the trial-by-trial and 5-point Likert Scale across all components; however, the 3-point method may be viewed as being the more feasible strategy within community programs.

Statewide Interagency Collaboration to Support Evidence-based Practice Scale Up: The California Autism Professional Training and Information Network (CAPTAIN).

The California Autism Professional Training and Information Network (CAPTAIN) is a statewide interagency collaboration with the goal of scaling up use of evidence-based practices (EBPs) for individuals with autism spectrum disorder (ASD). CAPTAIN began as a clinical initiative then further developed under the influence of implementation science methodology. The Exploration, Preparation, Implementation and Sustainment framework (EPIS) has impacted targeted strategy use for this statewide scale up of EBPs by informing the development of key partnerships, implementation goals, and collaborative processes within CAPTAIN. Currently, CAPTAIN has over 407 members representing 140 school and community agencies who provide training and coaching in EBP and meet regularly with regional teams. Outcome data indicate 51.9% of members provide training and coaching to more than three direct service providers/programs per year. Primary barriers to implementation of EBP were time for training (25.6%), lack of substitute teachers (16.5%), and staff lacking foundational skills (11.5%). Facilitators of implementation and sustainment of the CAPTAIN model include active participation in effective dissemination practices, creative funding and leveraging of local resources, development of the regional collaboratives with active membership, member commitment to EBP for ASD, and use of implementation science to identify and overcome barriers. The purpose of this paper is to highlight CAPTAIN as a model for statewide scale up of EBP in schools as well as other community agencies. Although these efforts have focused on EBP for ASD, the concepts, partnerships and procedures will likely be transferable to other focal issues and may be generalized across service sectors.

Concordance between a U.S. Educational Autism Classification and the Autism Diagnostic Observation Schedule.

States in the United States differ in how they determine special education eligibility for autism services. Few states include an autism-specific diagnostic tool in their evaluation. In research, the Autism Diagnostic Observation Schedule (ADOS for first edition, ADOS-2 for second edition) is considered the gold-standard autism assessment. The purpose of this study was to estimate the proportion of children with an educational classification of autism who exceed the ADOS/ADOS-2 threshold for autism spectrum (concordance rate). Data were drawn from 4 school-based studies across 2 sites (Philadelphia, Pennsylvania, and San Diego, California). Participants comprised 627 children (2-12 years of age; 83% male) with an autism educational classification. Analyses included (a) calculating the concordance rate between educational and ADOS/ADOS-2 classifications and (b) estimating the associations between concordance and child's cognitive ability, study site, and ADOS/ADOS-2 administration year using logistic regression. More San Diego participants (97.5%, all assessed with the ADOS-2) met ADOS/ADOS-2 classification than did Philadelphia participants assessed with the ADOS-2 (92.2%) or ADOS (82.9%). Children assessed more recently were assessed with the ADOS-2; this group was more likely to meet ADOS/ADOS-2 classification than the group assessed longer ago with the ADOS. Children with higher IQ were less likely to meet ADOS/ADOS-2 classification. Most children with an educational classification of autism meet ADOS/ADOS-2 criteria, but results differ by site and by ADOS version and/or recency of assessment. Educational classification may be a reasonable but imperfect measure to include children in community-based trials.

Characterizing Shared and Unique Implementation Influences in Two Community Services Systems for Autism: Applying the EPIS Framework to Two Large-Scale Autism Intervention Community Effectiveness Trials.

The purpose of this study was to examine common and unique factors influencing implementation process for two evidence-based interventions for children with autism spectrum disorder (ASD) in mental health and education service contexts. This study prospectively collected qualitative data from intervention developers and research staff on the implementation process within the context of two separate ASD intervention effectiveness trials. Results reveal common and unique factors influencing implementation in both study contexts. Implementation leadership and provider attitudes and motivation emerge as key influences on implementation across systems. These findings provide promising targets for modular implementation interventions that can be leveraged within growing, large-scale translation efforts in usual care.

Examining relationships between child skills and potential key components of an evidence-based practice in ASD.

BACKGROUND: Research in autism spectrum disorders (ASD) has identified a need to understand key components of complex evidence-based practices (EBP). One approach involves examining the relationship between component use and child behavior. AIMS: This study provides initial evidence for identifying key components in a specific EBP, Pivotal Response Training (PRT). We examined which components were related to child response and evaluated relationships between provider characteristics, child characteristics and component intensity. METHODS: Trained coders reviewed archival videos (n = 278) for PRT fidelity and child behavior. We completed multi-level regression and latent profile analysis to examine relationships between intensity of individual or combinations of PRT components and child behavior, and moderators of component use. RESULTS: Analyses indicated differential relationships between specific components and child behaviors which may support methods of altering intensity of components to individualize intervention. Profile analysis suggested relatively intensive use of most PRT components, especially antecedent strategies, may maximize child responsivity. Providers with postgraduate education trended toward higher intensity component use. Child characteristics did not moderate use. IMPLICATIONS: Careful examination of key components of ASD interventions may helps clarify the mechanisms of action. Recommendations specific to PRT implementation and use of the methodology for other interventions are discussed.

Leadership profiles associated with the implementation of behavioral health evidence-based practices for autism spectrum disorder in schools.

Implementation of evidence-based practice (EBP) for autism spectrum disorder (ASD) in the education system is a public health priority. Leadership is a critical driver of EBP implementation but little is known about the types of leadership behaviors exhibited by school leaders and how this influences the context of EBP implementation, particularly for students with ASD. The objectives of this study were to determine (1) the leadership profiles of principals involved in EBP implementation for students with ASD and (2) how these leadership profiles related to school characteristics and implementation climate. The Exploration, Preparation, Implementation, Sustainment (EPIS) framework was used to guide the design and analysis of this study. Participants (n = 296) included principals, teachers, and classroom support staff. They provided demographic information and completed the Multifactor Leadership Questionnaire and Implementation Climate Scale. Using latent profile analysis, a three-pattern solution was identified: Disengaged (6% of sample), Undifferentiated (23% of sample), and Optimal (71% of sample). Principals in schools with higher proportions of students with an individualized education program were more likely to be classified as Undifferentiated than Optimal. The Optimal group was associated with more positive implementation climate than the Undifferentiated or Disengaged groups. Findings suggest that leadership behaviors rated by principals and their staff involved in implementation of common autism EBPs can be meaningfully clustered into three discernible profiles that are shaped by organizational context and linked to strategic implementation climate. Our study findings have implications for leadership training and service delivery in schools by underscoring the critical nature of school leadership during implementation of EBPs for children with autism and the interplay between specific leadership behaviors and strategic implementation climate.

Correction to: Exploring multi-level system factors facilitating educator training and implementation of evidence-based practices (EBP): a study protocol.

After publication of the original article [1] it was brought to our attention that author Elizabeth McGhee Hassrick was erroneously included as Elizabeth McGee Hassrick. The correct spelling of the author's name is included in the author list of this erratum.

Exploring multi-level system factors facilitating educator training and implementation of evidence-based practices (EBP): a study protocol.

BACKGROUND: This study examines how system-wide (i.e., region, district, and school) mechanisms such as leadership support, training requirements, structure, collaboration, and education affect the use of evidence-based practices (EBPs) in schools and how this affects the outcomes for students with autism spectrum disorder (ASD). Despite growing evidence for the positive effects of EBPs for ASD, these practices are not consistently or effectively used in schools. Although special education programs are mandated to use EBPs, there are very few evidence-based methods for selecting, implementing, and sustaining EBPs. Research focuses primarily on teacher training, without attention to contextual factors (e.g., implementation climate, attitudes toward EBPs, resource allocation, and social networks) that may impact outcomes. Using an implementation science framework, this project will prospectively examine relations between system-wide factors and teachers' use of EBPs and student education outcomes. METHODS/DESIGN: Survey data will be collected from approximately 85 regional special education directors, 170 regional program specialists, 265 district special education directors, 265 behavior specialists, 925 school principals, 3538 special education teachers, and 2700 paraprofessionals. Administrative data for the students with ASD served by participating teachers will be examined. A total of 79 regional-, district-, and school-level personnel will also participate in social network interviews. Mixed methods, including surveys, administrative data, and observational checklists, will be used to gather in-depth information about system-wide malleable factors that relate to positive teacher implementation of EBPs and student outcomes. Multi-level modeling will be used to assess system-wide malleable factors related to EBP implementation which will be linked to the trainer, teacher, and student outcomes and examined based on moderators (e.g., district size, Special Education Local Plan Area structure, teachers' ASD experience). Finally, a dynamic social network approach will be used to map EBP-related connectivity across all levels of the system for selected regions. Dynamic network analysis will be used to gauge the degree to which and ways that EBP trainings, resources, and interventions are shared (or not shared) among school staff. DISCUSSION: Results are expected to inform the development of system-wide interventions to improve the school-based implementation of EBPs for students with ASD.

Parent and Teacher Concordance of Child Outcomes for Youth with Autism Spectrum Disorder.

Cross-informant ratings of are considered gold standard for child behavioral assessment. To date, little work has examined informant ratings of adaptive functioning for youth with autism spectrum disorder (ASD). In a large, diverse sample of youth with ASD, this study evaluated parent-teacher concordance of ratings of adaptive functioning and ASD-specific symptomatology across time. The impact of child clinical characteristics on concordance was also examined. Participants included 246 children, their caregivers and teachers. Parent-teacher concordance was variable but generally consistent across time. Concordance was significantly impacted by autism severity and child cognitive abilities. Findings inform the broader concordance literature and support the need to consider child clinical factors when assessing child functioning in samples of children with ASD.

Examination of the prevalence of stimulus overselectivity in children with ASD.

Many individuals with autism spectrum disorders (ASD) display stimulus overselectivity, wherein a subset of relevant components in a compound stimulus controls responding, which impairs discrimination learning. The original experimental research on stimulus overselectivity in ASD was conducted several decades ago; however, interventions for children with ASD now typically include programming to target conditional discriminations in ways that might minimize the prevalence of stimulus overselectivity. The present study assessed 42 children who had been diagnosed or educationally identified with ASD using a discrimination learning assessment. Of these 42 children, 19% displayed overselective responding, which is a lower percentage than that seen in early research. Possible explanations for this decreased percentage, implications for intervention, and future directions for research are discussed.