Examining Multiteam Systems Across Context and Type: A Historiometric Analysis of Failed MTS Performance.

Multiteam systems (MTSs) are complex organizational forms comprising interdependent teams that work towards their own proximal goals within and across teams to also accomplish a shared superordinate goal. MTSs operate within high-stakes, dangerous contexts with high consequences for suboptimal performance. We answer calls for nuanced exploration and cross-context comparison of MTSs "in the wild" by leveraging the MTS action sub-phase behavioral taxonomy to determine where and how MTS failures occur. To our knowledge, this is the first study to also examine how key MTS attributes (boundary status, goal type) influence MTS processes and performance. We conducted historiometric analysis on 40 cases of failed MTS performance across various contexts (e.g., emergency response, commercial transportation, military, and business) to uncover patterns of within- and between-team behaviors of failing MTSs, resulting in four themes. First, component teams of failing MTSs over-engaged in within-team alignment behaviors (vs. between-team behaviors) by enacting acting, monitoring, and recalibrating behaviors more often within than between teams. Second, failing MTSs over-focused on acting behaviors (vs. monitoring or recalibrating) and tended to not fully enact the action sub-phase cycle. Third and fourth, boundary status and goal type exacerbated these behavioral patterns, as external and physical MTSs were less likely to enact sufficient between-team behaviors or fully enact the action sub-phase cycle compared to internal and intellectual MTSs. We propose entrainment as a mechanism for facilitating MTS performance wherein specific, cyclical behavioral patterns enacted by teams align to facilitate goal achievement via three multilevel behavioral cycles (i.e., acting-focused, alignment-focused, and adjustment-focused). We argue that the degree to which these cycles are aligned both between teams and with the overarching MTS goal determines whether and how an MTS fails. Our findings add nuance beyond single-context MTS studies by showing that the identified behavioral patterns hold both across contexts and almost all types of MTS action-phase behaviors. We show that these patterns vary by MTS boundary status and goal type. Our findings inform MTS training best practices, which should be structured to integrate all component teams and tailored to both MTS attributes (i.e., boundary status, goal type) and situation type (e.g., contingency planning).

Influence of Simulation-based Training on Reflective Practice.

BACKGROUND: Simulation-based training (SBT) is often evaluated based on the transfer of specific knowledge and skills. In contrast, the degree to which reflective practice is inculcated by SBT is rarely considered. Because reflection is a pillar of adult learning theories, we sought to examine the degree to which participation in SBT was associated with increased reflective practice. METHOD: Eighty one healthcare professionals completed a survey which included the number of SBTs they participated in during the past two years, content- and administrative-related features of those SBTs, and a key aspect of reflective practice (i.e., self-appraisal). RESULTS: The number of SBTs healthcare professionals participated in during the past two years was positively associated with reflective self-appraisal. This relationship was not moderated by the inclusion of reflection components in SBTs nor by the voluntary/mandatory nature of participation in SBTs. Furthermore, the facilitator was ranked as the most important feature of the overall learning experience in SBTs. Also, no significant differences were found between the number of technical skills-based and non-technical skills-based SBTs. CONCLUSION: These findings demonstrate the importance (of evaluating) SBTs for facilitating reflective learning mindsets that healthcare practitioners can apply beyond the specific skills trained by SBTs.

Validation of the Implementation Climate Scale in Nursing.

BACKGROUND: One critical factor in effective implementation of evidence-based practices (EBPs) in nursing is an organizational context that facilitates and supports implementation efforts. Measuring implementation climate can add useful insights on the extent to which the organizational context supports EBP implementation. AIMS: This study cross-validates and examines the psychometric properties of the Implementation Climate Scale (ICS), which measures nurses' perceptions of their unit's climate for EBP implementation. METHODS: This study analyzed ICS data from two cross-sectional studies, including 203 nurses from California and 301 nurses from Florida. Analyses included evaluation of internal consistency, multilevel aggregation statistics, and confirmatory factor analyses. RESULTS: The 18-item ICS demonstrated comparable psychometric properties to the original measure development paper in both samples. Confirmatory factor analyses provided support for the scale's factor structure in both samples. LINKING EVIDENCE TO PRACTICE: The ICS is a pragmatic measure that can be used to assess unit implementation climate in nursing contexts. Results from the ICS from nurses and nurse leaders can provide insights into implementation-specific barriers and facilitators within the organizational context.

Validation of a pragmatic measure of implementation citizenship behavior in substance use disorder treatment agencies.

The organizational context in which substance use disorder treatment (SUDT) evidence-based practices (EBPs) are implemented plays a critical role in successful implementation. Employee behaviors that go above and beyond typical job requirements to support EBP implementation have been suggested to facilitate the likelihood of overall implementation success. The current study explored the psychometric properties of the Implementation Citizenship Behavior Scale (ICBS) within SUDT settings. Utilizing a sample of 322 direct service providers and 60 of their respective supervisors from three SUDT agencies, results from a confirmatory factor analysis and construct validity analysis support the use of the ICBS in the SUDT context. Validation of the ICBS provides a useful, pragmatic tool for both researchers and practitioners to assess employee citizenship behavior to support EBP implementation. The ICBS can provide critical insights into how providers respond to organizational context that may facilitate EBP implementation.

Leader-member exchange and organizational climate effects on clinician turnover intentions.

PURPOSE: Clinician turnover in mental health settings impacts service quality, including availability and delivery of evidence-based practices. Leadership is associated with organizational climate, team functioning and clinician turnover intentions (TI). This study examines leader-member exchange (LMX), reflecting the relationship between a supervisor and each supervisee, using mean team LMX, dispersion of individual clinician ratings compared to team members (i.e. relative LMX) and team level variability (i.e. LMX differentiation), in relation to organizational climate and clinician TI. DESIGN/METHODOLOGY/APPROACH: Survey data were collected from 363 clinicians, nested in children's mental health agency workgroups, providing county-contracted outpatient services to youth and families. A moderated mediation path analysis examined cross-level associations of leader-member exchange with organizational climate and turnover intentions. FINDINGS: Lower relative LMX and greater LMX differentiation were associated with higher clinician TI. Higher team-level demoralizing climate also predicted higher TI. These findings indicate that poorer LMX and more variability in LMX at the team level are related to clinician TI. ORIGINALITY/VALUE: This study describes both team- and clinician-level factors on clinician TI. Few studies have examined LMX in mental health, and fewer still have examined relative LMX and LMX differentiation associations with organizational climate and TI. These findings highlight the importance of leader-follower relationships and organizational climate and their associations with clinician TIs. Mental health service systems and organizations can address these issues through fostering more positive supervisor-supervisee relationships.

EBP Implementation Leadership of Frontline Nurse Managers: Validation of the Implementation Leadership Scale in Acute Care.

BACKGROUND: Frontline nurse managers influence the implementation of evidence-based practices (EBP); however, there is a need for valid and reliable instruments to measure their leadership behaviors for EBP implementation in acute care settings. AIM: The aim of this study was to evaluate the validity and reliability of the Implementation Leadership Scale (ILS) in acute care settings using two unique nurse samples. METHODS: This study is a secondary analysis of ILS data obtained through two distinct multisite cross-sectional studies. Sample 1 included 200 registered nurses from one large Californian health system. Sample 2 was 284 registered nurses from seven Midwest and Northeast U.S. hospitals. Two separate studies by different research teams collected responses using written and electronic questionnaires. We analyzed each sample independently. Descriptive statistics described individual item, total, and subscale scores. We analyzed validity using confirmatory factor analysis and within-unit agreement (awg). We evaluated factorial invariance using multigroup confirmatory factor analyses and evaluating change in chi-square and comparative fit index values. We evaluated reliability using Cronbach's alpha. RESULTS: Confirmatory factor analyses in both samples provided strong support for first- and second-order factor structure of the ILS. The factor structure did not differ between the two samples. Across both samples, internal consistency reliability was strong (Cronbach's alpha: 0.91-0.98), as was within-unit agreement (awg: 0.70-0.80). LINKING EVIDENCE TO ACTION: Frontline manager implementation leadership is a critical contextual factor influencing EBP implementation. This study provides strong evidence supporting the validity and reliability of the ILS to measure implementation leadership behaviors of nursing frontline managers in acute care. The ILS can help clinicians, researchers, and leaders in nursing contexts assess frontline manager implementation leadership, deliver interventions to target areas needing improvement, and improve implementation of EBP.

Validation of the Implementation Climate Scale (ICS) in substance use disorder treatment organizations.

BACKGROUND: One critical factor in the implementation of evidence-based practice (EBP) in substance use disorder treatment organizations is an inner organizational context that clearly supports implementation efforts. The Implementation Climate Scale (ICS) has been developed to allow researchers and organizations to assess climate for EBP implementation in health and allied health service organizations. The ICS consists of 18 items and measures six dimensions of implementation climate: focus on EBP, educational support for EBP, recognition for EBP, rewards for EBP, selection for EBP, and selection for openness. The ICS was initially developed in a mental health context; thus, the goal of this study was to provide initial validation of the ICS in substance use disorder (SUD) treatment settings. METHODS: Confirmatory factor analysis (CFA) was used to assess the psychometric functioning of the ICS using survey data from 326 providers in 65 teams in SUD treatment programs. Cronbach's alpha was examined to assess internal consistency of the ICS, and individual and team level construct-based validity was examined by comparing its correlations with service climate, molar climate, and organizational change. RESULTS: We found evidence for the reliability, factor structure, and validity of the ICS in SUD services. The psychometric functioning of the ICS in SUD treatment settings was comparable to that found in mental health contexts. CONCLUSIONS: The ICS is a brief and pragmatic tool for researchers to better understand a critical antecedent for implementation effectiveness in SUD treatment and for organizational leaders in SUD treatment organizations to evaluate the extent to which providers perceive that their organization supports EBP implementation.

The Development of a Brief Version of the Posttraumatic Cognitions Inventory (PTCI-9).

Negative posttraumatic cognitions lead to the development and maintenance of posttraumatic stress disorder symptoms. There is a need for a brief measure to assess these cognitions. Participants were administered the Posttraumatic Cognitions Inventory (PTCI) and measures of mental health symptomatology. These data were used to develop a brief version of the PTCI (PTCI-9) in 223 male and female veterans, which was then examined in a sample of 117 female civilians. Confirmatory factor analyses demonstrated an acceptable fit in both samples. The PTCI-9 total and subscale scores showed strong internal consistencies (Cronbach's αs = .80-.87) and strong correlations with the PTCI in veterans ( rs = .90-.96) and civilians ( rs = .91-.96). Measurement invariance testing demonstrated partial invariance between the two samples. The PTCI-9 significantly correlated with measures of PTSD, depression, and quality of life. These findings demonstrate that the PTCI-9 is a reliable and valid measure of posttraumatic cognitions that can reduce patient and provider burden.

Leading for the long haul: a mixed-method evaluation of the Sustainment Leadership Scale (SLS).

BACKGROUND: Despite our progress in understanding the organizational context for implementation and specifically the role of leadership in implementation, its role in sustainment has received little attention. This paper took a mixed-method approach to examine leadership during the sustainment phase of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Utilizing the Implementation Leadership Scale as a foundation, we sought to develop a short, practical measure of sustainment leadership that can be used for both applied and research purposes. METHODS: Data for this study were collected as a part of a larger mixed-method study of evidence-based intervention, SafeCare®, sustainment. Quantitative data were collected from 157 providers using web-based surveys. Confirmatory factor analysis was used to examine the factor structure of the Sustainment Leadership Scale (SLS). Qualitative data were collected from 95 providers who participated in one of 15 focus groups. A framework approach guided qualitative data analysis. Mixed-method integration was also utilized to examine convergence of quantitative and qualitative findings. RESULTS: Confirmatory factor analysis supported the a priori higher order factor structure of the SLS with subscales indicating a single higher order sustainment leadership factor. The SLS demonstrated excellent internal consistency reliability. Qualitative analyses offered support for the dimensions of sustainment leadership captured by the quantitative measure, in addition to uncovering a fifth possible factor, available leadership. CONCLUSIONS: This study found qualitative and quantitative support for the pragmatic SLS measure. The SLS can be used for assessing leadership of first-level leaders to understand how staff perceive leadership during sustainment and to suggest areas where leaders could direct more attention in order to increase the likelihood that EBIs are institutionalized into the normal functioning of the organization.

Validation of the Implementation Leadership Scale (ILS) with Supervisors' Self-Ratings.

Although often discussed, there is a lack of empirical research on the role of leadership in the management and delivery of health services. The implementation leadership scale (ILS) assesses the degree to which leaders are knowledgeable, proactive, perseverant, and supportive during evidence-based practice (EBP) implementation. The purpose of this study was to examine the psychometric properties of the ILS for leaders' self-ratings using a sample of mental health clinic supervisors (N = 119). Supervisors (i.e., leaders) completed surveys including self-ratings of their implementation leadership. Confirmatory factor analysis, reliability, and validity of the ILS were evaluated. The ILS factor structure was supported in the sample of supervisors. Results demonstrated internal consistency reliability and validity. Cronbach alpha's ranged from 0.92 to 0.96 for the ILS subscales and 0.95 for the ILS overall scale. The factor structure replication and reliability of the ILS in a sample of supervisors demonstrates its applicability with employees across organizational levels.

The Evidence-based Practice Attitude Scale-36 (EBPAS-36): a brief and pragmatic measure of attitudes to evidence-based practice validated in US and Norwegian samples.

BACKGROUND: Short and valid instruments for measuring factors facilitating or hindering implementation efforts are called for. This article describes (1) the adaptation of a shorter version of the Evidence-based Practice Attitude Scale (EBPAS-50 items), and (2) the psychometric properties of the shortened version in both US and Norwegian data. METHODS: The US participants were mental health service providers (N = 418) recruited from clinics providing mental health services in San Diego County, California. The Norwegian participants were psychologists, psychiatric nurses, and psychology students (N = 838) recruited from the Norwegian Psychological Association and the Norwegian Nurses Organization. A confirmatory factor analysis (CFA) approach was used. RESULTS: The reduction resulted in 36 items named EBPAS-36, and the original 12 factor model was maintained. The EBPAS-36 had acceptable model fit, as indicated by a low degree of misspecification errors in both the US (RMSEA = .045 (CI90% .040-.049); SRMR = .05) and the Norwegian data (RMSEA = .052 (CI90% .047-.056, SRMR = .07). Incremental model fit was fair in the US (CFI = .93, TLI = .91) and in the Norwegian samples (CFI = .91, TLI = .89). The internal consistency (Cronbach's α) in the US and the Norwegian samples were good for the total EBPAS-36 score (.79 and .86, respectively) and were ranged from adequate to excellent for the subscales (US .60-.91 and Norway .61-.92). CONCLUSIONS: The EBPAS-36 has adequate psychometric properties both in US and Norwegian samples, hence indicating cross-cultural validity. It is a brief, pragmatic, and more user-friendly instrument than the EBPAS-50, yet maintains a broad scope by retaining the original 12 measurement domains.

Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol.

BACKGROUND: Evidence-based practice (EBP) implementation represents a strategic change in organizations that requires effective leadership and alignment of leadership and organizational support across organizational levels. As such, there is a need for combining leadership development with organizational strategies to support organizational climate conducive to EBP implementation. The leadership and organizational change for implementation (LOCI) intervention includes leadership training for workgroup leaders, ongoing implementation leadership coaching, 360° assessment, and strategic planning with top and middle management regarding how they can support workgroup leaders in developing a positive EBP implementation climate. METHODS: This test of the LOCI intervention will take place in conjunction with the implementation of motivational interviewing (MI) in 60 substance use disorder treatment programs in California, USA. Participants will include agency executives, 60 program leaders, and approximately 360 treatment staff. LOCI will be tested using a multiple cohort, cluster randomized trial that randomizes workgroups (i.e., programs) within agency to either LOCI or a webinar leadership training control condition in three consecutive cohorts. The LOCI intervention is 12 months, and the webinar control intervention takes place in months 1, 5, and 8, for each cohort. Web-based surveys of staff and supervisors will be used to collect data on leadership, implementation climate, provider attitudes, and citizenship. Audio recordings of counseling sessions will be coded for MI fidelity. The unit of analysis will be the workgroup, randomized by site within agency and with care taken that co-located workgroups are assigned to the same condition to avoid contamination. Hierarchical linear modeling (HLM) will be used to analyze the data to account for the nested data structure. DISCUSSION: LOCI has been developed to be a feasible and effective approach for organizations to create a positive climate and fertile context for EBP implementation. The approach seeks to cultivate and sustain both effective general and implementation leadership as well as organizational strategies and support that will remain after the study has ended. Development of a positive implementation climate for MI should result in more positive service provider attitudes and behaviors related to the use of MI and, ultimately, higher fidelity in the use of MI. TRIAL REGISTRATION: This study is registered with Clinicaltrials.gov ( NCT03042832 ), 2 February 2017, retrospectively registered.

The Humble Leader: Association of Discrepancies in Leader and Follower Ratings of Implementation Leadership With Organizational Climate in Mental Health.

OBJECTIVES: Discrepancies, or perceptual distance, between leaders' self-ratings and followers' ratings of the leader are common but usually go unrecognized. Research on discrepancies is limited, but there is evidence that discrepancies are associated with organizational context. This study examined the association of leader-follower discrepancies in Implementation Leadership Scale (ILS) ratings of mental health clinic leaders and the association of those discrepancies with organizational climate for involvement and performance feedback. Both involvement and performance feedback are important for evidence-based practice (EBP) implementation in mental health. METHODS: A total of 593 individuals-supervisors (leaders, N=80) and clinical service providers (followers, N=513)-completed surveys that included ratings of implementation leadership and organizational climate. Polynomial regression and response surface analyses were conducted to examine the associations of discrepancies in leader-follower ILS ratings with organizational involvement climate and performance feedback climate, aspects of climate likely to support EBP implementation. RESULTS: Both involvement climate and performance feedback climate were highest where leaders rated themselves low on the ILS and their followers rated those leaders high on the ILS ("humble leaders"). CONCLUSIONS: Teams with "humble leaders" showed more positive organizational climate for involvement and for performance feedback, contextual factors important during EBP implementation and sustainment. Discrepancy in leader and follower ratings of implementation leadership should be a consideration in understanding and improving leadership and organizational climate for mental health services and for EBP implementation and sustainment in mental health and other allied health settings.

The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study.

If evidence-based interventions (EBIs) are not sustained, investments are wasted and public health impact is limited. Leadership has been suggested as a key determinant of implementation and sustainment; however, little empirical work has examined this factor. This mixed-methods study framed using the Exploration, Preparation, Implementation, Sustainment (EPIS) conceptual framework examines leadership in both the outer service system context and inner organizational context in eleven system-wide implementations of the same EBI across two U.S. states and 87 counties. Quantitative data at the outer context (i.e., system) and inner context (i.e., team) levels demonstrated that leadership predicted future sustainment and differentiated between sites with full, partial, or no sustainment. In the outer context positive sustainment leadership was characterized as establishing a project's mission and vision, early and continued planning for sustainment, realistic project plans, and having alternative strategies for project survival. Inner context frontline transformational leadership predicted sustainment while passive-avoidant leadership predicted non-sustainment. Qualitative results found that sustainment was associated with outer context leadership characterized by engagement in ongoing supportive EBI championing, marketing to stakeholders; persevering in these activities; taking action to institutionalize the EBI with funding, contracting, and system improvement plans; and fostering ongoing collaboration between stakeholders at state and county, and community stakeholder levels. For frontline leadership the most important activities included championing the EBI and providing practical support for service providers. There was both convergence and expansion that identified unique contributions of the quantitative and qualitative methods. Greater attention to leadership in both the outer system and inner organizational contexts is warranted to enhance EBI implementation and sustainment.

Validation of the Implementation Leadership Scale (ILS) in Substance use Disorder Treatment Organizations.

There have been recent calls for pragmatic measures to assess factors that influence evidence-based practice (EBP) implementation processes and outcomes. The Implementation Leadership Scale (ILS) is a brief and efficient measure that can be used for research or organizational development purposes to assess leader behaviors and actions that actively support effective EBP implementation. The ILS was developed and validated in mental health settings. This study validates the ILS factor structure with providers in alcohol and other drug (AOD) use treatment agencies. Participants were 323 service providers working in 72 workgroups from three AOD use treatment agencies. Confirmatory factor analyses and reliability analyses were conducted to examine the psychometric properties of the ILS. Convergent and discriminant validity were also assessed. Confirmatory factor analyses demonstrated good fit to the hypothesized first and second order factor structure. Internal consistency reliability was excellent. Convergent and discriminant validity was supported. The ILS psychometric characteristics, reliability, and validity were supported in AOD use treatment agencies. The ILS is a brief and pragmatic measure that can be used for research and practice to assess leadership for EBP implementation in AOD use treatment agencies.

Cross-Validation of the Implementation Leadership Scale (ILS) in Child Welfare Service Organizations.

The Implementation Leadership Scale (ILS) is a brief, pragmatic, and efficient measure that can be used for research or organizational development to assess leader behaviors and actions that actively support effective implementation of evidence-based practices (EBPs). The ILS was originally validated with mental health clinicians. This study validates the ILS factor structure with providers in community-based organizations (CBOs) providing child welfare services. Participants were 214 service providers working in 12 CBOs that provide child welfare services. All participants completed the ILS, reporting on their immediate supervisor. Confirmatory factor analyses were conducted to examine the factor structure of the ILS. Internal consistency reliability and measurement invariance were also examined. Confirmatory factor analyses showed acceptable fit to the hypothesized first- and second-order factor structure. Internal consistency reliability was strong and there was partial measurement invariance for the first-order factor structure when comparing child welfare and mental health samples. The results support the use of the ILS to assess leadership for implementation of EBPs in child welfare organizations.

Validating the Implementation Climate Scale (ICS) in child welfare organizations.

There is increasing emphasis on the use of evidence-based practices (EBPs) in child welfare settings and growing recognition of the importance of the organizational environment, and the organization's climate in particular, for how employees perceive and support EBP implementation. Recently, Ehrhart, Aarons, and Farahnak (2014) reported on the development and validation of a measure of EBP implementation climate, the Implementation Climate Scale (ICS), in a sample of mental health clinicians. The ICS consists of 18 items and measures six critical dimensions of implementation climate: focus on EBP, educational support for EBP, recognition for EBP, rewards for EBP, selection or EBP, and selection for openness. The goal of the current study is to extend this work by providing evidence for the factor structure, reliability, and validity of the ICS in a sample of child welfare service providers. Survey data were collected from 215 child welfare providers across three states, 12 organizations, and 43 teams. Confirmatory factor analysis demonstrated good fit to the six-factor model and the alpha reliabilities for the overall measure and its subscales was acceptable. In addition, there was general support for the invariance of the factor structure across the child welfare and mental health sectors. In conclusion, this study provides evidence for the factor structure, reliability, and validity of the ICS measure for use in child welfare service organizations.