Evaluation of a Measurement System to Assess ICU Team Performance.

OBJECTIVE: Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs. DESIGN: Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects. SETTING: A 15-bed surgical ICU at a large academic hospital. SUBJECTS: One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises. INTERVENTIONS: No intervention was conducted for this study. MEASUREMENTS AND MAIN RESULTS: Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64-0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity. CONCLUSIONS: Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.

Smart agent system for insulin infusion protocol management: a simulation-based human factors evaluation study.

OBJECTIVE: To compare the insulin infusion management of critically ill patients by nurses using either a common standard (ie, human completion of insulin infusion protocol steps) or smart agent (SA) system that integrates the electronic health record and infusion pump and automates insulin dose selection. DESIGN: A within subjects design where participants completed 12 simulation scenarios, in 4 blocks of 3 scenarios each. Each block was performed with either the manual standard or the SA system. The initial starting condition was randomised to manual standard or SA and alternated thereafter. SETTING: A simulation-based human factors evaluation conducted at a large academic medical centre. SUBJECTS: Twenty critical care nurses. INTERVENTIONS: A systems engineering intervention, the SA, for insulin infusion management. MEASUREMENTS: The primary study outcomes were error rates and task completion times. Secondary study outcomes were perceived workload, trust in automation and system usability, all measured with previously validated scales. MAIN RESULTS: The SA system produced significantly fewer dose errors compared with manual calculation (17% (n=20) vs 0, p<0.001). Participants were significantly faster, completing the protocol using the SA system (p<0.001). Overall ratings of workload for the SA system were significantly lower than with the manual system (p<0.001). For trust ratings, there was a significant interaction between time (first or second exposure) and the system used, such that after their second exposure to the two systems, participants had significantly more trust in the SA system. Participants rated the usability of the SA system significantly higher than the manual system (p<0.001). CONCLUSIONS: A systems engineering approach jointly optimised safety, efficiency and workload considerations.

Identifying Information Resources for Patients in the Intensive Care Unit and Their Families.

BACKGROUND: Providing information to patients in intensive care units and their families is challenging. Patients often are admitted unexpectedly and experience stress and uncertainty. One source of stress has been identified as unclear, uncoordinated, or inconsistent communication and information. Despite the need for information, no centrally located, easily accessible, standardized intensive care unit education content exists. OBJECTIVE: To identify educational content for patients in the intensive care unit and their families across 4 different hospitals, develop a general content database, and organize the general content into a framework for education of patients and their families. METHODS: Educational content for patients in the intensive care units of 4 participating hospitals was collected and a gap analysis was performed. RESULTS: Key content format and categories were identified. Educational content was organized into an information pathway divided into 3 phases: intensive care unit arrival; understanding the intensive care unit and partnering in care; and intensive care unit transitions. The gap analysis revealed substantial variation in content format and categories. CONCLUSIONS: Structuring a digital learning center using different stages of the patient's stay in the intensive care unit and placing resources in the context of an information pathway can help coordinate education for these patients and their families, and creates a consistent communication guide for clinicians as well. The optimal digital format should be considered in designing the learning center.

A systematic review of behavioural marker systems in healthcare: what do we know about their attributes, validity and application?

OBJECTIVE: Behavioural marker systems are advocated as a method for providing accurate assessments, directing feedback and determining the impact of teamwork improvement initiatives. The present article reports on the state of quality surrounding their use in healthcare and discusses the implications of these findings for future research, development and application. In doing so, this article provides a practical resource where marker systems can be selected and evaluated based on their strengths and limitations. METHODS: Four research questions framed this review: what are the attributes of behavioural marker systems? What evidence of reliability and validity exists? What skills and expertise are required for their use? How have they been applied to investigate the relationship between teamwork and other constructs? RESULTS: Behavioural markers systems are generally designed for specific work domains or tasks. They often cover similar content with inconsistent terminology, which complicates the comparison of research findings across clinical domains. Although several approaches were used to establish the reliability and validity of marker systems, the marker system literature, as a whole, requires more robust reliability and validity evidence. The impact of rater training on rater proficiency was mixed, but evidence suggests that improvements can be made over time. CONCLUSIONS: A consensus of definitions for teamwork constructs must be reached to ensure that the meaning behind behavioural measurement is understood across disciplines, work domains and task types. Future development efforts should focus on the cost effectiveness and feasibility of measurement tools including time spent training raters. Further, standards for the testing and reporting of psychometric evidence must be established. Last, a library of tools should be generated around whether the instrument measures general or domain-specific behaviours.