Data-driven coaching to improve statewide outcomes in CABG: before and after interventional study.

BACKGROUND: The impact of quality improvement initiatives program (QIP) on coronary artery bypass grafting surgery (CABG) remains scarce, despite improved outcomes in other surgical areas. This study aims to evaluate the impact of a package of QIP on mortality rates among patients undergoing CABG. MATERIALS AND METHODS: This prospective cohort study utilized data from the multicenter database Registro Paulista de Cirurgia Cardiovascular II (REPLICCAR II), spanning from July 2017 to June 2019. Data from 4018 isolated CABG adult patients were collected and analyzed in three phases: before-implementation, implementation, and after-implementation of the intervention (which comprised QIP training for the hospital team). Propensity Score Matching was used to balance the groups of 2170 patients each for a comparative analysis of the following outcomes: reoperation, deep sternal wound infection/mediastinitis ≤30 days, cerebrovascular accident, acute kidney injury, ventilation time >24 h, length of stay <6 days, length of stay >14 days, morbidity and mortality, and operative mortality. A multiple regression model was constructed to predict mortality outcomes. RESULTS: Following implementation, there was a significant reduction of operative mortality (61.7%, P =0.046), as well as deep sternal wound infection/mediastinitis ( P <0.001), sepsis ( P =0.002), ventilation time in hours ( P <0.001), prolonged ventilation time ( P =0.009), postoperative peak blood glucose ( P <0.001), total length of hospital stay ( P <0.001). Additionally, there was a greater use of arterial grafts, including internal thoracic ( P <0.001) and radial ( P =0.038), along with a higher rate of skeletonized dissection of the internal thoracic artery. CONCLUSIONS: QIP was associated with a 61.7% reduction in operative mortality following CABG. Although not all complications exhibited a decline, the reduction in mortality suggests a possible decrease in failure to rescue during the after-implementation period.

Surgical Sabermetrics: A Scoping Review of Technology-enhanced Assessment of Nontechnical Skills in the Operating Room.

OBJECTIVE: To evaluate the current evidence for surgical sabermetrics: digital methods of assessing surgical nontechnical skills and investigate the implications for enhancing surgical performance. BACKGROUND: Surgeons need high-quality, objective, and timely feedback to optimize performance and patient safety. Digital tools to assess nontechnical skills have the potential to reduce human bias and aid scalability. However, we do not fully understand which of the myriad of digital metrics of performance assessment have efficacy for surgeons. METHODS: A systematic review was conducted by searching PubMed, EMBASE, CINAHL, and PSYCINFO databases following PRISMA-ScR guidelines. MeSH terms and keywords included "Assessment," "Surgeons," and "Technology". Eligible studies included a digital assessment of nontechnical skills for surgeons, residents, and/or medical students within an operative context. RESULTS: From 19,229 articles screened, 81 articles met the inclusion criteria. The studies varied in surgical specialties, settings, and outcome measurements. A total of 122 distinct objective, digital metrics were utilized. Studies digitally measured at least 1 category of surgical nontechnical skill using a single (n=54) or multiple objective measures (n=27). The majority of studies utilized simulation (n=48) over live operative settings (n=32). Surgical Sabermetrics has been demonstrated to be beneficial in measuring cognitive load (n=57), situation awareness (n=24), communication (n=3), teamwork (n=13), and leadership (n=2). No studies measured intraoperative decision-making. CONCLUSIONS: The literature detailing the intersection between surgical data science and operative nontechnical skills is diverse and growing rapidly. Surgical Sabermetrics may provide a promising modifiable technique to achieve desirable outcomes for both the surgeon and the patient. This study identifies a diverse array of measurements possible with sensor devices and highlights research gaps, including the need for objective assessment of decision-making. Future studies may advance the integration of physiological sensors to provide a holistic assessment of surgical performance.

Nontechnical Skills for Intraoperative Team Members.

Nontechnical skills, defined as the set of cognitive and social skills used by individuals and teams to reduce error and improve performance in complex systems, have become increasingly recognized as a key contributor to patient safety. Efforts to characterize, quantify, and teach nontechnical skills in the context of perioperative care continue to evolve. This review article summarizes the essential behaviors for safety, described in taxonomies for nontechnical skills assessments developed for intraoperative clinical team members (eg, surgeons, anesthesiologists, scrub practitioners, perfusionists). Furthermore, the authors describe emerging methods to advance understanding of the impact of nontechnical skills on perioperative outcomes.

Implementing ultrasound-guided nerve blocks in the emergency department: A low-cost, low-fidelity training approach.

Background: Managing acute pain is a common challenge in the emergency department (ED). Though widely used in perioperative settings, ED-based ultrasound-guided nerve blocks (UGNBs) have been slow to gain traction. Here, we develop a low-cost, low-fidelity, simulation-based training curriculum in UGNBs for emergency physicians to improve procedural competence and confidence. Methods: In this pre-/postintervention study, ED physicians were enrolled to participate in a 2-h, in-person simulation training session composed of a didactic session followed by rotation through stations using handmade pork-based UGNB models. Learner confidence with performing and supervising UGNBs as well as knowledge and procedural-based competence were assessed pre- and posttraining via electronic survey quizzes. One-way repeated-measures ANOVAs and pairwise comparisons were conducted. The numbers of nerve blocks performed clinically in the department pre- and postintervention were compared. Results: In total, 36 participants enrolled in training sessions, eight participants completed surveys at all three data collection time points. Of enrolled participants, 56% were trainees, 39% were faculty, 56% were female, and 53% self-identified as White. Knowledge and competency scores increased immediately postintervention (mean ± SD t0 score 66.9 ± 8.9 vs. t1 score 90.4 ± 11.7; p < 0.001), and decreased 3 months postintervention but remained elevated above baseline (t2 scores 77.2 ± 11.5, compared to t0; p = 0.03). Self-reported confidence in performing UGNBs increased posttraining (t0 5.0 ± 2.3 compared to t1 score 7.1 ± 1.5; p = 0.002) but decreased to baseline levels 3 months postintervention (t2 = 6.0 ± 1.9, compared to t0; p = 0.30). Conclusions: A low-cost, low-fidelity simulation curriculum can improve ED provider procedural-based competence and confidence in performing UGNBs in the short term, with a trend toward sustained improvement in knowledge and confidence. Curriculum adjustments to achieve sustained improvement in confidence performing and supervising UGNBs long term are key to increased ED-based UGNB use.

Using Digital Biomarkers for Objective Assessment of Perfusionists' Workload and Acute Stress During Cardiac Surgery.

The cardiac operating room (OR) is a high-risk, high-stakes environment inserted into a complex socio-technical healthcare system. During cardiopulmonary bypass (CPB), the most critical phase of cardiac surgery, the perfusionist has a crucial role within the interprofessional OR team, being responsible for optimizing patient perfusion while coordinating other tasks with the surgeon, anesthesiologist, and nurses. The aim of this study was to investigate objective digital biomarkers of perfusionists' workload and stress derived from heart rate variability (HRV) metrics captured via a wearable physiological sensor in a real cardiac OR. We explored the relationships between several HRV parameters and validated self-report measures of surgical task workload (SURG-TLX) and acute stress (STAI-SF), as well as surgical processes and outcome measures. We found that the frequency-domain HRV parameter HF relative power - FFT (%) presented the strongest association with task workload (correlation coefficient: -0.491, p-value: 0.003). We also found that the time-domain HRV parameter RMSSD (ms) presented the strongest correlation with perfusionists' acute stress (correlation coefficient: -0.489, p-value: 0.005). A few workload and stress biomarkers were also associated with bypass time and patient length of stay in the hospital. The findings from this study will inform future research regarding which HRV-based biomarkers are best suited for the development of cognitive support systems capable of monitoring surgical workload and stress in real time.

The impact of a smartphone-based cognitive aid on clinical performance during cardiac arrest simulations: A randomized controlled trial.

Objectives: In-hospital cardiac arrests are common and associated with high mortality. Smartphone applications offer quick access to algorithms and timers but often lack real-time guidance. This study assesses the impact of the Code Blue Leader application on the performance of providers leading cardiac arrest simulations. Methods: This open-label randomized controlled trial included Advanced Cardiac Life Support (ACLS)-trained medical doctors (MD) and registered nurses (RN). Participants were randomized to lead the same ACLS simulation with or without the app. The primary outcome, "performance score," was assessed by a trained rater using a validated ACLS scoring system. Secondary outcomes included percentage of critical actions performed, number of incorrect actions, and chest compression fraction (percentage of time spent performing chest compressions). A sample size of 30 participants was calculated to detect a difference of 20% at the 0.05 alpha level with 90% power. Results: Fifteen MDs and 15 RNs underwent stratified randomization. The median (interquartile range) performance score in the app group was 95.3% (93.0%-100.0%) compared to 81.4% (60.5%-88.4%) in the control group, demonstrating an effect size of r = 0.69 (Z = -3.78, r = 0.69, p = 0.0002). The percentage of critical actions performed in the app group was 100% (96.2%-100.0%) compared to 85.0% (74.1%-92.4%) in the control group. The number of incorrect actions performed in the app group was 1 (1) compared to 4 (3-5) in the control group. Chest compression fraction in the app group was 75.5% (73.0%-84.0%) compared to 75.0% (72.0%-85.0%) in the control group. Conclusions: The Code Blue Leader smartphone app significantly improved the performance of ACLS-trained providers in cardiac arrest simulations.

An Open-Source, Interoperable Architecture for Generating Real-Time Surgical Team Cognitive Alerts from Heart-Rate Variability Monitoring.

Clinical alarm and decision support systems that lack clinical context may create non-actionable nuisance alarms that are not clinically relevant and can cause distractions during the most difficult moments of a surgery. We present a novel, interoperable, real-time system for adding contextual awareness to clinical systems by monitoring the heart-rate variability (HRV) of clinical team members. We designed an architecture for real-time capture, analysis, and presentation of HRV data from multiple clinicians and implemented this architecture as an application and device interfaces on the open-source OpenICE interoperability platform. In this work, we extend OpenICE with new capabilities to support the needs of the context-aware OR including a modularized data pipeline for simultaneously processing real-time electrocardiographic (ECG) waveforms from multiple clinicians to create estimates of their individual cognitive load. The system is built with standardized interfaces that allow for free interchange of software and hardware components including sensor devices, ECG filtering and beat detection algorithms, HRV metric calculations, and individual and team alerts based on changes in metrics. By integrating contextual cues and team member state into a unified process model, we believe future clinical applications will be able to emulate some of these behaviors to provide context-aware information to improve the safety and quality of surgical interventions.

Extended Reality Applications for Space Health.

INTRODUCTION: Spaceflight has detrimental effects on human health, imposing significant and unique risks to crewmembers due to physiological adaptations, exposure to physical and psychological stressors, and limited capabilities to provide medical care. Previous research has proposed and evaluated several strategies to support and mitigate the risks related to astronauts' health and medical exploration capabilities. Among these, extended reality (XR) technologies, including augmented reality (AR), virtual reality (VR), and mixed reality (MR) have increasingly been adopted for training, real-time clinical, and operational support in both terrestrial and aerospace settings, and only a few studies have reported research results on the applications of XR technologies for improving space health. This study aims to systematically review the scientific literature that has explored the application of XR technologies in the space health field. We also discuss the methodological and design characteristics of the existing studies in this realm, informing future research and development efforts on applying XR technologies to improve space health and enhance crew safety and performance.Ebnali M, Paladugu P, Miccile C, Park SH, Burian B, Yule S, Dias RD. Extended reality applications for space health. Aerosp Med Hum Perform. 2023; 94(3):122-130.

A tool to assess nontechnical skills of perfusionists in the cardiac operating room.

OBJECTIVES: This study aimed to develop the Perfusionists' Intraoperative Non-Technical Skills tool, specifically to the perfusionists' context, and test its inter-rater reliability. METHODS: An expert panel was convened to review existing surgical nontechnical skills taxonomies and develop the Perfusionists' Intraoperative Non-Technical Skills tool. During a workshop held at a national meeting, perfusionists completed the Perfusionists' Intraoperative Non-Technical Skills ratings after watching 4 videos displaying simulated cardiac operations. Two videos showed "good performance," and 2 videos showed "poor performance." Inter-rater reliability analysis was performed and intraclass correlation coefficient was reported. RESULTS: The final version of the Perfusionists' Intraoperative Non-Technical Skills taxonomy contains 4 behavioral categories (decision making, situation awareness, task management and leadership, teamwork and communication) with 4 behavioral elements each. Categories and elements are rated using an 8-point Likert scale ranging from 0.5 to 4.0. A total of 60 perfusionist raters were included and the comparison between rating distribution on "poor performance" and "good performance" videos yielded a statistically significant difference between groups, with a P value less than .001. A similar difference was found in all behavioral categories and elements. Reliability analysis showed moderate inter-rater reliability across overall ratings (intraclass correlation coefficient, 0.735; 95% confidence interval, 0.674-0.796; P < .001). Similar inter-rater reliability was found when raters were stratified by experience level. CONCLUSIONS: The Perfusionists' Intraoperative Non-Technical Skills tool presented moderate inter-rater reliability among perfusionists with varied levels of experience. This tool can be used to train and assess perfusionists in relevant nontechnical skills, with the potential to enhance safety and improve surgical outcomes.

Crew Autonomy During Simulated Medical Event Management on Long Duration Space Exploration Missions.

OBJECTIVE: Our primary aim was to investigate crew performance during medical emergencies with and without ground-support from a flight surgeon located at mission control. BACKGROUND: There are gaps in knowledge regarding the potential for unanticipated in-flight medical events to affect crew health and capacity, and potentially compromise mission success. Additionally, ground support may be impaired or periodically absent during long duration missions. METHOD: We reviewed video recordings of 16 three-person flight crews each managing four unique medical events in a fully immersive spacecraft simulator. Crews were randomized to two conditions: with and without telemedical flight surgeon (FS) support. We assessed differences in technical performance, behavioral skills, and cognitive load between groups. RESULTS: Crews with FS support performed better clinically, were rated higher on technical skills, and completed more clinical tasks from the medical checklists than crews without FS support. Crews with FS support also had better behavioral/non-technical skills (information exchange) and reported significantly lower cognitive demand during the medical event scenarios on the NASA-TLX scale, particularly in mental demand and temporal demand. There was no significant difference between groups in time to treat or in objective measures of cognitive demand derived from heart rate variability and electroencephalography. CONCLUSION: Medical checklists are necessary but not sufficient to support high levels of autonomous crew performance in the absence of real-time flight surgeon support. APPLICATION: Potential applications of this research include developing ground-based and in-flight training countermeasures; informing policy regarding autonomous spaceflight, and design of autonomous clinical decision support systems.

Temporally guided articulated hand pose tracking in surgical videos.

PURPOSE: Articulated hand pose tracking is an under-explored problem that carries the potential for use in an extensive number of applications, especially in the medical domain. With a robust and accurate tracking system on surgical videos, the motion dynamics and movement patterns of the hands can be captured and analyzed for many rich tasks. METHODS: In this work, we propose a novel hand pose estimation model, CondPose, which improves detection and tracking accuracy by incorporating a pose prior into its prediction. We show improvements over state-of-the-art methods which provide frame-wise independent predictions, by following a temporally guided approach that effectively leverages past predictions. RESULTS: We collect Surgical Hands, the first dataset that provides multi-instance articulated hand pose annotations for videos. Our dataset provides over 8.1k annotated hand poses from publicly available surgical videos and bounding boxes, pose annotations, and tracking IDs to enable multi-instance tracking. When evaluated on Surgical Hands, we show our method outperforms the state-of-the-art approach using mean Average Precision, to measure pose estimation accuracy, and Multiple Object Tracking Accuracy, to assess pose tracking performance. CONCLUSION: In comparison to a frame-wise independent strategy, we show greater performance in detecting and tracking hand poses and more substantial impact on localization accuracy. This has positive implications in generating more accurate representations of hands in the scene to be used for targeted downstream tasks.

Association Between Operating Room Noise and Team Cognitive Workload in Cardiac Surgery.

Excessive intra-operative noise in cardiac surgery has the potential to serve as source of distraction and additional cognitive workload for the surgical team, and may interfere with optimal performance. The separation from bypass phase is a technically complex phase of surgery, making it highly susceptible to communication breakdowns due to high cognitive demands and requiring tightly coupled team coordination. The objective of this study was to investigate team cognitive workload levels and communication in relation to intra-operative time periods representative of infrequent vs. frequent peaks in ambient noise. Compared to 5-minute segments with no peaks in noise at all, segments with the highest percentage of noise peaks (≥10%) were significantly associated with higher team members' heart rate before, during, and after noise segments analyzed. These noisier segments were also associated with a significantly higher level of case-irrelevant communication events. These data suggest that case-irrelevant conversations associated with a greater degree of excessive peaks in noise may be associated with team workload levels, warranting further investigation into efforts to standardize communication during critical surgical phases.

A Coding Framework for Usability Evaluation of Digital Health Technologies.

Several studies have reported low adherence and high resistance from clinicians to adopt digital health technologies into clinical practice, particularly the use of computer-based clinical decision support systems. Poor usability and lack of integration with the clinical workflow have been identified as primary issues. Few guidelines exist on how to analyze the collected data associated with the usability of digital health technologies. In this study, we aimed to develop a coding framework for the systematic evaluation of users' feedback generated during focus groups and interview sessions with clinicians, underpinned by fundamental usability principles and design components. This codebook also included a coding category to capture the user's clinical role associated with each specific piece of feedback, providing a better understanding of role-specific challenges and perspectives, as well as the level of shared understanding across the multiple clinical roles. Furthermore, a voting system was created to quantitatively inform modifications of the digital system based on usability data. As a use case, we applied this method to an electronic cognitive aid designed to improve coordination and communication in the cardiac operating room, showing that this framework is feasible and useful not only to better understand suboptimal usability aspects, but also to recommend relevant modifications in the design and development of the system from different perspectives, including clinical, technical, and usability teams. The framework described herein may be applied in other highly complex clinical settings, in which digital health systems may play an important role in improving patient care and enhancing patient safety.

Assessing Team Situational Awareness in the Operating Room via Computer Vision.

Situational awareness (SA) at both individual and team levels, plays a critical role in the operating room (OR). During the pre-incision time-out, the entire OR team comes together to deploy the surgical safety checklist (SSC). Worldwide, the implementation of the SSC has been shown to reduce intraoperative complications and mortality among surgical patients. In this study, we investigated the feasibility of applying computer vision analysis on surgical videos to extract team motion metrics that could differentiate teams with good SA from those with poor SA during the pre-incision time-out. We used a validated observation-based tool to assess SA, and a computer vision software to measure body position and motion patterns in the OR. Our findings showed that it is feasible to extract surgical team motion metrics captured via off-the-shelf OR cameras. Entropy as a measure of the level of team organization was able to distinguish surgical teams with good and poor SA. These findings corroborate existing studies showing that computer vision-based motion metrics have the potential to integrate traditional observation-based performance assessments in the OR.

Maintaining health professional education during war: A scoping review.

PURPOSE: War negatively impacts health professional education when health care is needed most. The aims of this scoping review are to describe the scope of barriers and targeted interventions to maintaining health professional education during war and summarise the research. METHODS: We conducted a scoping review between 20 June 2018 and 2 August 2018. The search was restricted to English publications including peer-reviewed publications without date ranges involving war and health professional education (medical school, residency training and nursing school), with interventions described to maintain educational activities. Two independent reviewers completed inclusion determinations and data abstraction. Thematic coding was performed using an inductive approach allowing dominant themes to emerge. The frequency of barrier and intervention themes and illustrative quotes were extracted. Articles were divided into modern/postmodern categories to permit temporal and historical analysis of thematic differences. RESULTS: Screening identified 3271 articles, with 56 studies meeting inclusion criteria. Publication dates ranged from 1914-2018 with 17 unique wars involving 17 countries. The studies concerned medical students (61.4%), residents (28.6%) and nursing students (10%). Half involved the modern era and half the postmodern era. Thematic coding identified five categories of barriers and targeted interventions in maintaining health professions education during war: curriculum, personnel, wellness, resources, and oversight, with most involving curriculum and personnel. The distribution of themes among various health professional trainees was similar. The frequency and specifics changed temporally reflecting innovations in medical education and war, with increased focus on oversight and personnel during the modern era and greater emphasis on wellness, curriculum, and resources during the postmodern era. CONCLUSIONS: There are overarching categories of barriers and targeted interventions in maintaining health professional education during war which evolve over time. These may serve as a useful framework to strategically support future research and policy efforts.

Learning NOTSS While Tying Knots: Integrating the Nontechnical Skills for Surgeons Course With Technical Surgical Skills Training, A Mixed Methods Study.

Objective: The aims of this study were to describe the process of integrating 2 established training programs, Nontechnical skills for surgeons, and a traditional essential surgical skills course and to measure the impact of this integrated course on the behaviors of interprofessional surgical teams in Rwandan district hospitals. Background: Surgical errors and resulting adverse events are due to variability in both technical and nontechnical surgical skills. Providing technical and nontechnical skills training to the perioperative team may enhance the learning of both of these skills and promote safe intraoperative patient care. Methods: A quality improvement framework guided the process of integrating essential surgical skills and nontechnical skills into a single training program for surgical teams. The resulting 2-day training program was delivered to 68-person teams from 17 hospitals. Nontechnical skills for surgeons was taught through didactics and in the operating room, where preoperative briefing, intraoperative interactions, and postoperative debriefing were used as essential and nontechnical skills teaching moments. Postcourse surveys, follow-up interviews, focus groups, and direct observation of participants in the operating room were conducted to assess how participants implemented the knowledge and skills from the training into practice. Results: Ninety-seven percent of the participants reported that they were satisfied with the course. Follow-up participant interviews and focus groups reported that the course helped them to improve their preoperative planning, intraoperative communication, decision-making, and postoperative debriefing. Conclusions: It is possible to implement an integrated essential surgical skills and nontechnical skills training course. Integrating nontechnical skills into essential surgical skills courses may enhance learning of these skills.

Measuring and Improving Emotional Intelligence in Surgery: A Systematic Review.

OBJECTIVE: Evaluate how emotional intelligence (EI) has been measured among surgeons and to investigate interventions implemented for improving EI. SUMMARY BACKGROUND: EI has relevant applications in surgery given its alignment with nontechnical skills. In recent years, EI has been measured in a surgical context to evaluate its relationship with measures such as surgeon burnout and the surgeon-patient relationship. METHODS: A systematic review was conducted by searching MEDLINE, EMBASE, CINAHL, and PSYCINFO databases using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. MeSH terms and keywords included "emotional intelligence," "surgery," and "surgeon." Eligible studies included an EI assessment of surgeons, surgical residents, and/or medical students within a surgical context. RESULTS: The initial search yielded 4627 articles. After duplicate removal, 4435 articles were screened by title and abstract and 49 articles proceeded to a full-text read. Three additional articles were found via hand search. A total of 37 articles were included. Studies varied in surgical specialties, settings, and outcome measurements. Most occurred in general surgery, residency programs, and utilized self-report surveys to estimate EI. Notably, EI improved in all studies utilizing an intervention. CONCLUSIONS: The literature entailing the intersection between EI and surgery is diverse but still limited. Generally, EI has been demonstrated to be beneficial in terms of overall well-being and job satisfaction while also protecting against burnout. EI skills may provide a promising modifiable target to achieve desirable outcomes for both the surgeon and the patient. Future studies may emphasize the relevance of EI in the context of surgical teamwork.

The Cognitive Relevance of a Formal Pre-incision Time-out in Surgery.

Surgical time-outs are designed to promote situation awareness, teamwork, and error prevention. The pre-incision time-out in particular aims to facilitate shared mental models prior to incision. Objective, unbiased measures to confirm its effectiveness are lacking. We hypothesized that providers' mental workload would reveal team psychophysiological mirroring during a formal, well-executed pre-incision time-out. Heart rate variability was collected during cardiac surgery cases from the surgeon, anesthesiologist, and perfusionist. Data were analyzed for six cases from patient arrival until sternal closure. Annotation of surgical phases was completed according to previously developed standardized process models of aortic valve replacement and coronary artery bypass graft procedures, producing thirteen total surgical phases. Statistical analysis revealed significant main effects. Tukey HSD post hoc tests revealed significant differences across provider roles within various phases, including Anesthesia Induction, Heparinization, Initiation of Bypass, Aortic Clamp and Cardioplegia, Anastomoses or Aortotomy, Separation from Bypass, and Sternal Closure. Despite these observed differences between providers over various surgical phases, the Pre-incision Time-out phase revealed almost negligible differences across roles. This preliminary work supports the utility of the pre-incision safety checklist to focus the attention of surgical team members and promote shared team mental models, measured via psychophysiological mirroring, using an objective mental workload measure. Future studies should investigate the relationship between psychophysiological mirroring among surgical team members and the effectiveness of the pre-incision time-out checklist.