Utility of a Device Briefing Tool to Improve Surgical Safety.

INTRODUCTION: Clear communication around surgical device use is crucial to patient safety. We evaluated the utility of the Device Briefing Tool (DBT) as an adjunct to the Surgical Safety Checklist. METHODS: A nonrandomized, controlled pilot of the DBT was conducted with surgical teams at an academic referral center. Intervention departments used the DBT in all cases involving a surgical device for 10 wk. Utility, relative advantage, and implementation effectiveness were evaluated via surveys. Trained observers assessed adherence and team performance using the Oxford NOTECHS system. RESULTS: Of 113 individuals surveyed, 91 responded. Most respondents rated the DBT as moderately to extremely useful. Utility was greatest for complex devices (89%) and new devices (88%). Advantages included insight into the team's familiarity with devices (70%) and improved teamwork and communication (68%). Users found it unrealistic to review all device instructional materials (54%). Free text responses suggested that the DBT heightened awareness of deficiencies in device familiarity and training but lacked a clear mechanism to correct them. DBT adherence was 82%. NOTECHS scores in intervention departments improved over the course of the study but did not significantly differ from comparator departments. CONCLUSIONS: The DBT was rated highly by both surgeons and nurses. Adherence was high and we found no evidence of "checklist fatigue." Centers interested in implementing the DBT should focus on devices that are complex or new to any surgical team member. Guidance for correcting deficiencies identified by the DBT will be provided in future iterations of the tool.

Electronic health record-supported implementation of an evidence-based pathway for perioperative surgical care.

OBJECTIVES: Enhanced recovery pathways (ERPs) are evidence-based approaches to improving perioperative surgical care. However, the role of electronic health records (EHRs) in their implementation is unclear. We examine how EHRs facilitate or hinder ERP implementation. MATERIALS AND METHODS: We conducted interviews with informaticians and clinicians from US hospitals participating in an ERP implementation collaborative. We used inductive thematic analysis to analyze transcripts and categorized hospitals into 3 groups based on process measure adherence. High performers exhibited a minimum 80% adherence to 6 of 9 metrics, high improvers demonstrated significantly better adherence over 12 months, and strivers included all others. We mapped interrelationships between themes using causal loop diagrams. RESULTS: We interviewed 168 participants from 8 hospitals and found 3 thematic clusters: (1) "EHR difficulties" with the technology itself and contextual factors related to (2) "EHR enablers," and (3) "EHR barriers" in ERP implementation. Although all hospitals experienced issues, high performers and improvers successfully integrated ERPs into EHRs through a dedicated multidisciplinary team with informatics expertise. Strivers, while enacting some fixes, were unable to overcome individual resistance to EHR-supported ERPs. DISCUSSION AND CONCLUSION: We add to the literature describing the limitations of EHRs' technological capabilities to facilitate clinical workflows. We illustrate how organizational strategies around engaging motivated clinical teams with informatics training and resources, especially with dedicated technical support, moderate the extent of EHRs' support to ERP implementation, causing downstream effects for hospitals to transform technological challenges into care-improving opportunities. Early and consistent involvement of informatics expertise with frontline EHR clinician users benefited the efficiency and effectiveness of ERP implementation and sustainability.

Embedding Equity into the Hospital Incident Command System: A Narrative Review.

BACKGROUND: Disasters exacerbate health inequities, with historically marginalized populations experiencing unjust differences in health care access and outcomes. Health systems plan and respond to disasters using the Hospital Incident Command System (HICS), an organizational structure that centralizes communication and decision-making. The HICS does not have an equity role or considerations built into its standard structure. The authors conducted a narrative review to identify and summarize approaches to embedding equity into the HICS. METHODS: The peer-reviewed (PubMed, SCOPUS) and gray literature was searched for articles from high-income countries that referenced the HICS or Incident Command System (ICS) and equity, disparities, or populations that experience inequities in disasters. The primary focus of the search strategy was health care, but the research also included governmental and public health system articles. Two authors used inductive thematic analysis to assess commonalities and refined the themes based on feedback from all authors. RESULTS: The database search identified 479 unique abstracts; 76 articles underwent full-text review, and 11 were included in the final analysis. The authors found 5 articles through cited reference searching and 13 from the gray literature search, which included websites, organizations, and non-indexed journal articles. Three themes from the articles were identified: including equity specialists in the HICS, modifying systems to promote equity, and sensitivity to the local community. CONCLUSION: Several efforts to embed equity into the HICS and disaster preparedness and response were discovered. This review provides practical strategies health system leaders can include in their HICS and emergency preparedness plans to promote equity in their disaster response.

Centering Health Equity in the Implementation of the Hospital Incident Command System: A Qualitative Case Comparison Study.

OBJECTIVE: Disasters exacerbate inequities in health care. Health systems use the Hospital Incident Command System (HICS) to plan and coordinate their disaster response. This study examines how 2 health systems prioritized equity in implementing the Hospital Incident Command System (HICS) during the coronavirus disease 2019 (COVID-19) pandemic and identifies factors that influenced implementation. METHODS: This is a qualitative case comparison study, involving semi-structured interviews with 29 individuals from 2 US academic health systems. Strategies for promoting health equity were categorized by social determinants of health. The Consolidated Framework for Implementation Research (CFIR) guided analysis using a hybrid inductive-deductive approach. RESULTS: The health systems used various strategies to incorporate health equity throughout implementation, addressing all 5 social determinants of health domains. Facilitators included HICS principles, external partnerships, community relationships, senior leadership, health equity experts and networks, champions, equity-stratified data, teaming, and a culture of health equity. Barriers encompassed clarity of the equity representative role, role ambiguity for equity representatives, tokenism, competing priorities, insufficient resource allocation, and lack of preparedness. CONCLUSIONS: These findings elucidate how health systems centered equity during HICS implementation. Health systems and regulatory bodies can use these findings as a foundation to revise the HICS and move toward a more equitable disaster response.

Transforming Team Performance Through Reimplementation of the Surgical Safety Checklist.

Importance: Patient safety interventions, like the World Health Organization Surgical Safety Checklist, require effective implementation strategies to achieve meaningful results. Institutions with underperforming checklists require evidence-based guidance for reimplementing these practices to maximize their impact on patient safety. Objective: To assess the ability of a comprehensive system of safety checklist reimplementation to change behavior, enhance safety culture, and improve outcomes for surgical patients. Design, Setting, and Participants: This prospective type 2 hybrid implementation-effectiveness study took place at 2 large academic referral centers in Singapore. All operations performed at either hospital were eligible for observation. Surveys were distributed to all operating room staff. Intervention: The study team developed a comprehensive surgical safety checklist reimplementation package based on the Exploration, Preparation, Implementation, Sustainment framework. Best practices from implementation science and human factors engineering were combined to redesign the checklist. The revised instrument was reimplemented in November 2021. Main Outcomes and Measures: Implementation outcomes included penetration and fidelity. The primary effectiveness outcome was team performance, assessed by trained observers using the Oxford Non-Technical Skills (NOTECH) system before and after reimplementation. The Agency for Healthcare Research and Quality Hospital Survey on Patient Safety Culture was used to assess safety culture and observers tracked device-related interruptions (DRIs). Patient safety events, near-miss events, 30-day mortality, and serious complications were tracked for exploratory analyses. Results: Observers captured 252 cases (161 baseline and 91 end point). Penetration of the checklist was excellent at both time points, but there were significant improvements in all measures of fidelity after reimplementation. Mean NOTECHS scores increased from 37.1 to 42.4 points (4.3 point adjusted increase; 95% CI, 2.9-5.7; P < .001). DRIs decreased by 86.5% (95% CI, -22.1% to -97.8%; P = .03). Significant improvements were noted in 9 of 12 composite areas on culture of safety surveys. Exploratory analyses suggested reductions in patient safety events, mortality, and serious complications. Conclusions and Relevance: Comprehensive reimplementation of an established checklist intervention can meaningfully improve team behavior, safety culture, patient safety, and patient outcomes. Future efforts will expand the reach of this system by testing a structured guidebook coupled with light-touch implementation guidance in a variety of settings.

CheckPOINT: a simple tool to measure Surgical Safety Checklist implementation fidelity.

INTRODUCTION: The WHO Surgical Safety Checklist (SSC) is a communication tool that improves teamwork and patient outcomes. SSC effectiveness is dependent on implementation fidelity. Administrative audits fail to capture most aspects of SSC implementation fidelity (ie, team communication and engagement). Existing research tools assess behaviours during checklist performance, but were not designed for routine quality assurance and improvement. We aimed to create a simple tool to assess SSC implementation fidelity, and to test its reliability using video simulations, and usability in clinical practice. METHODS: The Checklist Performance Observation for Improvement (CheckPOINT) tool underwent two rounds of face validity testing with surgical safety experts, clinicians and quality improvement specialists. Four categories were developed: checklist adherence, communication effectiveness, attitude and engagement. We created a 90 min training programme, and four trained raters independently scored 37 video simulations using the tool. We calculated intraclass correlation coefficients (ICC) to assess inter-rater reliability (ICC>0.75 indicating excellent reliability). We then trained two observers, who tested the tool in the operating room. We interviewed the observers to determine tool usability. RESULTS: The CheckPOINT tool had excellent inter-rater reliability across SSC phases. The ICC was 0.83 (95% CI 0.67 to 0.98) for the sign-in, 0.77 (95% CI 0.63 to 0.92) for the time-out and 0.79 (95% CI 0.59 to 0.99) for the sign-out. During field testing, observers reported CheckPOINT was easy to use. In 98 operating room observations, the total median (IQR) score was 25 (23-28), checklist adherence was 7 (6-7), communication effectiveness was 6 (6-7), attitude was 6 (6-7) and engagement was 6 (5-7). CONCLUSIONS: CheckPOINT is a simple and reliable tool to assess SSC implementation fidelity and identify areas of focus for improvement efforts. Although CheckPOINT would benefit from further testing, it offers a low-resource alternative to existing research tools and captures elements of adherence and team behaviours.

Adapting the World Health Organization's Surgical Safety Checklist to High-Income Settings: A Hybrid Effectiveness-Implementation Trial Protocol.

Objectives: The proposed study aims to assess users' perceptions of a surgical safety checklist (SSC) reimplementation toolkit and its impact on SSC attitudes and operating room (OR) culture, meaningful checklist use, measures of surgical safety, and OR efficiency at 3 different hospital sites. Background: The High-Performance Checklist toolkit (toolkit) assists surgical teams in modifying and implementing or reimplementing the World Health Organization's SSC. Through the explore, prepare, implement, and sustain implementation framework, the toolkit provides a process and set of tools to facilitate surgical teams' modification, implementation, training on, and evaluation of the SSC. Methods: A pre-post intervention design will be used to assess the impact of the modified SSC on surgical processes, team culture, patient experience, and safety. This mixed-methods study includes quantitative and qualitative data derived from surveys, semi-structured interviews, patient focus groups, and SSC performance observations. Additionally, patient outcome and OR efficiency data will be collected from the study sites' health surveillance systems. Data analysis: Statistical data will be analyzed using Statistical Product and Service Solutions, while qualitative data will be analyzed thematically using NVivo. Furthermore, interview data will be analyzed using the Consolidated Framework for Implementation Research and reach, effectiveness, adoption, implementation, maintenance implementation frameworks. Setting: The toolkit will be introduced at 3 diverse surgical sites in Alberta, Canada: an urban hospital, university hospital, and small regional hospital. Anticipated impact: We anticipate the results of this study will optimize SSC usage at the participating surgical sites, help shape and refine the toolkit, and improve its usability and application at future sites.

Going beyond compliance: A qualitative study of the practice of surgical safety checklist.

BACKGROUND: The World Health Organization Surgical Safety Checklist (SSC) is a tool designed to enhance team communication and patient safety. When used properly, the SSC acts as a layer of defence against never events. In this study, we performed secondary qualitative analysis of operating theatres (OT) SSC observational notes to examine how the SSC was used after an intensive SSC re-implementation effort and drew on relevant theories to shed light on the observed patterns of behaviours. We aimed to go beyond assessing checklist compliance and to understand potential sociopsychological mechanisms of the variations in SSC practices. METHODS: Direct observation notes of 109 surgical procedures across 13 surgical disciplines were made by two trained nurses in the OT of a large tertiary hospital in Singapore from February to April 2022, three months after SSC re-implementation. Only notes relevant to the use of SSC were extracted and analyzed using reflexive thematic analysis. Data were coded following an inductive process to identify themes or patterns of SSC practices. These patterns were subsequently interpreted against a relevant theory to appreciate the potential sociopsychological forces behind them. RESULTS: Two broad types of SSC practices and their respective sub-themes were identified. Type 1 (vs. Type 2) SSC practices are characterized by patience and thoroughness (vs. hurriedness and omission) in carrying out the SSC process, dedication and attention (vs. delegation and distraction) to the SSC safety checks, and frequent (vs. absence of) safety voices during the conduct of SSC. These patterns were conceptualized as safety-seeking action vs. ritualistic action using Merton's social deviance theory. CONCLUSION: Ritualistic practice of the SSC can undermine surgical safety by creating conditions conducive to never events. To fully realize the SSC's potential as an essential tool for communication and safety, a concerted effort is needed to balance thoroughness with efficiency. Additionally, fostering a culture of collaboration and collegiality is crucial to reinforce and enhance the culture of surgical safety.

Reducing the Risk of Delayed Colorectal Cancer Diagnoses Through an Ambulatory Safety Net Collaborative.

BACKGROUND: An estimated 12 million adults in the United States experience delayed diagnoses and other diagnostic errors annually. Ambulatory safety nets (ASNs) are an intervention to reduce delayed diagnoses by identifying patients with abnormal results overdue for follow-up using registries, workflow redesign, and patient navigation. The authors sought to co-design a collaborative and implement colorectal cancer (CRC) ASNs across various health care settings. METHODS: A working group was convened to co-design implementation guidance, measures, and the collaborative model. Collaborative sites were recruited through a medical professional liability insurance program and chose to begin with developing an ASN for positive at-home CRC screening or overdue surveillance colonoscopy. The 18-month Breakthrough Series Collaborative ran from January 2022 to July 2023, with sites continuing to collect data while sustaining their ASNs. Data were collected from sites monthly on patients in the ASN, including the proportion that was successfully contacted, scheduled, and completed a follow-up colonoscopy. RESULTS: Six sites participated; four had an operational ASN at the end of the Breakthrough Series, with the remaining sites launching three months later. From October 2022 through February 2024, the Collaborative ASNs collectively identified 5,165 patients from the registry as needing outreach. Among patients needing outreach, 3,555 (68.8%) were successfully contacted, 2,060 (39.9%) were scheduled for a colonoscopy, and 1,504 (29.1%) completed their colonoscopy. CONCLUSION: The Collaborative successfully identified patients with previously abnormal CRC screening and facilitated completion of follow-up testing. The CRC ASN Implementation Guide offers a comprehensive road map for health care leaders interested in implementing CRC ASNs.

The Patient Safety Adoption Framework: A Practical Framework to Bridge the Know-Do Gap.

OBJECTIVES: Many patient safety initiatives fail to be adopted and implemented, even when proven effective. This creates the well-recognized know-do gap, referring to the discrepancy between what healthcare workers know should be done based on evidence and what takes place in practice. We aimed to develop a framework to improve the adoption and implementation of patient safety initiatives. METHODS: We conducted a background literature review followed by qualitative interviews with patient safety leaders to identify barriers and facilitators to adoption and implementation. Inductive thematic analysis was used to generate themes that informed the development of the framework. We used a consensus-building approach to co-create the framework and guidance tool with an Ad Hoc Committee made up of subject-matter experts and patient family advisors. The framework was tested for utility, feasibility, and acceptability through qualitative interviews. RESULTS: The Patient Safety Adoption Framework contains 5 domains and 6 subdomains. The domains are leadership (subdomains: prioritization, accountability, governance), culture and context, process (subdomains: co-creation, high reliability, engagement), meaningful measurement, and person-centeredness. A guidance tool was developed to provide practical guidance for improvement teams using the framework. Testing affirmed the framework and guidance tool with a high degree of acceptability, feasibility, and utility among implementers and subject-matter experts. CONCLUSIONS: The Patient Safety Adoption Framework provides the essential components necessary to facilitate the adoption and implementation of patient safety initiatives. The framework offers a roadmap for healthcare organizations striving to close the know-do gap.

Creating a high-performance surgical safety checklist: A multimodal evaluation plan to reinvigorate the checklist.

RATIONALE, AIMS, AND OBJECTIVES: The WHO Surgical Safety Checklist is a communication tool designed to improve surgical safety processes and enhance teamwork. It has been widely adopted since its introduction over ten years ago. As surgical safety needs evolve, organizations should periodically review and update their checklists. A holistic evaluation of the checklist in the context of an organization is the first step to making informed updates. In this article, we describe a comprehensive but feasible strategy for checklist evaluation which we developed and implemented as part of a surgical safety initiative in a high-performing center. METHODS: A three-part evaluation plan was developed and carried out by a multidisciplinary team. The evaluation included assessment of 1. Quality of care through a review of surgical safety events; 2. Safety culture through a validated survey and informal feedback; and 3. Checklist performance through direct observations and a staff survey. To prepare for re-implementation the current institutional checklist was critically evaluated and a context assessment survey was administered to surgical staff. RESULTS: The evaluation revealed challenges in communication and teamwork, with surgical staff often perceived to be working in silos. The quality of care assessment indicated room for improvement in safety processes. Deficiencies in the safety culture measures of communication and feedback shed light on an overall lack of engagement with the checklist. Checklist performance demonstrated good adherence to the items on the checklist but limited engagement by the surgical team and minimal communication between subteams. These findings informed our revisions to the checklist and its implementation processes. CONCLUSIONS: We developed and implemented a comprehensive, scalable approach to checklist evaluation which directly informed improvements to the checklist that were tailored to the organization's current context. Organizations can apply this framework to breathe new life into their checklist and transform their safety culture.

Defining re-implementation.

BACKGROUND: The first attempt to implement a new tool or practice does not always lead to the desired outcome. Re-implementation, which we define as the systematic process of reintroducing an intervention in the same environment, often with some degree of modification, offers another chance at implementation with the opportunity to address failures, modify, and ultimately achieve the desired outcomes. This article proposes a definition and taxonomy for re-implementation informed by case examples in the literature. MAIN BODY: We conducted a scoping review of the literature for cases that describe re-implementation in concept or practice. We used an iterative process to identify our search terms, pilot testing synonyms or phrases related to re-implementation. We searched PubMed and CINAHL, including articles that described implementing an intervention in the same environment where it had already been implemented. We excluded articles that were policy-focused or described incremental changes as part of a rapid learning cycle, efforts to spread, or a stalled implementation. We assessed for commonalities among cases and conducted a thematic analysis on the circumstance in which re-implementation occurred. A total of 15 articles representing 11 distinct cases met our inclusion criteria. We identified three types of circumstances where re-implementation occurs: (1) failed implementation, where the intervention is appropriate, but the implementation process is ineffective, failing to result in the intended changes; (2) flawed intervention, where modifications to the intervention itself are required either because the tool or process is ineffective or requires tailoring to the needs and/or context of the setting where it is used; and (3) unsustained intervention, where the initially successful implementation of an intervention fails to be sustained. These three circumstances often co-exist; however, there are unique considerations and strategies for each type that can be applied to re-implementation. CONCLUSIONS: Re-implementation occurs in implementation practice but has not been consistently labeled or described in the literature. Defining and describing re-implementation offers a framework for implementation practitioners embarking on a re-implementation effort and a starting point for further research to bridge the gap between practice and science into this unexplored part of implementation.

Implementation of a Device Briefing Tool reduces interruptions in surgery: A nonrandomized controlled pilot trial.

BACKGROUND: Interruptions in operative flow are known to increase team stress and errors in the operating room. Device-related interruptions are an increasing area of focus for surgical safety, but common safety processes such as the Surgical Safety Checklist do not adequately address surgical devices. We assessed the impact of the Device Briefing Tool, a communication instrument for surgical teams, on device-related interruptions in a large academic referral center in Singapore. METHODS: The Device Briefing Tool was implemented in 4 general surgery departments, with 4 additional departments serving as a comparator group. Trained observers evaluated device-related interruption incidence in live operations at baseline and after implementation. Changes in device-related interruption frequency were assessed in each group using Poisson regression, with and without adjustment for surgical department and device complexity. Subgroup analyses assessed the impact of the Device Briefing Tool by device type. RESULTS: A total of 210 operations were evaluated by observers. In the Device Briefing Tool group, there were 38.6 and 27.2 device-related interruptions per 100 cases at baseline and after Device Briefing Tool implementation, respectively (difference -23%, P = .0047, adjusted difference -28%, P = .0013). Device-related interruption frequency in the comparator group remained stable across study periods. Point estimates indicated reductions in device-related interruptions for all device types, reaching statistical significance for circular staplers (-26%, P = .0049). CONCLUSION: Implementation of the Device Briefing Tool was associated with a 28% reduction in device-related interruptions. Proactive approaches to improving surgical device safety are crucial in the technology-driven landscape of modern surgical care. Future efforts will assess formal integration of the Device Briefing Tool into institution-wide surgical safety processes.

Non-technical skills in surgery during the COVID-19 pandemic: An observational study.

BACKGROUND: Non-technical skills are critical to surgical safety. We examined the impact of the COVID-19 pandemic on non-technical skills of operating room (OR) teams in Singapore. MATERIALS AND METHODS: Observers rated live operations using the Oxford NOTECHS system. Pre- and post-COVID observations were captured from November 2019 to January 2020 and from January 2021 to February 2021, respectively. Scores were compared using Schuirmann's Two One-Sided Test procedure. Multivariable linear regression was used to adjust for case mix. A 10% margin of equivalence was set a priori. RESULTS: Observers rated 159 cases: 75 pre-COVID and 84 post-COVID. There were significant differences between groups in surgical department and surgeon-reported case complexity (both P < 0.001). Total NOTECHS scores increased post-COVID on raw analysis (36.1 vs 38.0, P < 0.001) but remained within the margin of equivalence (90% CI 1.3 to 2.6, P < 0.001). Multivariable analysis demonstrated a similar increase within the margin of equivalence (2.0, 90% CI 1.3 to 2.7). Teamwork and cooperation scores increased by 1.0 post-COVID (90% CI 0.8 to 1.3); all other subcomponent scores were equivalent. CONCLUSION: Non-technical skills before and after the peak of the COVID-19 pandemic were equivalent but not equal. A small but statistically significant improvement post-COVID was driven by an increase in teamwork and cooperation skills. These findings may reflect an improvement in team cohesion, which has been observed in teams under duress in other settings such as the military. Future work should explore the effect of the pandemic on OR culture, team cohesion, and resilience.

Frequency of Device-Related Interruptions Using a Scalable Assessment Tool.

BACKGROUND: Surgical devices are implicated in approximately 15% of intraoperative interruptions and 25% of errors. Device-related interruptions (DRIs) are therefore an important target for surgical quality improvement, but scalable measurement methodologies are lacking. The researchers therefore developed, pilot tested, and refined a simple tool for assessing intraoperative DRIs. METHODS: Five DRI categories achieved face validity with frontline providers and surgical safety experts: improper/challenging assembly, device failure, loss of sterility, disconnection, and absent/wrong device. A data collection tool was created based on these categories as well as a free-text section to capture emergent DRI categories. After a brief training session, the tool was pilot tested by observers at a large academic referral center. RESULTS: In a sample of 210 operations, observers noted 66 DRIs across 39 cases. DRIs were most common in colorectal (38.0 per 100 cases), gynecologic (33.3 per 100 cases), and hepatopancreatobiliary surgery (32.1 per 100 cases). Device failure accounted for 30.3% of observed DRIs. Three emergent categories were identified: user unfamiliarity with the device (15.2%), video display malfunction (4.5%), and physical breakage of the device (1.5%). CONCLUSION: Measurement of DRIs by novice observers is a feasible and scalable approach to support quality improvement efforts focusing on surgical devices. This approach could provide actionable insights to improve device safety, such as informing educational and training programs, optimizing surgical tray composition, and improving the physical layout of the operating room.

Virtual non-technical skills assessment training is an effective, scalable approach for novice raters.

OBJECTIVES: The COVID-19 pandemic has forced a creative transition to virtual platforms due to physical distancing and travel restrictions. We designed and tested a highly scalable virtual training curriculum for novice raters using the Oxford NOTECHS non-technical skills rating system. DESIGN: A three-day training course comprising virtual didactics, virtually facilitated simulations, and independent live observations was implemented. NOTECHS scores were submitted for eleven standardized video simulations and four live operations. Intraclass correlation coefficients (ICCs) were calculated for total NOTECHS scores and subcomponent scores. Raters previously trained in-person with the same standardized videos served as a comparator group for equivalence testing. SETTING: All study activities were conducted in a large academic tertiary referral center in Singapore as part of an ongoing surgical safety initiative. PARTICIPANTS: Seven staff members underwent training (three virtually and four in-person).  None had prior surgical experience or non-technical skills assessment training. RESULTS: ICCs for total NOTECHS scores were 0.85 (95% CI, 0.73-0.98) for virtually trained raters and 0.83 for those trained in-person (95% CI, 0.68-0.99).  Scores were equivalent between groups within a 10% margin. CONCLUSIONS: Non-technical skills assessment can be reliably taught in a highly scalable virtual format. Virtual NOTECHS training is a valuable tool for educational and quality improvement initiatives during the COVID-19 pandemic and for centers that lack ready access to onsite non-technical skills training expertise.