Surgical Sabermetrics: A Scoping Review of Technology-enhanced Assessment of Nontechnical Skills in the Operating Room.

OBJECTIVE: To evaluate the current evidence for surgical sabermetrics: digital methods of assessing surgical nontechnical skills and investigate the implications for enhancing surgical performance. BACKGROUND: Surgeons need high-quality, objective, and timely feedback to optimize performance and patient safety. Digital tools to assess nontechnical skills have the potential to reduce human bias and aid scalability. However, we do not fully understand which of the myriad of digital metrics of performance assessment have efficacy for surgeons. METHODS: A systematic review was conducted by searching PubMed, EMBASE, CINAHL, and PSYCINFO databases following PRISMA-ScR guidelines. MeSH terms and keywords included "Assessment," "Surgeons," and "Technology". Eligible studies included a digital assessment of nontechnical skills for surgeons, residents, and/or medical students within an operative context. RESULTS: From 19,229 articles screened, 81 articles met the inclusion criteria. The studies varied in surgical specialties, settings, and outcome measurements. A total of 122 distinct objective, digital metrics were utilized. Studies digitally measured at least 1 category of surgical nontechnical skill using a single (n=54) or multiple objective measures (n=27). The majority of studies utilized simulation (n=48) over live operative settings (n=32). Surgical Sabermetrics has been demonstrated to be beneficial in measuring cognitive load (n=57), situation awareness (n=24), communication (n=3), teamwork (n=13), and leadership (n=2). No studies measured intraoperative decision-making. CONCLUSIONS: The literature detailing the intersection between surgical data science and operative nontechnical skills is diverse and growing rapidly. Surgical Sabermetrics may provide a promising modifiable technique to achieve desirable outcomes for both the surgeon and the patient. This study identifies a diverse array of measurements possible with sensor devices and highlights research gaps, including the need for objective assessment of decision-making. Future studies may advance the integration of physiological sensors to provide a holistic assessment of surgical performance.

Measuring and Improving Emotional Intelligence in Surgery: A Systematic Review.

OBJECTIVE: Evaluate how emotional intelligence (EI) has been measured among surgeons and to investigate interventions implemented for improving EI. SUMMARY BACKGROUND: EI has relevant applications in surgery given its alignment with nontechnical skills. In recent years, EI has been measured in a surgical context to evaluate its relationship with measures such as surgeon burnout and the surgeon-patient relationship. METHODS: A systematic review was conducted by searching MEDLINE, EMBASE, CINAHL, and PSYCINFO databases using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. MeSH terms and keywords included "emotional intelligence," "surgery," and "surgeon." Eligible studies included an EI assessment of surgeons, surgical residents, and/or medical students within a surgical context. RESULTS: The initial search yielded 4627 articles. After duplicate removal, 4435 articles were screened by title and abstract and 49 articles proceeded to a full-text read. Three additional articles were found via hand search. A total of 37 articles were included. Studies varied in surgical specialties, settings, and outcome measurements. Most occurred in general surgery, residency programs, and utilized self-report surveys to estimate EI. Notably, EI improved in all studies utilizing an intervention. CONCLUSIONS: The literature entailing the intersection between EI and surgery is diverse but still limited. Generally, EI has been demonstrated to be beneficial in terms of overall well-being and job satisfaction while also protecting against burnout. EI skills may provide a promising modifiable target to achieve desirable outcomes for both the surgeon and the patient. Future studies may emphasize the relevance of EI in the context of surgical teamwork.

Dissecting Cardiac Surgery: A Video-based Recall Protocol to Elucidate Team Cognitive Processes in the Operating Room.

OBJECTIVE: The aim of this study was to elucidate the cognitive processes involved in surgical procedures from the perspective of different team roles (surgeon, anesthesiologist, and perfusionist) and provide a comprehensive compilation of intraoperative cognitive processes. SUMMARY BACKGROUND DATA: Nontechnical skills play a crucial role in surgical team performance and understanding the cognitive processes underlying the intraoperative phase of surgery is essential to improve patient safety in the operating room (OR). METHODS: A mixed-methods approach encompassing semistructured interviews with 9 subject-matter experts. A cognitive task analysis was built upon a hierarchical segmentation of coronary artery bypass grafting procedures and a cued-recall protocol using video vignettes was used. RESULTS: A total of 137 unique surgical cognitive processes were identified, including 33 decision points, 23 critical communications, 43 pitfalls, and 38 strategies. Self-report cognitive workload varied substantially, depending on team role and surgical step. A web-based dashboard was developed, providing an integrated visualization of team cognitive processes in the OR that allows readers to intuitively interact with the study findings. CONCLUSIONS: This study advances the current body of knowledge by making explicit relevant cognitive processes involved during the intraoperative phase of cardiac surgery from the perspective of multiple OR team members. By displaying the research findings in an interactive dashboard, we provide trainees with new knowledge in an innovative fashion that could be used to enhance learning outcomes. In addition, the approach used in the present study can be used to deeply understand the cognitive factors underlying surgical adverse events and errors in the OR.

Analysis of Dynamic Changes in Cognitive Workload During Cardiac Surgery Perfusionists' Interactions With the Cardiopulmonary Bypass Pump.

OBJECTIVE: This novel preliminary study sought to capture dynamic changes in heart rate variability (HRV) as a proxy for cognitive workload among perfusionists while operating the cardiopulmonary bypass (CPB) pump during real-life cardiac surgery. BACKGROUND: Estimations of operators' cognitive workload states in naturalistic settings have been derived using noninvasive psychophysiological measures. Effective CPB pump operation by perfusionists is critical in maintaining the patient's homeostasis during open-heart surgery. Investigation into dynamic cognitive workload fluctuations, and their relationship with performance, is lacking in the literature. METHOD: HRV and self-reported cognitive workload were collected from three Board-certified cardiac perfusionists (N = 23 cases). Five HRV components were analyzed in consecutive nonoverlapping 1-min windows from skin incision through sternal closure. Cases were annotated according to predetermined phases: prebypass, three phases during bypass, and postbypass. Values from all 1min time windows within each phase were averaged. RESULTS: Cognitive workload was at its highest during the time between initiating bypass and clamping the aorta (preclamp phase during bypass), and decreased over the course of the bypass period. CONCLUSION: We identified dynamic, temporal fluctuations in HRV among perfusionists during cardiac surgery corresponding to subjective reports of cognitive workload. Not only does cognitive workload differ for perfusionists during bypass compared with pre- and postbypass phases, but differences in HRV were also detected within the three bypass phases. APPLICATION: These preliminary findings suggest the preclamp phase of CPB pump interaction corresponds to higher cognitive workload, which may point to an area warranting further exploration using passive measurement.

Surgery Task Load Index in Cardiac Surgery: Measuring Cognitive Load Among Teams.

Background. The most commonly used subjective assessment of perceived cognitive load, the NASA Task Load Index (TLX), has proven valuable in measuring individual load among general populations. The surgery task load index (SURG-TLX) was developed and validated to measure cognitive load specifically among individuals within a surgical team. Notably, the TLX lacks temporal sensitivity in its typical retrospective administration. Objective. This study sought to expand the utility of SURG-TLX by investigating individual measures of cognitive load over time during cardiac surgery, and the relationship between individual and team measures of cognitive load and proxies for surgical complexity. Materials & Methods. SURG-TLX was administered retrospectively in the operating room immediately following each case to approximate cognitive load before, during, and after cardiopulmonary bypass for cardiac surgery team members (surgeon, anesthesiologist, and perfusionist). Correlations were calculated to determine the relationship of individual and team measures of cognitive load over the entire procedure with bypass length and surgery length. Results. Results suggest that perceived cognitive load varies throughout the procedure such that cognitive load during bypass significantly differs compared to before or after bypass, across all 3 roles. While on bypass, results show that anesthesiologists experience significantly lower levels of perceived cognitive load than both surgeons and perfusionists. Correlational analyses reveal that perceived cognitive load of both the surgeon and the team had significant positive associations with bypass length and surgery length. Conclusion. Our findings support the utility of SURG-TLX in real cardiac cases as a measure of cognitive load over time, and on an individual and team-wide basis.

Physiological synchronization and entropy as measures of team cognitive load.

The operating room (OR) is a high-risk and complex environment, where multiple specialized professionals work as a team to effectively care for patients in need of surgical interventions. Surgical tasks impose high cognitive demands on OR staff and cognitive overload may have deleterious effects on team performance and patient safety. The aim of the present study was to investigate the feasibility and describe a novel methodological approach to characterize dynamic changes in team cognitive load by measuring synchronization and entropy of heart rate variability parameters during real-life cardiac surgery. Cognitive load was measured by capturing interbeat intervals (IBI) from three team members (surgeon, anesthesiologist and perfusionist) using an unobtrusive wearable heart rate sensor and transmitted in real-time to a smartphone application. Clinical data and operating room audio/video recordings were also collected to provide behavioral and contextual information. We developed symbolic representations of the transient cognitive state of individual team members (Individual Cognitive State - ICS), and overall team (Team Cognitive State - TCS) by comparing IBI data from each team member with themselves and with others. The distribution of TCS symbols during surgery enabled us to display and analyze temporal states and dynamic changes of team cognitive load. Shannon's entropy was calculated to estimate the changing levels of team organization and to detect fluctuations resulting from a variety of cognitive demands and/or specific situations (e.g. medical error, emergency, flow disruptions). An illustrative example from a real cardiac surgery team shows how cognitive load patterns shifted rapidly after an actual near-miss medication event, leading the team to a more organized and synchronized state. The methodological approach described in this study provides a measurement technique for the assessment of team physiological synchronization, which can be applied to many other team-based environments. Future research should gather additional validity evidence to support the proposed methods for team cognitive load measurement.

Interprofessional simulation to improve safety in the epilepsy monitoring unit.

OBJECTIVE: Patient safety is critical for epilepsy monitoring units (EMUs). Effective training is important for educating all personnel, including residents and nurses who frequently cover these units. We performed a needs assessment and developed a simulation-based team training curriculum employing actual EMU sentinel events to train neurology resident-nurse interprofessional teams to maximize effective responses to high-acuity events. METHODS: A mixed-methods design was used. This included the development of a safe-practice checklist to assess team response to acute events in the EMU using expert review with consensus (a modified Delphi process). All nineteen incoming first-year neurology residents and 2 nurses completed a questionnaire assessing baseline knowledge and attitudes regarding seizure management prior to and following a team training program employing simulation and postscenario debriefing. Four resident-nurse teams were recorded while participating in two simulated scenarios. Employing retrospective video review, four trained raters used the newly developed safe-practice checklist to assess team performance. We calculated the interobserver reliability of the checklist for consistency among the raters. We attempted to ascertain whether the training led to improvement in performance in the actual EMU by comparing 10 videos of resident-nurse team responses to seizures 4-8months into the academic year preceding the curricular training to 10 that included those who received the training within 4-8months of the captured video. RESULTS: Knowledge in seizure management was significantly improved following the program, but confidence in seizure management was not. Interrater agreement was moderate to high for consistency of raters for the majority of individual checklist items. We were unable to demonstrate that the training led to sustainable improvement in performance in the actual EMU by the method we used. CONCLUSIONS: A simulated team training curriculum using a safe-practice checklist to improve the management of acute events in an EMU may be an effective method of training neurology residents. However, translating the results into sustainable benefits and confidence in management in the EMU requires further study.

Formalizing the Hidden Curriculum of Performance Enhancing Errors.

Despite its inevitability, error remains an uncomfortable topic for discussion amongst surgeons. There are a range of reasons cited for this; significantly, there is an inextricable link between a surgeon's actions and their patient's outcomes. Attempts to reflect on error are often unstructured and without a defined end point, and modern surgical curricula lack content to guide residents' learning on recognizing and reflecting on sentinel events. There is a need to develop a tool to guide a standardized, safe, and constructive response to error. The current educational paradigm revolves around error avoidance. However, there is an evolving evidence base surrounding the inclusion of error management theory (EMT) into surgical training. This method explores and incorporates positive discussions surrounding errors, and has been demonstrated to improve long-term skill acquisition and training outcomes. We must harness the performance enhancing effects of our errors in the same way we do our successes. Implicated in all surgical performance is human factors science/ergonomics (HFE) - the interface between psychology, engineering, and performance. Developing a national HFE curriculum in the context of EMT would provide a common language to facilitate objective reflections regarding surgeons' operative performance and manage the stigma associated with fallibility.

Temporally guided articulated hand pose tracking in surgical videos.

PURPOSE: Articulated hand pose tracking is an under-explored problem that carries the potential for use in an extensive number of applications, especially in the medical domain. With a robust and accurate tracking system on surgical videos, the motion dynamics and movement patterns of the hands can be captured and analyzed for many rich tasks. METHODS: In this work, we propose a novel hand pose estimation model, CondPose, which improves detection and tracking accuracy by incorporating a pose prior into its prediction. We show improvements over state-of-the-art methods which provide frame-wise independent predictions, by following a temporally guided approach that effectively leverages past predictions. RESULTS: We collect Surgical Hands, the first dataset that provides multi-instance articulated hand pose annotations for videos. Our dataset provides over 8.1k annotated hand poses from publicly available surgical videos and bounding boxes, pose annotations, and tracking IDs to enable multi-instance tracking. When evaluated on Surgical Hands, we show our method outperforms the state-of-the-art approach using mean Average Precision, to measure pose estimation accuracy, and Multiple Object Tracking Accuracy, to assess pose tracking performance. CONCLUSION: In comparison to a frame-wise independent strategy, we show greater performance in detecting and tracking hand poses and more substantial impact on localization accuracy. This has positive implications in generating more accurate representations of hands in the scene to be used for targeted downstream tasks.

Nontechnical Skills for Intraoperative Team Members.

Nontechnical skills, defined as the set of cognitive and social skills used by individuals and teams to reduce error and improve performance in complex systems, have become increasingly recognized as a key contributor to patient safety. Efforts to characterize, quantify, and teach nontechnical skills in the context of perioperative care continue to evolve. This review article summarizes the essential behaviors for safety, described in taxonomies for nontechnical skills assessments developed for intraoperative clinical team members (eg, surgeons, anesthesiologists, scrub practitioners, perfusionists). Furthermore, the authors describe emerging methods to advance understanding of the impact of nontechnical skills on perioperative outcomes.

Assessing Team Situational Awareness in the Operating Room via Computer Vision.

Situational awareness (SA) at both individual and team levels, plays a critical role in the operating room (OR). During the pre-incision time-out, the entire OR team comes together to deploy the surgical safety checklist (SSC). Worldwide, the implementation of the SSC has been shown to reduce intraoperative complications and mortality among surgical patients. In this study, we investigated the feasibility of applying computer vision analysis on surgical videos to extract team motion metrics that could differentiate teams with good SA from those with poor SA during the pre-incision time-out. We used a validated observation-based tool to assess SA, and a computer vision software to measure body position and motion patterns in the OR. Our findings showed that it is feasible to extract surgical team motion metrics captured via off-the-shelf OR cameras. Entropy as a measure of the level of team organization was able to distinguish surgical teams with good and poor SA. These findings corroborate existing studies showing that computer vision-based motion metrics have the potential to integrate traditional observation-based performance assessments in the OR.

Analysis of Mirrored Psychophysiological Change of Cardiac Surgery Team Members During Open Surgery.

OBJECTIVE: Mirrored psychophysiological change in cognitive workload indices may reflect shared mental models and effective healthcare team dynamics. In this exploratory analysis, we investigated the frequency of mirrored changes, defined as concurrent peaks in heart rate variability (HRV) across team members, during cardiac surgery. DESIGN: Objective cognitive workload was evaluated via HRV collected from the primary surgical team during cardiac surgery cases (N = 15). Root mean square of the successive differences (RMSSD) was calculated as the primary HRV measure. Procedures were divided into consecutive nonoverlapping 5-minute segments, and RMSSD along with deviations from RMSSD were calculated for each segment. Segments with positive deflections represent above-average cognitive workload. Positive deflections and peaks across dyads within the same segment were counted. SETTING: Data collection for this study took place in the cardiovascular operating room during live surgeries. PARTICIPANTS: Physiological data were collected and analyzed from the attending surgeon, attending anesthesiologist, and primary perfusionist involved with the recorded cases. RESULTS: Of the 641 five-minute segments analyzed, 325 (50.7%) were positive deflections above average, concurrently across at least 2 team members. Within the 325 positive deflections, 26 (8%) represented concurrent peaks in HRV across at least 2 active team members. Mirrored peaks across team members were observed most commonly during the coronary anastomoses or valve replacement phase (N = 12). CONCLUSIONS: In this pilot study, mirrored physiological responses representing peaks in cognitive workload were observed uncommonly across dyads of cardiac surgery team members (1.73 peaks/case on average). Almost half of these occurred during the most technically demanding phases of cardiac surgery, which may underpin teamwork quality. Future work should investigate interactions between technical and nontechnical performance surrounding times of mirrored peaks and expand the sample size.

Novel Assessments of Technical and Nontechnical Cardiac Surgery Quality: Protocol for a Mixed Methods Study.

BACKGROUND: Of the 150,000 patients annually undergoing coronary artery bypass grafting, 35% develop complications that increase mortality 5 fold and expenditure by 50%. Differences in patient risk and operative approach explain only 2% of hospital variations in some complications. The intraoperative phase remains understudied as a source of variation, despite its complexity and amenability to improvement. OBJECTIVE: The objectives of this study are to (1) investigate the relationship between peer assessments of intraoperative technical skills and nontechnical practices with risk-adjusted complication rates and (2) evaluate the feasibility of using computer-based metrics to automate the assessment of important intraoperative technical skills and nontechnical practices. METHODS: This multicenter study will use video recording, established peer assessment tools, electronic health record data, registry data, and a high-dimensional computer vision approach to (1) investigate the relationship between peer assessments of surgeon technical skills and variability in risk-adjusted patient adverse events; (2) investigate the relationship between peer assessments of intraoperative team-based nontechnical practices and variability in risk-adjusted patient adverse events; and (3) use quantitative and qualitative methods to explore the feasibility of using objective, data-driven, computer-based assessments to automate the measurement of important intraoperative determinants of risk-adjusted patient adverse events. RESULTS: The project has been funded by the National Heart, Lung and Blood Institute in 2019 (R01HL146619). Preliminary Institutional Review Board review has been completed at the University of Michigan by the Institutional Review Boards of the University of Michigan Medical School. CONCLUSIONS: We anticipate that this project will substantially increase our ability to assess determinants of variation in complication rates by specifically studying a surgeon's technical skills and operating room team member nontechnical practices. These findings may provide effective targets for future trials or quality improvement initiatives to enhance the quality and safety of cardiac surgical patient care. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/22536.

Using Machine Learning to Assess Physician Competence: A Systematic Review.

PURPOSE: To identify the different machine learning (ML) techniques that have been applied to automate physician competence assessment and evaluate how these techniques can be used to assess different competence domains in several medical specialties. METHOD: In May 2017, MEDLINE, EMBASE, PsycINFO, Web of Science, ACM Digital Library, IEEE Xplore Digital Library, PROSPERO, and Cochrane Database of Systematic Reviews were searched for articles published from inception to April 30, 2017. Studies were included if they applied at least one ML technique to assess medical students', residents', fellows', or attending physicians' competence. Information on sample size, participants, study setting and design, medical specialty, ML techniques, competence domains, outcomes, and methodological quality was extracted. MERSQI was used to evaluate quality, and a qualitative narrative synthesis of the medical specialties, ML techniques, and competence domains was conducted. RESULTS: Of 4,953 initial articles, 69 met inclusion criteria. General surgery (24; 34.8%) and radiology (15; 21.7%) were the most studied specialties; natural language processing (24; 34.8%), support vector machine (15; 21.7%), and hidden Markov models (14; 20.3%) were the ML techniques most often applied; and patient care (63; 91.3%) and medical knowledge (45; 65.2%) were the most assessed competence domains. CONCLUSIONS: A growing number of studies have attempted to apply ML techniques to physician competence assessment. Although many studies have investigated the feasibility of certain techniques, more validation research is needed. The use of ML techniques may have the potential to integrate and analyze pragmatic information that could be used in real-time assessments and interventions.

First Reported Use of Team Cognitive Workload for Root Cause Analysis in Cardiac Surgery.

Cognitive workload data of members of the cardiac surgery team can be measured intraoperatively and stored for later analysis. We present a case of a near-miss (medication error) that underwent root cause analysis using workload data. Heart rate variability data, representing workload levels, were collected from the attending surgeon, attending anesthesiologist, and lead perfusionist using wireless heart rate monitors. An episode of cognitive overload of the anesthesiologist due to a distractor was associated with the preventable error. Additional studies are needed to better understand the role of psychophysiological data in enhancing surgical patient safety.

Cognitive Support to Promote Shared Mental Models during Safety-Critical Situations in Cardiac Surgery (Late Breaking Report).

To address the, currently unmet, need for intra-operative safety-critical cognitive support in cardiac surgery, we have developed, validated, and implemented a series of customized checklists to address intra-operative emergencies, using a simulated operative setting. These crisis checklists are designed to provide cognitive and communication support to the operative team to reduce the likelihood of adverse events and improve adherence to best-practice guidelines. We recruited a number of content specialists including members of the hospital safety network and intraoperative cardiac surgery team members, and utilized a Delphi consensus method to develop procedure-specific guidelines for select intraoperative crises. Cardiac surgery team members were subsequently trained on utilizing the developed checklists, performed operative simulations, and were surveyed to determine checklist facility and effectiveness. We developed and validated five checklists for the following cardiac surgery crisis scenarios: (a) Cardiopulmonary Bypass Failure; (b) Systemic Air Embolism; (c) Venous Air Lock; (d) Protamine Reaction; Heparin Resistance. Upon initiation of the crisis management, a crew resource management approach was triggered. A member of the operative team was designated as the "reader" for each scenario to guide the team through the process. After training, 89% of operative team members surveyed indicated that they would like the crisis checklist to be used if they had one of these events occurring to them. Crisis management challenges members of the cardiac surgery team in reasoning accurately and according to best practice during periods of high cognitive workload and psychological stress. These crisis checklists were developed, validated, and simulated with the goal of supporting human performance and shared mental models in the clinical setting.

Toward Improving Surgical Outcomes by Incorporating Cognitive Load Measurement into Process-Driven Guidance.

This paper summarizes the accomplishments and recent directions of our medical safety project. Our process-based approach uses a detailed, rigorously-defined, and carefully validated process model to provide a dynamically updated, context-aware and thus, "Smart" Checklist to help process performers understand and manage their pending tasks [7]. This paper focuses on support for teams of performers, working independently as well as in close collaboration, in stressful situations that are life critical. Our recent work has three main thrusts: provide effective real-time guidance for closely collaborating teams; develop and evaluate techniques for measuring cognitive load based on biometric observations and human surveys; and, using these measurements plus analysis and discrete event process simulation, predict cognitive load throughout the process model and propose process modifications to help performers better manage high cognitive load situations. This project is a collaboration among software engineers, surgical team members, human factors researchers, and medical equipment instrumentation experts. Experimental prototype capabilities are being built and evaluated based upon process models of two cardiovascular surgery processes, Aortic Valve Replacement (AVR) and Coronary Artery Bypass Grafting (CABG). In this paper we describe our approach for each of the three research thrusts by illustrating our work for heparinization, a common subprocess of both AVR and CABG. Heparinization is a high-risk error-prone procedure that involves complex team interactions and thus highlights the importance of this work for improving patient outcomes.

Evaluating Surgical Coaching: A Mixed Methods Approach Reveals More Than Surveys Alone.

OBJECTIVE: Traditionally, surgical educators have relied upon participant survey data for the evaluation of educational interventions. However, the ability of such subjective data to completely evaluate an intervention is limited. Our objective was to compare resident and attending surgeons' self-assessments of coaching sessions from surveys with independent observations from analysis of intraoperative and postoperative coaching transcripts. DESIGN: Senior residents were video-recorded operating. Each was then coached by the operative attending in a 1:1 video review session. Teaching points made in the operating room (OR) and in post-OR coaching sessions were coded by independent observers using dialogue analysis then compared using t-tests. Participants were surveyed regarding the degree of teaching dedicated to specific topics and perceived changes in teaching level, resident comfort, educational assessments, and feedback provision between the OR and the post-OR coaching sessions. SETTING: A single, large, urban, tertiary-care academic institution. PARTICIPANTS: Ten PGY4 to 5 general surgery residents and 10 attending surgeons. RESULTS: Although the reported experiences of teaching and coaching sessions by residents and faculty were similar (Pearson correlation coefficient = 0.88), these differed significantly from independent observations. Observers found that residents initiated a greater proportion of teaching points and had more educational needs assessments during coaching, compared to the OR. However, neither residents nor attendings reported a change between the 2 environments with regard to needs assessments nor comfort with asking questions or making suggestions. The only metric on which residents, attendings, and observers agreed was the provision of feedback. CONCLUSIONS: Participants' perspectives, although considered highly reliable by traditional metrics, rarely aligned with analysis of the associated transcripts from independent observers. Independent observation showed a distinct benefit of coaching in terms of frequency and type of learning points. These findings highlight the importance of seeking different perspectives, data sources, and methodologies when evaluating clinical education interventions. Surgical education can benefit from increased use of dialogue analyses performed by independent observers, which may represent a viewpoint distinct from that obtained by survey methodology.