Depth over RGB: automatic evaluation of open surgery skills using depth camera.

PURPOSE: In this paper, we present a novel approach to the automatic evaluation of open surgery skills using depth cameras. This work is intended to show that depth cameras achieve similar results to RGB cameras, which is the common method in the automatic evaluation of open surgery skills. Moreover, depth cameras offer advantages such as robustness to lighting variations, camera positioning, simplified data compression, and enhanced privacy, making them a promising alternative to RGB cameras. METHODS: Experts and novice surgeons completed two simulators of open suturing. We focused on hand and tool detection and action segmentation in suturing procedures. YOLOv8 was used for tool detection in RGB and depth videos. Furthermore, UVAST and MSTCN++ were used for action segmentation. Our study includes the collection and annotation of a dataset recorded with Azure Kinect. RESULTS: We demonstrated that using depth cameras in object detection and action segmentation achieves comparable results to RGB cameras. Furthermore, we analyzed 3D hand path length, revealing significant differences between experts and novice surgeons, emphasizing the potential of depth cameras in capturing surgical skills. We also investigated the influence of camera angles on measurement accuracy, highlighting the advantages of 3D cameras in providing a more accurate representation of hand movements. CONCLUSION: Our research contributes to advancing the field of surgical skill assessment by leveraging depth cameras for more reliable and privacy evaluations. The findings suggest that depth cameras can be valuable in assessing surgical skills and provide a foundation for future research in this area.

Shortcut assessment: Can residents' operative performance be determined in the first five minutes of an operative task?

BACKGROUND: The aim was to validate the potential use of a single, early procedure, operative task as a predictive metric for overall performance. The authors hypothesized that a shortcut psychomotor assessment would be as informative as a total procedural psychomotor assessment when evaluating laparoscopic ventral hernia repair performance on a simulator. METHODS: Using electromagnetic sensors, hand motion data were collected from 38 surgery residents during a simulated laparoscopic ventral hernia repair procedure. Three time-based phases of the procedure were defined: Early Phase (start time through completion of first anchoring suture), Mid Phase (start time through completion of second anchoring suture), and Total Operative Time. Correlations were calculated comparing time and motion metrics for each phase with the final laparoscopic ventral hernia repair score. RESULTS: Analyses revealed that execution time and motion, for the first anchoring suture, predicted procedural outcomes. Greater execution times and path lengths correlated to lesser laparoscopic ventral hernia repair scores (r = -0.56, P = .0008 and r = -0.51, P = .0025, respectively). Greater bimanual dexterity measures correlated to Greater LVH repair scores (r = + 0.47, P = .0058). CONCLUSIONS: This study provides validity evidence for use of a single, early operative task as a shortcut assessment to predict resident performance during a simulated laparoscopic ventral hernia repair procedure. With the continued development and decreasing costs of motion technology, faculty should be well-versed in the use of motion metrics for performance measurements. The results strongly support the use of dexterity and economy of motion (path length + execution time) metrics as early predictors of operative performance.

Residents' surgical performance during the laboratory years: an analysis of rule-based errors.

BACKGROUND: Nearly one-third of surgical residents will enter into academic development during their surgical residency by dedicating time to a research fellowship for 1-3 y. Major interest lies in understanding how laboratory residents' surgical skills are affected by minimal clinical exposure during academic development. A widely held concern is that the time away from clinical exposure results in surgical skills decay. This study examines the impact of the academic development years on residents' operative performance. We hypothesize that the use of repeated, annual assessments may result in learning even without individual feedback on participants simulated performance. METHODS: Surgical performance data were collected from laboratory residents (postgraduate years 2-5) during the summers of 2014, 2015, and 2016. Residents had 15 min to complete a shortened, simulated laparoscopic ventral hernia repair procedure. Final hernia repair skins from all participants were scored using a previously validated checklist. An analysis of variance test compared the mean performance scores of repeat participants to those of first time participants. RESULTS: Twenty-seven (37% female) laboratory residents provided 2-year assessment data over the 3-year span of the study. Second time performance revealed improvement from a mean score of 14 (standard error = 1.0) in the first year to 17.2 (SD = 0.9) in the second year, (F[1, 52] = 5.6, P = 0.022). Detailed analysis demonstrated improvement in performance for 3 grading criteria that were considered to be rule-based errors. There was no improvement in operative strategy errors. CONCLUSIONS: Analysis of longitudinal performance of laboratory residents shows higher scores for repeat participants in the category of rule-based errors. These findings suggest that laboratory residents can learn from rule-based mistakes when provided with annual performance-based assessments. This benefit was not seen with operative strategy errors and has important implications for using assessments not only for performance analysis but also as a learning experience.

Research Residents' perceptions of skill decay: Effects of repeated skills assessments and scenario difficulty.

INTRODUCTION: Skills decay is a known risk for surgical residents who have dedicated research time. We hypothesize that simulation-based assessments will reveal significant differences in perceived skill decay when assessing a variety of clinical scenarios in a longitudinal fashion. METHODS: Residents (N = 46; Returning: n = 16, New: n = 30) completed four simulated procedures: urinary catheterization, central line, bowel anastomosis, and laparoscopic ventral hernia repair. Perception surveys were administered pre- and post-simulation. RESULTS: Perceptions of skill decay and task difficulty were similar for both groups across three procedures pre- and post-simulation. Due to a simulation modification, new residents were more confident in urinary catheterization than returning residents (F(1,4) = 11.44, p = 0.002). In addition, when assessing expectations for skill reduction, returning residents perceived greater skill reduction upon reassessment when compared to first time residents (t(35) = 2.37, p = 0.023). CONCLUSION: Research residents may benefit from longitudinal skills assessments and a wider variety of simulation scenarios during their research years. TABLE OF CONTENTS SUMMARY: As part of a longitudinal study, we assessed research residents' confidence, perceptions of task difficulty and surgical skill reduction. Residents completed surveys pre- and post-experience with four simulated procedures: urinary catheterization, subclavian central line insertion, bowel anastomosis, and laparoscopic ventral hernia repair. Returning residents perceived greater skill reduction upon reassessment when compared to residents participating for the first time. In addition, modification of the clinical scenarios affected perceptions of skills decay.

Do errors and critical events relate to hernia repair outcomes?

BACKGROUND: The study aimed to validate an error checklist for simulated laparoscopic ventral hernia (LVH) repair procedures. We hypothesize that residents' errors can be assessed with a structured checklist and the results will correlate significantly with procedural outcomes. METHODS: Senior residents' (N = 7) performance on a LVH simulator were video-recorded and analyzed using a human error checklist. Junior residents (N = 38) performed two steps of the same simulated LVH procedure. Performance was evaluated using the error checklist and repair quality scores. RESULTS: There were no significant differences between senior and junior residents' checklist errors (p > 0.1). Junior residents' errors correlated with hernia repair quality (p = 0.05). CONCLUSIONS: The newly developed assessment tool showed significant correlations between performance errors, critical events, and hernia repair quality. These results provide validity evidence for the use of errors in performance assessments. SUMMARY: This study validated an error checklist for simulated laparoscopic ventral hernia (LVH) repair procedures. The checklist was designed based on errors committed by chief surgery residents during LVH repairs. In a separate data collection, junior residents were evaluated using the checklist. Hernia repair quality was also assessed. Errors significantly correlated with hernia repair quality (p = 0.05).

Do resident's leadership skills relate to ratings of technical skill?

BACKGROUND: This study sought to compare general surgery research residents' survey information regarding self-efficacy ratings to their observed performance during a simulated small bowel repair. Their observed performance ratings were based on their leadership skills in directing their assistant. METHODS: Participants were given 15 min to perform a bowel repair using bovine intestines with standardized injuries. Operative assistants were assigned to help assist with the repair. Before the procedure, participants were asked to rate their expected skills decay, task difficulty, and confidence in addressing the small bowel injury. Interactions were coded to identify the number of instructions given by the participants to the assistant during the repair. Statistical analyses assessed the relationship between the number of directional instructions and participants' perceptions self-efficacy measures. Directional instructions were defined as any dialog by the participant who guided the assistant to perform an action. RESULTS: Thirty-six residents (58.3% female) participated in the study. Participants who rated lower levels of decay in their intraoperative decision-making and small bowel repair skills were noted to use their assistant more by giving more instructions. Similarly, a higher number of instructions correlated with lower perceived difficulty in selecting the correct suture, suture pattern, and completing the entire surgical task. CONCLUSIONS: General surgery research residents' intraoperative leadership skills showed significant correlations to their perceptions of skill decay and task difficulty during a bowel repair. Evaluating resident's directional instructions may provide an additional individualized intraoperative assessment metric. Further evaluation relating to operative performance outcomes is warranted.

Resident performance in complex simulated urinary catheter scenarios.

BACKGROUND: Urinary catheter insertion is a common procedure performed in hospitals. Improper catheterization can lead to unnecessary catheter-associated urinary tract infections and urethral trauma, increasing patient morbidity. To prevent such complications, guidelines were created on how to insert and troubleshoot urinary catheters. As nurses have an increasing responsibility for catheter placement, resident responsibility has shifted to more complex scenarios. This study examines the clinical decision-making skills of surgical residents during simulated urinary catheter scenarios. We hypothesize that during urinary catheterization, residents will make inconsistent decisions relating to catheter choices and clinical presentations. METHODS: Forty-five general surgery residents (postgraduate year 2-4) in Midwest training programs were presented with three of four urinary catheter scenarios of varying difficulty. Residents were allowed 15 min to complete the scenarios with five different urinary catheter choices. A chi-square test was performed to examine the relation between initial and subsequent catheter choices and to evaluate for consistency of decision-making for each scenario. RESULTS: Eighty-two percent of residents performed scenario A; 49% performed scenario B; 64% performed scenario C, and 82% performed scenario D. For initial attempt for scenario A-C, the 16 French Foley catheter was the most common choice (38%, 54%, 50%, P's < 0.001), whereas for scenario D, the 16 French Coude was the most common choice (37%, P < 0.01). Residents were most likely to be successful in achieving urine output in the initial catheterization attempt (P < 0.001). Chi-square analyses showed no relationship between residents' first and subsequent catheter choices for each scenario (P's > 0.05). CONCLUSIONS: Evaluation of clinical decision-making shows that initial catheter choice may have been deliberate based on patient background, as evidenced by the most popular choice in scenario D. Analyses of subsequent choices in each of the catheterization models reveal inconsistency. These findings suggest a possible lack of competence or training in clinical decision-making with regard to urinary catheter choices in residents.

Exploring hand coordination as a measure of surgical skill.

BACKGROUND: The study aim was to identify residents' coordination between dominant and nondominant hands while grasping for sutures in a laparoscopic ventral hernia repair procedure simulation. We hypothesize residents will rely on their dominant and nondominant hands unequally while grasping for suture. METHODS: Surgical residents had 15 min to complete the mesh securing and mesh tacking steps of a laparoscopic ventral hernia repair procedure. Procedure videos were coded for manual coordination events during the active suture grasping phase. Manual coordination events were defined as: active motion of dominant, nondominant, or both hands; and bimanual or unimanual manipulation of hands. A chi-square test was used to discriminate between coordination choices. RESULTS: Thirty-six residents (postgraduate year, 1-5) participated in the study. Residents changed manual coordination types during active suture grasping 500 times, ranging between 5 and 24 events (M = 13.9 events, standard deviation [SD] = 4.4). Bimanual coordination was used most (40%) and required the most time on average (M = 20.6 s, SD = 27.2), while unimanual nondominant coordination was used least (2.2%; M = 7.9 s, SD = 6.9). Residents relied on their dominant and nondominant hands unequally (P < 0.001). During 24% of events, residents depended on their nondominant hand (n = 120), which was predominantly used to operate the suture passer device. CONCLUSIONS: Residents appeared to actively coordinate both dominant and nondominant hands almost half of the time to complete suture grasping. Bimanual task durations took longer than other tasks on average suggesting these tasks were characteristically longer or switching hands required a greater degree of coordination. Future work is necessary to understand how task completion time and overall performance are affected by residents' hand utilization and switching between dominant and nondominant hands in surgical tasks.

Can a virtual reality assessment of fine motor skill predict successful central line insertion?

BACKGROUND: Due to the increased use of peripherally inserted central catheter lines, central lines are not performed as frequently. The aim of this study is to evaluate whether a virtual reality (VR)-based assessment of fine motor skills can be used as a valid and objective assessment of central line skills. METHODS: Surgical residents (N = 43) from 7 general surgery programs performed a subclavian central line in a simulated setting. Then, they participated in a force discrimination task in a VR environment. Hand movements from the subclavian central line simulation were tracked by electromagnetic sensors. Gross movements as monitored by the electromagnetic sensors were compared with the fine motor metrics calculated from the force discrimination tasks in the VR environment. RESULTS: Long periods of inactivity (idle time) during needle insertion and lack of smooth movements, as detected by the electromagnetic sensors, showed a significant correlation with poor force discrimination in the VR environment. Also, long periods of needle insertion time correlated to the poor performance in force discrimination in the VR environment. CONCLUSIONS: This study shows that force discrimination in a defined VR environment correlates to needle insertion time, idle time, and hand smoothness when performing subclavian central line placement. Fine motor force discrimination may serve as a valid and objective assessment of the skills required for successful needle insertion when placing central lines.

Exploring Senior Residents' Intraoperative Error Management Strategies: A Potential Measure of Performance Improvement.

OBJECTIVE: The study aim was to determine whether residents' error management strategies changed across 2 simulated laparoscopic ventral hernia (LVH) repair procedures after receiving feedback on their initial performance. We hypothesize that error detection and recovery strategies would improve during the second procedure without hands-on practice. DESIGN: Retrospective review of participant procedural performances of simulated laparoscopic ventral herniorrhaphy. A total of 3 investigators reviewed procedure videos to identify surgical errors. Errors were deconstructed. Error management events were noted, including error identification and recovery. SETTING: Residents performed the simulated LVH procedures during a course on advanced laparoscopy. Participants had 30 minutes to complete a LVH procedure. After verbal and simulator feedback, residents returned 24 hours later to perform a different, more difficult simulated LVH repair. PARTICIPANTS: Senior (N = 7; postgraduate year 4-5) residents in attendance at the course participated in this study. RESULTS: In the first LVH procedure, residents committed 121 errors (M = 17.14, standard deviation = 4.38). Although the number of errors increased to 146 (M = 20.86, standard deviation = 6.15) during the second procedure, residents progressed further in the second procedure. There was no significant difference in the number of errors committed for both procedures, but errors shifted to the late stage of the second procedure. Residents changed the error types that they attempted to recover (χ25=24.96, p<0.001). For the second procedure, recovery attempts increased for action and procedure errors, but decreased for strategy errors. Residents also recovered the most errors in the late stage of the second procedure (p < 0.001). CONCLUSION: Residents' error management strategies changed between procedures following verbal feedback on their initial performance and feedback from the simulator. Errors and recovery attempts shifted to later steps during the second procedure. This may reflect residents' error management success in the earlier stages, which allowed further progression in the second simulation. Incorporating error recognition and management opportunities into surgical training could help track residents' learning curve and provide detailed, structured feedback on technical and decision-making skills.