Promoting breast health awareness: Can a sensor-enabled training system for patient education help?

INTRODUCTION: According to a 2009 study published in the Journal of Clinical Oncology, 79% of women (N = 222) diagnosed with breast cancer reported that they identified their cancers through breast self-exam (BSE). However, the U.S. Preventative Services Task Force does not require clinicians to teach women how to perform BSE. METHODS: To address this grave challenge, our team at the Technology Enabled Clinical Improvement (TECI) Center has developed a mobile, sensor-enabled haptic training system to teach women proper BSE technique. To validate the efficacy of the training system, our team deployed a data collection at the 2019 Breast Cancer and African Americans (BCAA) event where survey, sensor, and anecdotal data were collected from 61 participants. The custom-built breast model used in this study had a single, hard mass embedded in it. RESULTS: Participants at the BCAA event were able to successfully identify the mass 65% of the time and used an average force of 7.2 N. When looking at participants' confidence in their abilities to perform BSE, only 10% of respondents answered "very confident" pre-training whereas post-training, the reporting for "very confident" jumped to 66% (p < 0.01). CONCLUSION: By comparison, our previous work revealed that practitioners who use less than 10 N of force are 70% more likely to miss a lesion. The integration of sensors into the BSE haptic training system allowed for objective, evidence-based assessment of hands-on skill. In addition to teaching women proper BSE technique, this training empowered women to be informed advocates in their breast health journey. Future community-based training/feedback sessions will allow for continuous advancement of the training system.

Generating Rare Surgical Events Using CycleGAN: Addressing Lack of Data for Artificial Intelligence Event Recognition.

INTRODUCTION: Artificial Intelligence (AI) has shown promise in facilitating surgical video review through automatic recognition of surgical activities/events. There are few public video data sources that demonstrate critical yet rare events which are insufficient to train AI for reliable video event recognition. We suggest that a generative AI algorithm can create artificial massive bleeding images for minimally invasive lobectomy that can be used to augment the current lack of data in this field. MATERIALS AND METHODS: A generative adversarial network (GAN) algorithm was used (CycleGAN) to generate artificial massive bleeding event images. To train CycleGAN, six videos of minimally invasive lobectomies were utilized from which 1819 frames of nonbleeding instances and 3178 frames of massive bleeding instances were used. RESULTS: The performance of the CycleGAN algorithm was tested on a new video that was not used during the training process. The trained CycleGAN was able to alter the laparoscopic lobectomy images according to their corresponding massive bleeding images, where the contents of the original images were preserved (e.g., location of tools in the scene) and the style of each image is changed to massive bleeding (i.e., blood automatically added to appropriate locations on the images). CONCLUSIONS: The result could suggest a promising approach to supplement the lack of data for the rare massive bleeding event that can occur during minimally invasive lobectomy. Future work could be dedicated to developing AI algorithms to identify surgical strategies and actions that potentially lead to massive bleeding and warn surgeons prior to this event occurrence.

Do Individual Surgeon Preferences Affect Procedural Outcomes?

OBJECTIVES: Surgeon preferences such as instrument and suture selection and idiosyncratic approaches to individual procedure steps have been largely viewed as minor differences in the surgical workflow. We hypothesized that idiosyncratic approaches could be quantified and shown to have measurable effects on procedural outcomes. METHODS: At the American College of Surgeons (ACS) Clinical Congress, experienced surgeons volunteered to wear motion tracking sensors and be videotaped while evaluating a loop of porcine intestines to identify and repair 2 preconfigured, standardized enterotomies. Video annotation was used to identify individual surgeon preferences and motion data was used to quantify surgical actions. χ 2 analysis was used to determine whether surgical preferences were associated with procedure outcomes (bowel leak). RESULTS: Surgeons' (N=255) preferences were categorized into 4 technical decisions. Three out of the 4 technical decisions (repaired injuries together, double-layer closure, corner-stitches vs no corner-stitches) played a significant role in outcomes, P <0.05. Running versus interrupted did not affect outcomes. Motion analysis revealed significant differences in average operative times (leak: 6.67 min vs no leak: 8.88 min, P =0.0004) and work effort (leak-path length=36.86 cm vs no leak-path length=49.99 cm, P =0.001). Surgeons who took the riskiest path but did not leak had better bimanual dexterity (leak=0.21/1.0 vs no leak=0.33/1.0, P =0.047) and placed more sutures during the repair (leak=4.69 sutures vs no leak=6.09 sutures, P =0.03). CONCLUSIONS: Our results show that individual preferences affect technical decisions and play a significant role in procedural outcomes. Future analysis in more complex procedures may make major contributions to our understanding of contributors to procedure outcomes.

Developing a longitudinal database of surgical skills performance for practicing surgeons: A formal feasibility and acceptance inquiry.

BACKGROUND: We explored the feasibility and surgeons' perceptions of the utility of a longitudinal skills performance database. METHODS: A 10-station surgical skills assessment center was established at a national scientific meeting. Skills assessment volunteers (n = 189) completed a survey including opinions on practicing surgeons' skills evaluation, ethics, and interest in a longitudinal database. A subset (n = 23) participated in a survey-related interview. RESULTS: Nearly all participants reported interest in a longitudinal database and most believed there is an ethical obligation for such assessments to protect the public. Several interviewees specified a critical role for both formal and informal evaluation is to first create a safe and supportive environment. CONCLUSIONS: Participants support the construction of longitudinal skills databases that allow information sharing and establishment of professional standards. In a constructive environment, structured peer feedback was deemed acceptable to enhance and diversify surgeon skills. Large scale skills testing is feasible and scientific meetings may be the ideal location.

Can Deep Learning Algorithms Help Identify Surgical Workflow and Techniques?

BACKGROUND: Surgical videos are now being used for performance review and educational purposes; however, broad use is still limited due to time constraints. To make video review more efficient, we implemented Artificial Intelligence (AI) algorithms to detect surgical workflow and technical approaches. METHODS: Participants (N = 200) performed a simulated open bowel repair. The operation included two major phases: (1) Injury Identification and (2) Suture Repair. Accordingly, a phase detection algorithm (MobileNetV2+GRU) was implemented to automatically detect the two phases using video data. In addition, participants were noted to use three different technical approaches when running the bowel: (1) use of both hands, (2) use of one hand and one tool, or (3) use of two tools. To discern the three technical approaches, an object detection (YOLOv3) algorithm was implemented to recognize objects that were commonly used during the Injury Identification phase (hands versus tools). RESULTS: The phase detection algorithm achieved high precision (recall) when segmenting the two phases: Injury Identification (86 ± 9% [81 ± 12%]) and Suture Repair (81 ± 6% [81 ± 16%]). When evaluating three technical approaches in running the bowel, the object detection algorithm achieved high average precisions (Hands [99.32%] and Tools [94.47%]). The three technical approaches showed no difference in execution time (Kruskal-Wallis Test: P= 0.062) or injury identification (not missing an injury) (Chi-squared: P= 0.998). CONCLUSIONS: The AI algorithms showed high precision when segmenting surgical workflow and identifying technical approaches. Automation of these techniques for surgical video databases has great potential to facilitate efficient performance review.

Performance assessment using sensor technology.

Over the past 30 years, there have been numerous, noteworthy successes in the development, validation, and implementation of clinical skills assessments. Despite this progress, the medical profession has barely scratched the surface towards developing assessments that capture the true complexity of hands-on skills in procedural medicine. This paper highlights the development implementation and new discoveries in performance metrics when using sensor technology to assess cognitive and technical aspects of hands-on skills.

Multimodal Cardiopulmonary Bypass Skills Assessment Within a High-Fidelity Simulation Environment.

BACKGROUND: A high-fidelity simulator that uses a perfused porcine heart, cannulae, and tubing has been demonstrated to be a useful training adjunct. We hypothesized that multimodal assessment of cardiopulmonary bypass (CPB) skills within this high-fidelity simulated environment could discern expert from trainee performance. METHODS: Three traditional fellows (postgraduate year 6-8) and 3 attending surgeons each performed 3 aortic cannulations. The third sequence included venous cannulation, commencement of CPB, and placement of a cardioplegia catheter and aortic cross-clamp. Performance across 20 cognitive and 21 technical domains was evaluated. Surgeon and assistant hand movements and economy of motion were assessed by electromagnetic motion sensors worn under sterile gloves. RESULTS: Analysis showed a significant difference in cognitive (6.7 ± 2.3 vs 4.6 ± 2.7, P = .03) but not technical (6.2 ± 2.5 vs 5.8 ± 2.2, P = .7) scores favoring the experts. In addition, experts showed higher efficiency by spending 64 ± 14 seconds to construct a nonpledgeted aortic purse-string suture and secure it with a Rummel, while trainees spent 82 ± 30 seconds to complete this task (P = .03). Motion analysis revealed similar path lengths between experts and trainees for cannulation and CPB but significantly shorter path lengths for experts in cross-clamp (47.5 ± 15.5 m vs 91.9 ± 20.3 m, P = .04). CONCLUSIONS: Multimodal assessment using cognitive, technical, and motion analysis of basic CPB tasks using a high-fidelity simulation environment is a valid system to measure performance and discriminate experts from trainees. This construct may allow for development of "competence thresholds" with important implications for training and certification in cardiothoracic surgery.

Translating motion tracking data into resident feedback: An opportunity for streamlined video coaching.

BACKGROUND: We hypothesized that differences in motion data during a simulated laparoscopic ventral hernia repair (LVH) can be used to stratify top and lower tier performers and streamline video review. MATERIALS AND METHODS: Surgical residents (N = 94) performed a simulated partial LVH repair while wearing motion tracking sensors. We identified the top ten and lower ten performers based on a final product quality score (FPQS) of the repair. Two blinded raters independently reviewed motion plots to identify patterns and stratify top and lower tier performers. RESULTS: Top performers had significantly higher FPQS (23.3 ± 1.2 vs 5.7 ± 1.6 p < 0.01). Raters identified patterns and stratified top performers from lower tier performers (Rater 1 χ2 = 3.2 p = 0.07 and Rater 2 χ2 = 2.0 p = 0.16). During video review, we correlated motion plots with the relevant portion of the procedure. CONCLUSION: Differences in motion data can identify learning needs and enable rapid review of surgical videos for coaching.

Sensors and Psychomotor Metrics: A Unique Opportunity to Close the Gap on Surgical Processes and Outcomes.

The surgical process remains elusive to many. This paper presents two independent empirical investigations where psychomotor skill metrics were used to quantify elements of the surgical process in a procedural context during surgical tasks in a simulated environment. The overarching goal of both investigations was to address the following hypothesis: Basic motion metrics can be used to quantify specific aspects of the surgical process including instrument autonomy, psychomotor efficiency, procedural readiness, and clinical errors. Electromagnetic motion tracking sensors were secured to surgical trainees' (N = 64) hands for both studies, and several motion metrics were investigated as a measure of surgical skill. The first study assessed performance during a bowel repair and laparoscopic ventral hernia (LVH) repair in comparison to a suturing board task. The second study assessed performance in a VR task in comparison to placement of a subclavian central line. The findings of the first study support our subhypothesis that motion metrics have a generalizable application to surgical skill by showing significant correlations in instrument autonomy and psychomotor efficiency during the suturing task and bowel repair (idle time: r = 0.46, p < 0.05; average velocity: r = 0.57, p < 0.05) and the suturing task and LVH repair (jerk magnitude: r = 0.36, p < 0.05; bimanual dexterity: r = 0.35, p < 0.05). In the second study, performance in VR (steering and jerkiness) correlated to clinical errors (r = 0.58, p < 0.05) and insertion time (r = 0.55, p < 0.05) in placement of a subclavian central line. Both gross (dexterity) and fine motor skills (steering) were found to be important as well as efficiency (i.e., idle time, duration, velocity) when seeking to understand the quality of surgical performance. Both studies support our hypotheses that basic motion metrics can be used to quantify specific aspects of the surgical process and that the use of different technologies and metrics are important for comprehensive investigations of surgical skill.

Use of sensors to quantify procedural idle time: Validity evidence for a new mastery metric.

BACKGROUND: Quantification of mastery is the first step in using objective metrics for teaching. We hypothesized that during orotracheal intubation, top tier performers have less idle time compared to lower tier performers. METHODS: At the Anesthesiology 2018 Annual Meeting, 82 participants intubated a normal airway simulator and a burnt airway simulator. The movements of the participant's laryngoscope were quantified using electromagnetic motion sensors. Top tier performers were defined as participants who intubated both simulators successfully in less than the median time for each simulator. Idle time was defined as the duration of time when the laryngoscope was not moving. RESULTS: Top performers showed less Idle Time when intubating the normal airway compared to lower tier performers (14.5 ± 9.8 seconds vs 34.0 ± 52.0 seconds, respectively P < .01). Likewise, top performers showed less Idle Time when intubating the burnt airway compared to lower tier performers (18.6 ± 15.2 seconds vs 63.4 ± 59.11 seconds; P < .01). Comparing performance on the burnt airway to the normal airway, there was a difference for lower tier performers (63.4 ± 59.1 seconds vs 34.0 ± 52.0 seconds; P < .01) but not for top tier performers (18.6 ± 15.2 seconds vs 14.5 ± 9.8 seconds; P = .07). CONCLUSION: Similar to our previous findings with other procedures, Idle Time was shown to have known group validity evidence when comparing top performers with lower tier performers. Further, Idle Time was correlated with procedure difficulty in our prior work. We observed statistically significant differences in Idle Times for lower tier performers when comparing the normal airway to the burnt airway but not for top tier performers. Our findings support the continued exploration of Idle Time for development of objective assessment and curricula.

Can Virtual Reality Be Used to Track Skills Decay During the Research Years?

BACKGROUND: Time away from surgical practice can lead to skills decay. Research residents are thought to be prone to skills decay, given their limited experience and reduced exposure to clinical activities during their research training years. This study takes a cross-sectional approach to assess differences in residents' skills at the beginning and end of their research years using virtual reality. We hypothesized that research residents will have measurable decay in psychomotor skills when evaluated using virtual reality. METHODS: Surgical residents (n = 28) were divided into two groups; the first group was just beginning their research time (clinical residents: n = 19) and the second group (research residents: n = 9) had just finished at least 2 y of research. All participants were asked to perform a target-tracking task using a haptic device, and their performance was compared using Welch's t-test. RESULTS: Research residents showed a higher level of "tracking error" (1.69 ± 0.44 cm versus 1.40 ± 0.19 cm; P = 0.04) and a similar level of "path length" (62.5 ± 10.5 cm versus 62.1 ± 5.2 cm; P = 0.92) when compared with clinical residents. CONCLUSIONS: The increased "tracking error" among residents at the end of their research time suggests fine psychomotor skills decay in residents who spend time away from clinical duties during laboratory time. This decay demonstrates the need for research residents to regularly participate in clinical activities, simulation, or assessments to minimize and monitor skills decay while away from clinical practice. Additional longitudinal studies may help better map learning and decay curves for residents who spend time away from clinical practice.

Screening surgical residents' laparoscopic skills using virtual reality tasks: Who needs more time in the sim lab?

BACKGROUND: This study investigated the possibility of using virtual reality perceptual-motor tasks as a screening tool for laparoscopic ability. We hypothesized that perceptual-motor skills assessed using virtual reality will correlate with the quality of simulated laparoscopic ventral hernia repair. MATERIALS AND METHODS: Surgical residents (N = 37), performed 2 virtual reality perceptual-motor tasks: (1) force matching and (2) target tracking. Participants also performed a laparoscopic ventral hernia repair on a simulator and final product quality score, and endoscopic visualization errors were calculated. Correlational analysis was performed to assess the relationship between performance on virtual reality tasks and laparoscopic ventral hernia repair. RESULTS: Residents with poor performance on force matching in virtual reality-"peak deflection" (r = -0.34, P < .05) and "summation distance" (r = -0.36, P < .05)-had lower final product quality scores. Likewise, poor performance in virtual reality-based target tracking-"path length" (r = -0.49, P < .05) and "maximum distance" (r = -0.37, P < .05)-correlated with a lower final product quality score. CONCLUSION: Our findings support the notion that virtual reality could be used as a screening tool for perceptual-motor skill. Trainees identified as having poor perceptual-motor skill can benefit from focused curricula, allowing them to hone personal areas of weakness and maximize technical skill.

Combining Metrics From Clinical Simulators and Sensorimotor Tasks Can Reveal the Training Background of Surgeons.

BACKGROUND: Skill assessment in surgery traditionally has relied on the expert observation and qualitative scoring. Our novel study design demonstrates how analysis of performance in sensorimotor tasks and bench-top surgical simulators can provide inferences about the technical proficiency as well as the training history of surgeons. METHODS: We examined metrics for basic sensorimotor tasks in a virtual reality interface as well as motion metrics in clinical scenario simulations. As indicators of the training level, we considered survey responses from surgery residents, including the number of postgraduation years (PGY, four levels), research years (RY, three levels), and clinical years (CY, three levels). Next, we performed a linear discriminant analysis with cross-validation (90% training, 10% testing) to relate the training levels to the selected metrics. RESULTS: Using combined metrics from all stations, we found greater than chance predictions for each survey category, with an overall accuracy of 43.4 ± 2.9% for identifying the level for post-graduate years, 79.1 ± 1.0% accuracy for research training years, and 64.2 ± 1.0% for clinical training years. Our main finding was that combining metrics from all stations resulted in more accurate predictions than using only sensorimotor or clinical scenario tasks. In addition, we found that metrics related to the ability to cope with changes in the task environment were the most important predictors of training level. CONCLUSIONS: These results suggest that each simulator-type provided crucial information for evaluating surgical proficiency. The methods developed in this paper could improve evaluations of a surgeon's clinical proficiency as well as training potential in terms of basic sensorimotor ability.

Dynamic Visual Feedback During Junctional Tourniquet Training.

BACKGROUND: This project involved the development and evaluation of a new visual bleeding feedback (VBF) system for tourniquet training. We hypothesized that dynamic VBF during junctional tourniquet training would be helpful and well received by trainees. MATERIALS AND METHODS: We designed the VBF to simulate femoral bleeding. Medical students (n = 15) and emergency medical service (EMS) members (n = 4) were randomized in a single-blind, crossover study to the VBF or without feedback groups. Poststudy surveys assessing VBF usefulness and recommendations were conducted along with participants' reported confidence using a 7-point Likert scale. Data from the different groups were compared using Wilcoxon signed-rank and rank-sum tests. RESULTS: Participants rated the helpfulness of the VBF highly (6.53/7.00) and indicated they were very likely to recommend the VBF simulator to others (6.80/7.00). Pre- and post-VBF confidence were not statistically different (P = 0.59). Likewise, tourniquet application times for VBF and without feedback before crossover were not statistically different (P = 0.63). Although participant confidence did not change significantly from beginning to end of the study (P = 0.46), application time was significantly reduced (P = 0.001). CONCLUSIONS: New tourniquet learners liked our VBF prototype and found it useful. Although confidence did not change over the course of the study for any group, application times improved. Future studies using outcomes of this study will allow us to continue VBF development as well as incorporate other quantitative measures of task performance to elucidate VBF's true benefit and help trainees achieve mastery in junctional tourniquet skills.

Shortcut assessment: Can residents' operative performance be determined in the first five minutes of an operative task?

BACKGROUND: The aim was to validate the potential use of a single, early procedure, operative task as a predictive metric for overall performance. The authors hypothesized that a shortcut psychomotor assessment would be as informative as a total procedural psychomotor assessment when evaluating laparoscopic ventral hernia repair performance on a simulator. METHODS: Using electromagnetic sensors, hand motion data were collected from 38 surgery residents during a simulated laparoscopic ventral hernia repair procedure. Three time-based phases of the procedure were defined: Early Phase (start time through completion of first anchoring suture), Mid Phase (start time through completion of second anchoring suture), and Total Operative Time. Correlations were calculated comparing time and motion metrics for each phase with the final laparoscopic ventral hernia repair score. RESULTS: Analyses revealed that execution time and motion, for the first anchoring suture, predicted procedural outcomes. Greater execution times and path lengths correlated to lesser laparoscopic ventral hernia repair scores (r = -0.56, P = .0008 and r = -0.51, P = .0025, respectively). Greater bimanual dexterity measures correlated to Greater LVH repair scores (r = + 0.47, P = .0058). CONCLUSIONS: This study provides validity evidence for use of a single, early operative task as a shortcut assessment to predict resident performance during a simulated laparoscopic ventral hernia repair procedure. With the continued development and decreasing costs of motion technology, faculty should be well-versed in the use of motion metrics for performance measurements. The results strongly support the use of dexterity and economy of motion (path length + execution time) metrics as early predictors of operative performance.

Residents' response to bleeding during a simulated robotic surgery task.

BACKGROUND: The aim of this study was to assess performance measurement validity of our newly developed robotic surgery task trainer. We hypothesized that residents would exhibit wide variations in their intercohort performance as well as a measurable difference compared to surgeons in fellowship training. MATERIALS AND METHODS: Our laboratory synthesized a model of a pelvic tumor that simulates unexpected bleeding. Surgical residents and fellows of varying specialties completed a demographic survey and were allowed 20 minutes to resect the tumor using the da Vinci robot and achieve hemostasis. At a standardized event in the simulation, venous bleeding began, and participants attempted hemostasis using suture ligation. A motion tracking system, using electromagnetic sensors, recorded participants' hand movements. A postparticipation Likert scale survey evaluated participants' assessment of the model's realism and usefulness. RESULTS: Three of the seven residents (postgraduate year 2-5), and the fellow successfully resected the tumor in the allotted time. Residents showed high variability in performance and blood loss (125-700 mL) both within their cohort and compared to the fellow (150 mL blood). All participants rated the model as having high realism and utility for trainees. CONCLUSIONS: The results support that our bleeding pelvic tumor simulator has the ability to discriminate resident performance in robotic surgery. The combination of motion, decision-making, and blood loss metrics offers a multilevel performance assessment, analyzing both technical and decision-making abilities.

Residents' surgical performance during the laboratory years: an analysis of rule-based errors.

BACKGROUND: Nearly one-third of surgical residents will enter into academic development during their surgical residency by dedicating time to a research fellowship for 1-3 y. Major interest lies in understanding how laboratory residents' surgical skills are affected by minimal clinical exposure during academic development. A widely held concern is that the time away from clinical exposure results in surgical skills decay. This study examines the impact of the academic development years on residents' operative performance. We hypothesize that the use of repeated, annual assessments may result in learning even without individual feedback on participants simulated performance. METHODS: Surgical performance data were collected from laboratory residents (postgraduate years 2-5) during the summers of 2014, 2015, and 2016. Residents had 15 min to complete a shortened, simulated laparoscopic ventral hernia repair procedure. Final hernia repair skins from all participants were scored using a previously validated checklist. An analysis of variance test compared the mean performance scores of repeat participants to those of first time participants. RESULTS: Twenty-seven (37% female) laboratory residents provided 2-year assessment data over the 3-year span of the study. Second time performance revealed improvement from a mean score of 14 (standard error = 1.0) in the first year to 17.2 (SD = 0.9) in the second year, (F[1, 52] = 5.6, P = 0.022). Detailed analysis demonstrated improvement in performance for 3 grading criteria that were considered to be rule-based errors. There was no improvement in operative strategy errors. CONCLUSIONS: Analysis of longitudinal performance of laboratory residents shows higher scores for repeat participants in the category of rule-based errors. These findings suggest that laboratory residents can learn from rule-based mistakes when provided with annual performance-based assessments. This benefit was not seen with operative strategy errors and has important implications for using assessments not only for performance analysis but also as a learning experience.

Can a virtual reality assessment of fine motor skill predict successful central line insertion?

BACKGROUND: Due to the increased use of peripherally inserted central catheter lines, central lines are not performed as frequently. The aim of this study is to evaluate whether a virtual reality (VR)-based assessment of fine motor skills can be used as a valid and objective assessment of central line skills. METHODS: Surgical residents (N = 43) from 7 general surgery programs performed a subclavian central line in a simulated setting. Then, they participated in a force discrimination task in a VR environment. Hand movements from the subclavian central line simulation were tracked by electromagnetic sensors. Gross movements as monitored by the electromagnetic sensors were compared with the fine motor metrics calculated from the force discrimination tasks in the VR environment. RESULTS: Long periods of inactivity (idle time) during needle insertion and lack of smooth movements, as detected by the electromagnetic sensors, showed a significant correlation with poor force discrimination in the VR environment. Also, long periods of needle insertion time correlated to the poor performance in force discrimination in the VR environment. CONCLUSIONS: This study shows that force discrimination in a defined VR environment correlates to needle insertion time, idle time, and hand smoothness when performing subclavian central line placement. Fine motor force discrimination may serve as a valid and objective assessment of the skills required for successful needle insertion when placing central lines.