Opportunities and Applications of Educational Technologies in Surgical Education and Assessment.

BACKGROUND: Surgical education is challenged by continuously increasing clinical content, greater subspecialization, and public scrutiny of access to high quality surgical care. Since the last Blue Ribbon Committee on surgical education, novel technologies have been developed including artificial intelligence and telecommunication. OBJECTIVES AND METHODS: The goals of this Blue Ribbon Sub-Committee were to describe the latest technological advances and construct a framework for applying these technologies to improve the effectiveness and efficiency of surgical education and assessment. An additional goal was to identify implementation frameworks and strategies for centers with different resources and access. All sub-committee recommendations were included in a Delphi consensus process with the entire Blue Ribbon Committee (N=67). RESULTS: Our sub-committee found several new technologies and opportunities that are well poised to improve the effectiveness and efficiency of surgical education and assessment (see Tables 1-3). Our top recommendation was that a Multidisciplinary Surgical Educational Council be established to serve as an oversight body to develop consensus, facilitate implementation, and establish best practices for technology implementation and assessment. This recommendation achieved 93% consensus during the first round of the Delphi process. CONCLUSION: Advances in technology-based assessment, data analytics, and behavioral analysis now allow us to create personalized educational programs based on individual preferences and learning styles. If implemented properly, education technology has the promise of improving the quality and efficiency of surgical education and decreasing the demands on clinical faculty.

Precision Education: The Future of Lifelong Learning in Medicine.

ABSTRACT: The goal of medical education is to produce a physician workforce capable of delivering high-quality equitable care to diverse patient populations and communities. To achieve this aim amidst explosive growth in medical knowledge and increasingly complex medical care, a system of personalized and continuous learning, assessment, and feedback for trainees and practicing physicians is urgently needed. In this perspective, the authors build on prior work to advance a conceptual framework for such a system: precision education (PE).PE is a system that uses data and technology to transform lifelong learning by improving personalization, efficiency, and agency at the individual, program, and organization levels. PE "cycles" start with data inputs proactively gathered from new and existing sources, including assessments, educational activities, electronic medical records, patient care outcomes, and clinical practice patterns. Through technology-enabled analytics , insights are generated to drive precision interventions . At the individual level, such interventions include personalized just-in-time educational programming. Coaching is essential to provide feedback and increase learner participation and personalization. Outcomes are measured using assessment and evaluation of interventions at the individual, program, and organizational levels, with ongoing adjustment for repeated cycles of improvement. PE is rooted in patient, health system, and population data; promotes value-based care and health equity; and generates an adaptive learning culture.The authors suggest fundamental principles for PE, including promoting equity in structures and processes, learner agency, and integration with workflow (harmonization). Finally, the authors explore the immediate need to develop consensus-driven standards: rules of engagement between people, products, and entities that interact in these systems to ensure interoperability, data sharing, replicability, and scale of PE innovations.

Haptics: The Science of Touch As a Foundational Pathway to Precision Education and Assessment.

ABSTRACT: Clinical touch is the cornerstone of the doctor-patient relationship and can impact patient experience and outcomes. In the current era, driven by an ever-increasing infusion of point-of-care technologies, physical exam skills have become undervalued. Moreover, touch and hands-on skills have been difficult to teach due to inaccurate assessments and difficulty with learning transfer through observation. In this article, the authors argue that haptics, the science of touch, provides a unique opportunity to explore new pathways to facilitate touch training. Furthermore, haptics can dramatically increase the density of touch-based assessments without increasing human rater burden-essential for realizing precision assessment. The science of haptics is reviewed, including the benefits of using haptics-informed language for objective structured clinical examinations. The authors describe how haptic devices and haptic language have and can be used to facilitate learning, communication, documentation and a much-needed reinvigoration of physical examination, and touch excellence at the point of care. The synergy of haptic devices, artificial intelligence, and virtual reality environments are discussed. The authors conclude with challenges of scaling haptic technology in medical education, such as cost and translational needs, and opportunities to achieve wider adoption of this transformative approach to precision education.

Surgical Instant Replay-A National Video-Based Performance Assessment Toolbox.

This article discusses the widespread implementation of surgical video replay to improve technical and nontechnical performance of surgeons.

An American Board of Surgery Pilot of Video Assessment of Surgeon Technical Performance in Surgery.

OBJECTIVE: The American Board of Surgery (ABS) sought to investigate the suitability of video-based assessment (VBA) as an adjunct to certification for assessing technical skills. BACKGROUND: Board certification is based on the successful completion of a residency program coupled with knowledge and reasoning assessments. VBA is a new modality for evaluating operative skills that have been shown to correlate with patient outcomes after surgery. METHODS: Diplomates of the ABS were initially assessed for background knowledge and interest in VBA. Surgeons were then solicited to participate in the pilot. Three commercially available VBA platforms were identified and used for the pilot assessment. All participants served as reviewers and reviewees for videos. After the interaction, participants were surveyed regarding their experiences and recommendations to the ABS. RESULTS: To the initial survey, 4853/25,715 diplomates responded. The majority were neither familiar with VBA, nor the tools used for operative assessments. Two hundred seventy-four surgeons actively engaged in the subsequent pilot. One hundred sixty-nine surgeons completed the postpilot survey. Most participants found the process straightforward. Of the participants, 74% felt that the feedback would help their surgical practice. The majority (81%) remain interested in VBA for continuing medical education credits. Using VBA in continuous certification could improve surgeon skills felt by 70%. Two-thirds of participants felt VBA could help identify and remediate underperforming surgeons. Identified barriers to VBA included limitations for open surgery, privacy issues, and technical concerns. CONCLUSIONS: VBA is promising as an adjunct to the current board certification process and should be further considered by the ABS.

Diversity, equity, and inclusion in presidential leadership of academic medical and surgical societies.

BACKGROUND: Our aim was to identify gender and racial disparities in presidential leadership for national medical and surgical organizations. METHODS: We located publicly sourced information on national medical organizations. Years between or since the first diverse presidents were analyzed using descriptive statistics and Mann Whitney U tests. RESULTS: Sixty-seven national medical and surgical organizations were surveyed. 70.8% (n = 34) diversified via gender first (White-female), whereas 26.1% (n = 14) had racial diversity first. Organizations with gender diversity first followed with an African American male president sooner than organizations who first diversified by race (14.7 ± 11.8 v. 27.6 ± 11.3 years, p = 0.018). No significant difference was observed for the third tier of diversification. CONCLUSIONS: Significant gender and racial leadership disparities in national medical organizations are still present. It is notable that organizations with female leaders had a shorter timeline to racial diversity. These findings help to inform strategies to promote and increase diversity, equity, and inclusion in national leadership.

A disturbing trend: An analysis of the decline in surgical critical care fellowship training of Black and Hispanic surgeons.

BACKGROUND: Underrepresented minorities in medicine (URiMs) are disproportionally represented in surgery training programs. Rates of URiMs applying to and completing General Surgery residency remain low. We hypothesized that the patterns of URiMs disparities would persist into surgical critical care (SCC) fellowship applicants, matriculants, and graduates. METHODS: We performed a retrospective analysis of SCC applicants, matriculants, and graduates from 2005 to 2020 using the graduate medical education resident survey and analyzed applicant characteristics using the Surgical Critical Care and Acute Care Surgery Fellowship Application Service from 2018 to 2020. The data were stratified by race/ethnicity and sex. Indicator variables were created for Asian, Hispanic, White, and Black trainees. Yearly proportions for each race/ethnicity and sex categories completing or enrolling in a program were calculated and plotted over time with Loess smoothing lines and overlying 95% confidence bands. The yearly rate and statistical significance of change over time were tested with linear regression models with race/ethnicity and sex proportion as the dependent variables and year as the explanatory variable. RESULTS: From 2005 to 2020, there were a total of 2,481 graduates. Black men accounted for 4.7% of male graduates with a significant decline of 0.3% per year for the study period of those completing the fellowship (p = 0.02). Black women comprised 6.4% of female graduates and had a 0.6% decline each year (p < 0.01). A similar trend was seen with Hispanic men, who comprised 3.2% of male graduates and had a 0.3% annual decline (p = 0.02). White men had a significant increase in both matriculation to and graduation from SCC fellowships during the same interval. Similarly, Black and Hispanic applicants declined from 2019 to 2020, while the percentage of White applicants increased. CONCLUSION: Disparities in URiMs representation remain omnipresent in surgery and extend from residency training to SCC fellowship. Efforts to enhance the recruitment and retention of URiMs in SCC training are warranted. LEVEL OF EVIDENCE: Prognostic and Epidemiologic; level IV.

Video-based fully automatic assessment of open surgery suturing skills.

PURPOSE: The goal of this study was to develop a new reliable open surgery suturing simulation system for training medical students in situations where resources are limited or in the domestic setup. Namely, we developed an algorithm for tools and hands localization as well as identifying the interactions between them based on simple webcam video data, calculating motion metrics for assessment of surgical skill. METHODS: Twenty-five participants performed multiple suturing tasks using our simulator. The YOLO network was modified to a multi-task network for the purpose of tool localization and tool-hand interaction detection. This was accomplished by splitting the YOLO detection heads so that they supported both tasks with minimal addition to computer run-time. Furthermore, based on the outcome of the system, motion metrics were calculated. These metrics included traditional metrics such as time and path length as well as new metrics assessing the technique participants use for holding the tools. RESULTS: The dual-task network performance was similar to that of two networks, while computational load was only slightly bigger than one network. In addition, the motion metrics showed significant differences between experts and novices. CONCLUSION: While video capture is an essential part of minimal invasive surgery, it is not an integral component of open surgery. Thus, new algorithms, focusing on the unique challenges open surgery videos present, are required. In this study, a dual-task network was developed to solve both a localization task and a hand-tool interaction task. The dual network may be easily expanded to a multi-task network, which may be useful for images with multiple layers and for evaluating the interaction between these different layers.

Performance assessment using sensor technology.

Over the past 30 years, there have been numerous, noteworthy successes in the development, validation, and implementation of clinical skills assessments. Despite this progress, the medical profession has barely scratched the surface towards developing assessments that capture the true complexity of hands-on skills in procedural medicine. This paper highlights the development implementation and new discoveries in performance metrics when using sensor technology to assess cognitive and technical aspects of hands-on skills.

From Listening to Action: Academic Surgical Departmental Response to Social Injustice Through Curricular Development.

OBJECTIVE: The aim of this study was to describe the development and evaluation of a structured department wide cultural competency curriculum. SUMMARY BACKGROUND DATA: Despite numerous organizational policies and statements, social injustice and bias still exist. Our department committed to assist individuals of the entire department to develop foundational knowledge and skills to combat implicit bias and systemic racism through the creation of a cultural competency curriculum. The purpose of this manuscript is to detail our curriculum and the evaluation of its effectiveness. METHODS: Using a well-established curriculum development framework, a cultural competency curriculum was developed focusing on knowledge, skills and attitudes at the individual level, for all members of the department. The curriculum was implemented through 6-hour-long sessions over a 9-week period. Effectiveness was assessed through a post curriculum survey. RESULTS: Twenty percent of the respondents had experienced bias based on race, ethnicity, or sexual orientation in the past 12 months, whereas 30% had experienced bias based on sex. Seventy-one percent independently explored related topics. The curriculum was overall well received and generally achieved the goals and objectives. CONCLUSION: Using a standard curriculum development framework, an effective department-wide cultural competency curriculum can be developed and implemented.

Multimodal Cardiopulmonary Bypass Skills Assessment Within a High-Fidelity Simulation Environment.

BACKGROUND: A high-fidelity simulator that uses a perfused porcine heart, cannulae, and tubing has been demonstrated to be a useful training adjunct. We hypothesized that multimodal assessment of cardiopulmonary bypass (CPB) skills within this high-fidelity simulated environment could discern expert from trainee performance. METHODS: Three traditional fellows (postgraduate year 6-8) and 3 attending surgeons each performed 3 aortic cannulations. The third sequence included venous cannulation, commencement of CPB, and placement of a cardioplegia catheter and aortic cross-clamp. Performance across 20 cognitive and 21 technical domains was evaluated. Surgeon and assistant hand movements and economy of motion were assessed by electromagnetic motion sensors worn under sterile gloves. RESULTS: Analysis showed a significant difference in cognitive (6.7 ± 2.3 vs 4.6 ± 2.7, P = .03) but not technical (6.2 ± 2.5 vs 5.8 ± 2.2, P = .7) scores favoring the experts. In addition, experts showed higher efficiency by spending 64 ± 14 seconds to construct a nonpledgeted aortic purse-string suture and secure it with a Rummel, while trainees spent 82 ± 30 seconds to complete this task (P = .03). Motion analysis revealed similar path lengths between experts and trainees for cannulation and CPB but significantly shorter path lengths for experts in cross-clamp (47.5 ± 15.5 m vs 91.9 ± 20.3 m, P = .04). CONCLUSIONS: Multimodal assessment using cognitive, technical, and motion analysis of basic CPB tasks using a high-fidelity simulation environment is a valid system to measure performance and discriminate experts from trainees. This construct may allow for development of "competence thresholds" with important implications for training and certification in cardiothoracic surgery.

The what? How? And Who? Of video based assessment.

BACKGROUND: Currently, there is significant variability in the development, implementation and overarching goals of video review for assessment of surgical performance. METHODS: This paper evaluates the current methods in which video review is used for evaluation of surgical performance and identifies which processes are critical for successful, widespread implementation of video-based assessment. RESULTS: Despite the advances in video capture technology and growing interest in video-based assessment, there is a notable gap in the implementation and longitudinal use of formative and summative assessment using video. CONCLUSION: Validity, scalability and discoverability are current but removable barriers to video-based assessment.

A Call to Action: Black/African American Women Surgeon Scientists, Where are They?

OBJECTIVE: To determine the representation of Black/AA women surgeons in academic medicine among U.S. medical school faculty and to assess the number of NIH grants awarded to Black/AA women surgeon-scientists over the past 2 decades. SUMMARY OF BACKGROUND DATA: Despite increasing ethnic/racial and sex diversity in U.S. medical schools and residencies, Black/AA women have historically been underrepresented in academic surgery. METHODS: A retrospective review of the Association of American Medical Colleges 2017 Faculty Roster was performed and the number of grants awarded to surgeons from the NIH (1998-2017) was obtained. Data from the Association of American Medical Colleges included the total number of medical school surgery faculty, academic rank, tenure status, and department Chair roles. Descriptive statistics were performed. RESULTS: Of the 15,671 U.S. medical school surgical faculty, 123 (0.79%) were Black/AA women surgeons with only 11 (0.54%) being tenured faculty. When stratified by academic rank, 15 (12%) Black/AA women surgeons were instructors, 73 (59%) were assistant professors, 19 (15%) were associate professors, and 10 (8%) were full professors of surgery. Of the 372 U.S. department Chairs of surgery, none were Black/AA women. Of the 9139 NIH grants awarded to academic surgeons from 1998 and 2017, 31 (0.34%) grants were awarded to fewer than 12 Black/AA women surgeons. CONCLUSION: A significant disparity in the number of Black/AA women in academic surgery exists with few attaining promotion to the rank of professor with tenure and none ascending to the role of department Chair of surgery. Identifying and removing structural barriers to promotion, NIH grant funding, and academic advancement of Black/AA women as leaders and surgeon-scientists is needed.

The Society of Black Academic Surgeons CV benchmarking initiative: Early career trends of academic surgical leaders.

BACKGROUND: Surgeons from under-represented backgrounds are less likely to receive academic tenure and obtain leadership positions. Our objective was to query the curriculum vitaes (CVs) of SBAS leadership to develop a benchmarking tool to promote and guide careers in academic surgery. METHODS: CVs from academic leaders were reviewed for academic productivity at early career stages-the first 5-and 10-years. Variables queried: peer-reviewed publications, grant funding, surgical societal involvement, invited lectureships and visiting professorships. RESULTS: Of 20 CVs, 41 leadership positions including 13 SBAS Presidents were identified. At 5- and 10-years, respectively, the academic productivity increased: 20.6 and 52.3 publications; 4.7 and 9.7 grants; 18 and 42.6 lectures/professorships. CONCLUSION: The CV benchmarking tool may be a useful framework for aspiring academic surgeons to track their progress relative to successful SBAS members. Creative strategies like these, paired with faculty mentorship and sponsorship are necessary to improve the ethnic diversity in academic surgery.

Modeling Surgical Technical Skill Using Expert Assessment for Automated Computer Rating.

OBJECTIVE: Computer vision was used to predict expert performance ratings from surgeon hand motions for tying and suturing tasks. SUMMARY BACKGROUND DATA: Existing methods, including the objective structured assessment of technical skills (OSATS), have proven reliable, but do not readily discriminate at the task level. Computer vision may be used for evaluating distinct task performance throughout an operation. METHODS: Open surgeries was videoed and surgeon hands were tracked without using sensors or markers. An expert panel of 3 attending surgeons rated tying and suturing video clips on continuous scales from 0 to 10 along 3 task measures adapted from the broader OSATS: motion economy, fluidity of motion, and tissue handling. Empirical models were developed to predict the expert consensus ratings based on the hand kinematic data records. RESULTS: The predicted versus panel ratings for suturing had slopes from 0.73 to 1, and intercepts from 0.36 to 1.54 (Average R2 = 0.81). Predicted versus panel ratings for tying had slopes from 0.39 to 0.88, and intercepts from 0.79 to 4.36 (Average R2 = 0.57). The mean square error among predicted and expert ratings was consistently less than the mean squared difference among individual expert ratings and the eventual consensus ratings. CONCLUSIONS: The computer algorithm consistently predicted the panel ratings of individual tasks, and were more objective and reliable than individual assessment by surgical experts.

Measuring Competence in Surgical Training through Assessment of Surgical Entrustable Professional Activities.

OBJECTIVE: To assess competency in surgical residents with bundled assessments using a surgical entrustable professional activity (SEPA) framework. DESIGN: A pilot study was conducted using a combination of validated assessment tools (multiple choice exam (MCE) questions from the Surgical Council on Resident Education [SCORE], the Clinical Assessment and Management Examination - Outpatient (CAMEO) form, the Virtual Surgical Patient (VSP) website, and a procedure-specific Operative Performance Rating System [OPRS]) to determine competency in surgical residents in the treatment of breast cancer and gallbladder disease, respectively. SETTING: A large academic institution with a surgical training program in the Mid West of the United States. PARTICIPANTS: A total of 10 categorical surgical residents were invited to participate. Five completed the breast surgical EPA (SEPA) and 5 different residents completed the gallbladder SEPA. RESULTS: In terms of performance on the assessments, for the breast SEPA, scores did not appear to be related to PGY level, and residents' performance in general was the least strong on the MCE and the VSP case. The gallbladder SEPA showed a more expected pattern, distinguishing between junior and senior residents. As expected, all junior residents were required to remediate the OPRS assessment, while the senior residents passed. For the OPRS, senior level residents consistently were rated as "excellent" in terms of operative flow (5/5), while junior residents were all given a score of "good" (3/5). CONCLUSIONS: Assessing competence among surgical residents has been a discussion for several years. Varying methods of assessing competence have been proposed, but surgical competence is presently defined in a very general way through both the ACGME and American Board of Surgery (ABS). Using a SEPA format, as proposed, we could ensure specific understanding of each graduating resident's ability. These results show that the SEPA may be a valid tool for defining and capturing multiple areas of competence that are associated with different disease processes.

Shortcut assessment: Can residents' operative performance be determined in the first five minutes of an operative task?

BACKGROUND: The aim was to validate the potential use of a single, early procedure, operative task as a predictive metric for overall performance. The authors hypothesized that a shortcut psychomotor assessment would be as informative as a total procedural psychomotor assessment when evaluating laparoscopic ventral hernia repair performance on a simulator. METHODS: Using electromagnetic sensors, hand motion data were collected from 38 surgery residents during a simulated laparoscopic ventral hernia repair procedure. Three time-based phases of the procedure were defined: Early Phase (start time through completion of first anchoring suture), Mid Phase (start time through completion of second anchoring suture), and Total Operative Time. Correlations were calculated comparing time and motion metrics for each phase with the final laparoscopic ventral hernia repair score. RESULTS: Analyses revealed that execution time and motion, for the first anchoring suture, predicted procedural outcomes. Greater execution times and path lengths correlated to lesser laparoscopic ventral hernia repair scores (r = -0.56, P = .0008 and r = -0.51, P = .0025, respectively). Greater bimanual dexterity measures correlated to Greater LVH repair scores (r = + 0.47, P = .0058). CONCLUSIONS: This study provides validity evidence for use of a single, early operative task as a shortcut assessment to predict resident performance during a simulated laparoscopic ventral hernia repair procedure. With the continued development and decreasing costs of motion technology, faculty should be well-versed in the use of motion metrics for performance measurements. The results strongly support the use of dexterity and economy of motion (path length + execution time) metrics as early predictors of operative performance.

Faculty perceptions of resident skills decay during dedicated research fellowships.

BACKGROUND: Residents engaging in dedicated research experiences may return to clinical training with less surgical skill. The study aims were 1) to evaluate faculty perceptions of residents skills decay during dedicated research fellowships, and 2) to compare faculty and resident perceptions of residents skills decay. METHODS: Faculty and residents were surveyed on resident research practices and perceptions of resident skills decay. RESULTS: Faculty thought residents returning from research demonstrate less technical skill (Median = 4; 5-point Likert scale, 1 = Strongly disagree, 5 = Strongly agree), demonstrate less confidence (Median = 4), and require more instruction (Median = 4). Both faculty and residents perceived the largest skill reduction in complex procedures, technical surgical skills, and knowledge of procedure steps (p < 0.05). CONCLUSION: While dedicated research experiences provide valuable academic experience, there is a cost to clinical skills retention and confidence specifically in the areas of complex operative procedures and technical surgical skills.