Opportunities and Applications of Educational Technologies in Surgical Education and Assessment.

BACKGROUND: Surgical education is challenged by continuously increasing clinical content, greater subspecialization, and public scrutiny of access to high quality surgical care. Since the last Blue Ribbon Committee on surgical education, novel technologies have been developed including artificial intelligence and telecommunication. OBJECTIVES AND METHODS: The goals of this Blue Ribbon Sub-Committee were to describe the latest technological advances and construct a framework for applying these technologies to improve the effectiveness and efficiency of surgical education and assessment. An additional goal was to identify implementation frameworks and strategies for centers with different resources and access. All sub-committee recommendations were included in a Delphi consensus process with the entire Blue Ribbon Committee (N=67). RESULTS: Our sub-committee found several new technologies and opportunities that are well poised to improve the effectiveness and efficiency of surgical education and assessment (see Tables 1-3). Our top recommendation was that a Multidisciplinary Surgical Educational Council be established to serve as an oversight body to develop consensus, facilitate implementation, and establish best practices for technology implementation and assessment. This recommendation achieved 93% consensus during the first round of the Delphi process. CONCLUSION: Advances in technology-based assessment, data analytics, and behavioral analysis now allow us to create personalized educational programs based on individual preferences and learning styles. If implemented properly, education technology has the promise of improving the quality and efficiency of surgical education and decreasing the demands on clinical faculty.

Structured feedback and operative video debriefing with critical view of safety annotation in training of laparoscopic cholecystectomy: a randomized controlled study.

BACKGROUND: The learning curve in minimally invasive surgery (MIS) is lengthened compared to open surgery. It has been reported that structured feedback and training in teams of two trainees improves MIS training and MIS performance. Annotation of surgical images and videos may prove beneficial for surgical training. This study investigated whether structured feedback and video debriefing, including annotation of critical view of safety (CVS), have beneficial learning effects in a predefined, multi-modal MIS training curriculum in teams of two trainees. METHODS: This randomized-controlled single-center study included medical students without MIS experience (n = 80). The participants first completed a standardized and structured multi-modal MIS training curriculum. They were then randomly divided into two groups (n = 40 each), and four laparoscopic cholecystectomies (LCs) were performed on ex-vivo porcine livers each. Students in the intervention group received structured feedback after each LC, consisting of LC performance evaluations through tutor-trainee joint video debriefing and CVS video annotation. Performance was evaluated using global and LC-specific Objective Structured Assessments of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS) scores. RESULTS: The participants in the intervention group had higher global and LC-specific OSATS as well as global and LC-specific GOALS scores than the participants in the control group (25.5 ± 7.3 vs. 23.4 ± 5.1, p = 0.003; 47.6 ± 12.9 vs. 36 ± 12.8, p < 0.001; 17.5 ± 4.4 vs. 16 ± 3.8, p < 0.001; 6.6 ± 2.3 vs. 5.9 ± 2.1, p = 0.005). The intervention group achieved CVS more often than the control group (1. LC: 20 vs. 10 participants, p = 0.037, 2. LC: 24 vs. 8, p = 0.001, 3. LC: 31 vs. 8, p < 0.001, 4. LC: 31 vs. 10, p < 0.001). CONCLUSIONS: Structured feedback and video debriefing with CVS annotation improves CVS achievement and ex-vivo porcine LC training performance based on OSATS and GOALS scores.

A foundation for evaluating the surgical artificial intelligence literature.

With increasing growth in applications of artificial intelligence (AI) in surgery, it has become essential for surgeons to gain a foundation of knowledge to critically appraise the scientific literature, commercial claims regarding products, and regulatory and legal frameworks that govern the development and use of AI. This guide offers surgeons a framework with which to evaluate manuscripts that incorporate the use of AI. It provides a glossary of common terms, an overview of prerequisite knowledge to maximize understanding of methodology, and recommendations on how to carefully consider each element of a manuscript to assess the quality of the data on which an algorithm was trained, the appropriateness of the methodological approach, the potential for reproducibility of the experiment, and the applicability to surgical practice, including considerations on generalizability and scalability.

Trends in bariatric surgery training: bariatric operations performed by Fellowship Council trainees from 2012 to 2019.

BACKGROUND: The American Society for Metabolic and Bariatric Surgery (ASMBS) Fellowship Certificate was created to ensure satisfactory training and requires a minimum number of anastomotic cases. With laparoscopic sleeve gastrectomy becoming the most common bariatric procedure in the United States, this may present a challenge for fellows to obtain adequate numbers for ASMBS certification. OBJECTIVES: To investigate bariatric fellowship trends from 2012 to 2019, the types, numbers, and approaches of surgical procedures performed by fellows were examined. SETTING: Academic training centers in the United States. METHODS: Data were obtained from Fellowship Council records of all cases performed by fellows in ASMBS-accredited bariatric surgery training programs between 2012 and 2019. A retrospective analysis using standard descriptive statistical methods was performed to investigate trends in total case volume and cases per fellow for common bariatric procedures. RESULTS: From 2012 to 2019, sleeve gastrectomy cases performed by all Fellowship Council fellows nearly doubled from 6,514 to 12,398, compared with a slight increase for gastric bypass, from 8,486 to 9,204. Looking specifically at bariatric fellowships, the mean number of gastric bypass cases per fellow dropped over time, from 91.1 cases (SD = 46.8) in 2012-2013 to 52.6 (SD = 62.1) in 2018-2019. Mean sleeve gastrectomy cases per fellow increased from 54.7 (SD = 31.5) in 2012-2013 to a peak of 98.6 (SD = 64.3) in 2015-2016. Robotic gastric bypasses also increased from 4% of all cases performed in 2012-2013 to 13.3% in 2018-2019. CONCLUSIONS: Bariatric fellowship training has seen a decrease in gastric bypasses, an increase in sleeve gastrectomies, and an increase in robotic surgery completed by each fellow from 2012 to 2019.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Practical Guide to Machine Learning and Artificial Intelligence in Surgical Education Research.

This Guide to Statistics and Methods gives an overview of artificial intelligence techniques and tools in surgical education research.

Demystifying AI: Current State and Future Role in Medical Education Assessment.

ABSTRACT: Medical education assessment faces multifaceted challenges, including data complexity, resource constraints, bias, feedback translation, and educational continuity. Traditional approaches often fail to adequately address these issues, creating stressful and inequitable learning environments. This article introduces the concept of precision education, a data-driven paradigm aimed at personalizing the educational experience for each learner. It explores how artificial intelligence (AI), including its subsets machine learning (ML) and deep learning (DL), can augment this model to tackle the inherent limitations of traditional assessment methods.AI can enable proactive data collection, offering consistent and objective assessments while reducing resource burdens. It has the potential to revolutionize not only competency assessment but also participatory interventions, such as personalized coaching and predictive analytics for at-risk trainees. The article also discusses key challenges and ethical considerations in integrating AI into medical education, such as algorithmic transparency, data privacy, and the potential for bias propagation.AI's capacity to process large datasets and identify patterns allows for a more nuanced, individualized approach to medical education. It offers promising avenues not only to improve the efficiency of educational assessments but also to make them more equitable. However, the ethical and technical challenges must be diligently addressed. The article concludes that embracing AI in medical education assessment is a strategic move toward creating a more personalized, effective, and fair educational landscape. This necessitates collaborative, multidisciplinary research and ethical vigilance to ensure that the technology serves educational goals while upholding social justice and ethical integrity.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Concept Graph Neural Networks for Surgical Video Understanding.

Analysis of relations between objects and comprehension of abstract concepts in the surgical video is important in AI-augmented surgery. However, building models that integrate our knowledge and understanding of surgery remains a challenging endeavor. In this paper, we propose a novel way to integrate conceptual knowledge into temporal analysis tasks using temporal concept graph networks. In the proposed networks, a knowledge graph is incorporated into the temporal video analysis of surgical notions, learning the meaning of concepts and relations as they apply to the data. We demonstrate results in surgical video data for tasks such as verification of the critical view of safety, estimation of the Parkland grading scale as well as recognizing instrument-action-tissue triplets. The results show that our method improves the recognition and detection of complex benchmarks as well as enables other analytic applications of interest.

Symptom duration is not associated with per oral endoscopic myotomy (POEM) failure.

INTRODUCTION: Per oral endoscopic myotomy (POEM) is a relatively novel technique to address achalasia; however, little is known about the efficacy of POEM for patients with long-standing achalasia. We hypothesize that patients with long-standing achalasia prior to intervention will be more recalcitrant to POEM than patients with symptoms for a short duration. METHODS: We performed a retrospective analysis of patients with achalasia who received a POEM at a single institution from 2012 to 2022. Patients were grouped into cohorts based on the time of symptom duration: < 1 year, 1-3 years, 4-10 years, > 10 years. POEM failure was defined as need for repeat intervention, symptom recurrence, and a high postoperative Eckart score. Demographic and clinical data were compared between cohorts. Measures of failure multivariable logistic regression analyzed the association between symptom duration and response to POEM. RESULTS: During the study period, 132 patients met inclusion criteria. Patient age at surgery, sex, BMI, Charleston-Deyo Comorbidity Index, and patients with diabetes with and without end organ complications, connective tissue diseases, and patients with ulcer diseases did not differ among cohorts. Patients who have had symptoms for greater than 10 years had significantly more endoscopic interventions prior to their POEM (30% vs, 60% p = 0.002). Patients in all cohorts experienced the same number of symptoms post-POEM. Manometric measurements did not vary across cohorts after POEM. Symptom recurrence, need for repeat endoscopic intervention, repeat surgical intervention, or repeat POEM also did not vary across cohorts. Having symptoms of achalasia > 10 years did not increase the odds POEM failure on multivariable logistical regression. CONCLUSIONS: These data suggest that longer symptom duration is not associated with increased rates of POEM failure. This is promising as clinicians should not exclude patients for POEM eligibility based on duration of symptoms alone.

Preoperative manometric data can predict symptom improvement after per oral endoscopic myotomy for treatment of achalasia.

BACKGROUND: While per oral endoscopic myotomy (POEM) has been shown to be efficacious in the treatment of achalasia, it can be difficult to predict who will have a robust and durable response. Historically, high lower esophageal sphincter pressures have been shown to predict a worse response to endoscopic therapies such as botox therapy. This study was designed to evaluate if modern preoperative manometric data could predict a response to therapy after POEM. METHODS: This was a retrospective study of 144 patients who underwent a POEM at a single institution by a single surgeon over an 8-year period (2014-2022) who had high-resolution manometry performed preoperatively and had an Eckardt symptom score performed both preoperatively and postoperatively. The achalasia type and integrated relaxation pressures (IRP) were then tested for potential correlation with need for any further achalasia interventions postoperatively as well as the degree of Eckardt score reduction using univariate analysis. RESULTS: The achalasia type on preoperatively manometry was not predictive of need for further interventions or degree of Eckardt score reduction (p = 0.74 and 0.44, respectively). A higher IRP was not predictive of need for further interventions however it was predictive of a greater reduction in postoperative Eckardt scores (p = 0.03) as shown by a nonzero regression slope. CONCLUSION: In this study, achalasia type was not a predictive factor in need for further interventions or degree of symptom relief. While IRP was not predictive of need for further interventions, a higher IRP did predict better symptomatic relief postoperatively. This result is opposite that of other endoscopic treatment modalities. Therefore, patients with higher IRP on high-resolution manometry would likely benefit from myotomy which provides significant symptomatic relief postoperatively.

Do preoperative endoscopic interventions predict symptom recurrence or reintervention after peroral endoscopic myotomy?

BACKGROUND: Studies assessing outcomes of patients undergoing peroral endoscopic myotomy (POEM) after botulinum injection or dilation have had various results with respect to failure, although this has not been differentiated between lack of clinical response and recurrence. We hypothesize that patients with previous endoscopic intervention(s) are more likely to recur than treatment-naïve patients. METHODS: This is a retrospective cohort study of patients that underwent POEM for achalasia at a single tertiary care center between 2011 and 2022. Patients were excluded if they had previous myotomy (POEM or Heller). The remaining patients were stratified into treatment-naïve patients (TN), those with previous botulinum injection (BTX), those with previous dilatation (BD), and those with both previous endoscopic interventions (BOTH). Primary outcome was recurrence indicated by clinical symptoms or need for repeat endoscopic intervention or surgery after originally having clinical resolution (Eckardt ≤ 3). Multivariate logistic regression using preoperative and intraoperative factors was completed to assess odds of recurrence. RESULTS: A total of 164 patients were included in the analysis, 90 TN, 34 BD, 28 BTX, and 12 BOTH. There were no other significant differences in demographics or in preoperative Eckardt score (p = 0.53). There was no difference in the proportion of patients that had postoperative manometry (p = 0.74), symptom recurrence (p = 0.59), surgical intervention (p = 0.16). BTX (14.3%) and BOTH (16.7%) patients had a higher rate of repeat endoscopic intervention than BD and TN patients (5.9% and 1.1%). In the logistic regression analysis, there was no association among the BTX, BD, or BOTH groups compared to the TN group. No odds ratios achieved statistical significance. CONCLUSIONS: There were no increased likelihood of recurrence with botulinum injection or dilatation prior to POEM, implying that they are similarly good candidates compared to treatment-naïve patients.

The Use of Artificial Intelligence Tools to Prepare Medical School Applications.

Advances in artificial intelligence (AI) have been changing the landscape in daily life and the practice of medicine. As these tools have evolved to become consumer-friendly, AI has become more accessible to many individuals, including applicants to medical school. With the rise of AI models capable of generating complex passages of text, questions have arisen regarding the appropriateness of using such tools to assist in the preparation of medical school applications. In this commentary, the authors offer a brief history of AI tools in medicine and describe large language models, a form of AI capable of generating natural language text passages. They question whether AI assistance should be considered inappropriate in preparing applications and compare it with the assistance some applicants receive from family, physician friends, or consultants. They call for clearer guidelines on what forms of assistance-human and technological-are permitted in the preparation of medical school applications. They recommend that medical schools steer away from blanket bans on AI tools in medical education and instead consider mechanisms for knowledge sharing about AI between students and faculty members, incorporation of AI tools into assignments, and the development of curricula to teach the use of AI tools as a competency.

Preoperative patient factors and anatomy do not predict who will develop reflux after per oral endoscopic myotomy.

BACKGROUND: Per oral endoscopic myotomy (POEM) has been shown to be an efficacious and safe therapy for the treatment of achalasia. Compared to laparoscopic Heller myotomy however, no antireflux procedure is routinely combined with POEM and therefore the development of symptomatic or silent reflux is of concern. This study was designed to determine if various patient factors and anatomy would predict the development of gastroesophageal reflux disease post-operatively. METHODS: This was a retrospective cohort study of all patients who underwent a POEM at a single institution by a single surgeon over an eight-year period (2014-2022). It has been our practice to obtain a postoperative ambulatory pH test on all patients 6 months after POEM off all acid reducing medications. Patients without a postoperative ambulatory esophageal pH monitoring test were excluded. Age, sex, obesity (BMI > 30), achalasia type, presence of a hiatal hernia, history of prior endoscopic achalasia treatments or myotomy were analyzed using univariate analysis as predictive factors for the development of postoperative GERD (DeMeester score > 14.7 on ambulatory pH monitoring). RESULTS: There were 179 total patients included in the study with 42 patients (23.5%) having undergone postoperative ambulatory pH testing. The majority of patients (137 or 76.5%) were lost to follow up and did not undergo ambulatory pH testing. Twenty-three out of those 42 patients (55%) had evidence of GERD on ambulatory pH testing. Multiple preoperative patient characteristics including demographics, manometric results, EGD findings, and history of prior achalasia interventions did not correlate with the development of post-operative GERD. CONCLUSIONS: Despite the high rate of reflux after POEM, there does not appear to be any reliable preoperative indicators of which patients have a higher risk of developing post-operative GERD after POEM.

An analysis of publications originating from abstracts presented at the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) Meeting.

BACKGROUND: Research presentation has benefits, including CV building, networking, and collaboration. A measurable standard for achievement is publication in a peer-reviewed journal. Expectations regarding the likelihood of publication are unknown for studies presented at a national surgical scientific meeting. This study aims to evaluate predictors of manuscript publication arising from abstracts presented at a national surgical scientific meeting. METHODS: Abstracts presented at the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) Meeting 2019 were reviewed. Identification of published manuscripts was completed using MedLine, Embase, and Google Scholar 28 months after the presentation to allow for time for publication. Factors evaluated for association with publication included author and abstract measures. Descriptive analyses and multivariable statistics were performed. RESULTS: 724 abstracts (160 podiums, 564 posters) were included. Of the podium presentations, 128 (80%) were published in a median of 4 months after the presentation. On univariable and multivariable analyses, there was no association between publication and abstract topic, gender, degree, number of publications, or H-indices of first and senior authors. 154 (27.3%) poster presentations were published with a median of 13 months. On univariable analysis, there was a statistically significant difference regarding the abstract topic (p = 0.015) and senior author degree (p = 0.01) between published and unpublished posters. Multivariable analysis demonstrated that colorectal surgery (OR 2.52; CI 1.02-6.23) and metabolic/obesity (OR 2.53; CI 1.09-5.84) are associated with an increased odd of publication. There was an inverse association with female senior authors (OR 0.53; CI 0.29-0.98), while additional degrees (e.g., doctorate and/or master's degree) of the senior authors were associated with an increased publication rate (OR 1.80; CI 1.00-3.22). CONCLUSION: 80% of podiums but only 27% of posters were ultimately published. While some predictors of poster publication were noted, it is unclear if these are why these projects fail to publish. Future research is warranted to determine if there are effective strategies to increase poster publication rates.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions.

OBJECTIVE: To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting. BACKGROUND: To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown. METHODS: Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. RESULTS: Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy. CONCLUSIONS: Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

SAGES video acquisition framework-analysis of available OR recording technologies by the SAGES AI task force.

BACKGROUND: Surgical video recording provides the opportunity to acquire intraoperative data that can subsequently be used for a variety of quality improvement, research, and educational applications. Various recording devices are available for standard operating room camera systems. Some allow for collateral data acquisition including activities of the OR staff, kinematic measurements (motion of surgical instruments), and recording of the endoscopic video streams. Additional analysis through computer vision (CV), which allows software to understand and perform predictive tasks on images, can allow for automatic phase segmentation, instrument tracking, and derivative performance-geared metrics. With this survey, we summarize available surgical video acquisition technologies and associated performance analysis platforms. METHODS: In an effort promoted by the SAGES Artificial Intelligence Task Force, we surveyed the available video recording technology companies. Of thirteen companies approached, nine were interviewed, each over an hour-long video conference. A standard set of 17 questions was administered. Questions spanned from data acquisition capacity, quality, and synchronization of video with other data, availability of analytic tools, privacy, and access. RESULTS: Most platforms (89%) store video in full-HD (1080p) resolution at a frame rate of 30 fps. Most (67%) of available platforms store data in a Cloud-based databank as opposed to institutional hard drives. CV powered analysis is featured in some platforms: phase segmentation in 44% platforms, out of body blurring or tool tracking in 33%, and suture time in 11%. Kinematic data are provided by 22% and perfusion imaging in one device. CONCLUSION: Video acquisition platforms on the market allow for in depth performance analysis through manual and automated review. Most of these devices will be integrated in upcoming robotic surgical platforms. Platform analytic supplementation, including CV, may allow for more refined performance analysis to surgeons and trainees. Most current AI features are related to phase segmentation, instrument tracking, and video blurring.