Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Opportunities and Applications of Educational Technologies in Surgical Education and Assessment.

BACKGROUND: Surgical education is challenged by continuously increasing clinical content, greater subspecialization, and public scrutiny of access to high quality surgical care. Since the last Blue Ribbon Committee on surgical education, novel technologies have been developed including artificial intelligence and telecommunication. OBJECTIVES AND METHODS: The goals of this Blue Ribbon Sub-Committee were to describe the latest technological advances and construct a framework for applying these technologies to improve the effectiveness and efficiency of surgical education and assessment. An additional goal was to identify implementation frameworks and strategies for centers with different resources and access. All sub-committee recommendations were included in a Delphi consensus process with the entire Blue Ribbon Committee (N=67). RESULTS: Our sub-committee found several new technologies and opportunities that are well poised to improve the effectiveness and efficiency of surgical education and assessment (see Tables 1-3). Our top recommendation was that a Multidisciplinary Surgical Educational Council be established to serve as an oversight body to develop consensus, facilitate implementation, and establish best practices for technology implementation and assessment. This recommendation achieved 93% consensus during the first round of the Delphi process. CONCLUSION: Advances in technology-based assessment, data analytics, and behavioral analysis now allow us to create personalized educational programs based on individual preferences and learning styles. If implemented properly, education technology has the promise of improving the quality and efficiency of surgical education and decreasing the demands on clinical faculty.

Key Issues in Surgical Residency Education: Recommendations of the Blue Ribbon II Committee Residency Education Subcommittee.

In September 2022, a summit was convened by the American Board of Surgery (ABS) to discuss competency-based reform in surgical education. A key output of that summit was the recommendation that the prior work of the Blue Ribbon I Committee convened 20 years earlier be revived. With leadership from the American College of Surgeons (ACS) and the American Surgical Association (ASA) , the Blue Ribbon Committee (BRC) II was subsequently convened. This paper describes the output of the Residency Education Subcommittee of the BRC II Committee. The Subcommittee organized its work around prioritized themes including curriculum, assessment, and transition to practice. Top recommendations, time-based action steps, potential barriers, and required resources were detailed and vetted through group discussion, broader Committee review and critique, and subsequent refinement. Primary concluding emphases included transitioning to a competency-based training model, facilitating dynamically capable curricular reform emphasizing the digital transformation of surgical care, using predictive analytic assessment strategies to optimize training effectiveness and efficiency, and creating mentorship strategies to govern the transition from training to independent practice in an outcomes-accountable fashion. It was recognized that coordinated efforts across existing organizational structures will be required, informed by dataset integration strategies that meaningfully measure educational and related patient outcomes.

Evaluation of postoperative esophagram following peroral endoscopic myotomy (POEM).

INTRODUCTION: The routine use of post-operative esophogram has come under evaluation for multiple upper GI surgeries such as with bariatric surgery and gastric resections. A major complication following Per Oral Endoscopic Myotomy (POEM) is a leak from the myotomy site. A post-operative contrast esophogram is often utilized to evaluate the presence of a leak, however it is not a standardized care practice for all patients. Presently it is selectively performed depending on physician assessment intra-operatively. This project will evaluate the necessity of post-operative contrast esophogram following POEM. MATERIALS AND METHODS: We retrospectively reviewed 277 patients diagnosed with achalasia who underwent POEM by two surgeons from 2011 to 2022. 173 patients met the inclusion criteria. A post-operative esophogram was used for the evaluation of a leak. Post-operative esophagram were selectively performed on day 1 following surgery using a water-soluble material. Data was evaluated using Stata. RESULTS: There were 3 detected leaks in the group that underwent esophagrams compared to the non-esophagram group in the early post-operative period. The overall complication rate was 5.5% in the non-esophagram versus 7.9% in the esophagram group. Length of stay was 1.48 days in the non-UGI vs 1.76 days in the esophagram group. Readmission rate was 10.9% in non-esophagram versus 8.7% in esophagram group. CONCLUSION: There was no statistically significant difference in outcomes in patients undergoing POEM who received post-operative esophagram verses patients who did not receive post-operative esophagram. The routine use of a contrast esophogram to detect a leak following POEM may not be justified. This study suggests that esophagrams should be performed depending on the clinical signs/symptoms post-operatively that would warrant imaging and intervention.

LapBot-Safe Chole: validation of an artificial intelligence-powered mobile game app to teach safe cholecystectomy.

BACKGROUND: Gaming can serve as an educational tool to allow trainees to practice surgical decision-making in a low-stakes environment. LapBot is a novel free interactive mobile game application that uses artificial intelligence (AI) to provide players with feedback on safe dissection during laparoscopic cholecystectomy (LC). This study aims to provide validity evidence for this mobile game. METHODS: Trainees and surgeons participated by downloading and playing LapBot on their smartphone. Players were presented with intraoperative LC scenes and required to locate their preferred location of dissection of the hepatocystic triangle. They received immediate accuracy scores and personalized feedback using an AI algorithm ("GoNoGoNet") that identifies safe/dangerous zones of dissection. Player scores were assessed globally and across training experience using non-parametric ANOVA. Three-month questionnaires were administered to assess the educational value of LapBot. RESULTS: A total of 903 participants from 64 countries played LapBot. As game difficulty increased, average scores (p < 0.0001) and confidence levels (p < 0.0001) decreased significantly. Scores were significantly positively correlated with players' case volume (p = 0.0002) and training level (p = 0.0003). Most agreed that LapBot should be incorporated as an adjunct into training programs (64.1%), as it improved their ability to reflect critically on feedback they receive during LC (47.5%) or while watching others perform LC (57.5%). CONCLUSIONS: Serious games, such as LapBot, can be effective educational tools for deliberate practice and surgical coaching by promoting learner engagement and experiential learning. Our study demonstrates that players' scores were correlated to their level of expertise, and that after playing the game, most players perceived a significant educational value.

Structured feedback and operative video debriefing with critical view of safety annotation in training of laparoscopic cholecystectomy: a randomized controlled study.

BACKGROUND: The learning curve in minimally invasive surgery (MIS) is lengthened compared to open surgery. It has been reported that structured feedback and training in teams of two trainees improves MIS training and MIS performance. Annotation of surgical images and videos may prove beneficial for surgical training. This study investigated whether structured feedback and video debriefing, including annotation of critical view of safety (CVS), have beneficial learning effects in a predefined, multi-modal MIS training curriculum in teams of two trainees. METHODS: This randomized-controlled single-center study included medical students without MIS experience (n = 80). The participants first completed a standardized and structured multi-modal MIS training curriculum. They were then randomly divided into two groups (n = 40 each), and four laparoscopic cholecystectomies (LCs) were performed on ex-vivo porcine livers each. Students in the intervention group received structured feedback after each LC, consisting of LC performance evaluations through tutor-trainee joint video debriefing and CVS video annotation. Performance was evaluated using global and LC-specific Objective Structured Assessments of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS) scores. RESULTS: The participants in the intervention group had higher global and LC-specific OSATS as well as global and LC-specific GOALS scores than the participants in the control group (25.5 ± 7.3 vs. 23.4 ± 5.1, p = 0.003; 47.6 ± 12.9 vs. 36 ± 12.8, p < 0.001; 17.5 ± 4.4 vs. 16 ± 3.8, p < 0.001; 6.6 ± 2.3 vs. 5.9 ± 2.1, p = 0.005). The intervention group achieved CVS more often than the control group (1. LC: 20 vs. 10 participants, p = 0.037, 2. LC: 24 vs. 8, p = 0.001, 3. LC: 31 vs. 8, p < 0.001, 4. LC: 31 vs. 10, p < 0.001). CONCLUSIONS: Structured feedback and video debriefing with CVS annotation improves CVS achievement and ex-vivo porcine LC training performance based on OSATS and GOALS scores.

Demystifying AI: Current State and Future Role in Medical Education Assessment.

ABSTRACT: Medical education assessment faces multifaceted challenges, including data complexity, resource constraints, bias, feedback translation, and educational continuity. Traditional approaches often fail to adequately address these issues, creating stressful and inequitable learning environments. This article introduces the concept of precision education, a data-driven paradigm aimed at personalizing the educational experience for each learner. It explores how artificial intelligence (AI), including its subsets machine learning (ML) and deep learning (DL), can augment this model to tackle the inherent limitations of traditional assessment methods.AI can enable proactive data collection, offering consistent and objective assessments while reducing resource burdens. It has the potential to revolutionize not only competency assessment but also participatory interventions, such as personalized coaching and predictive analytics for at-risk trainees. The article also discusses key challenges and ethical considerations in integrating AI into medical education, such as algorithmic transparency, data privacy, and the potential for bias propagation.AI's capacity to process large datasets and identify patterns allows for a more nuanced, individualized approach to medical education. It offers promising avenues not only to improve the efficiency of educational assessments but also to make them more equitable. However, the ethical and technical challenges must be diligently addressed. The article concludes that embracing AI in medical education assessment is a strategic move toward creating a more personalized, effective, and fair educational landscape. This necessitates collaborative, multidisciplinary research and ethical vigilance to ensure that the technology serves educational goals while upholding social justice and ethical integrity.

Trends in bariatric surgery training: bariatric operations performed by Fellowship Council trainees from 2012 to 2019.

BACKGROUND: The American Society for Metabolic and Bariatric Surgery (ASMBS) Fellowship Certificate was created to ensure satisfactory training and requires a minimum number of anastomotic cases. With laparoscopic sleeve gastrectomy becoming the most common bariatric procedure in the United States, this may present a challenge for fellows to obtain adequate numbers for ASMBS certification. OBJECTIVES: To investigate bariatric fellowship trends from 2012 to 2019, the types, numbers, and approaches of surgical procedures performed by fellows were examined. SETTING: Academic training centers in the United States. METHODS: Data were obtained from Fellowship Council records of all cases performed by fellows in ASMBS-accredited bariatric surgery training programs between 2012 and 2019. A retrospective analysis using standard descriptive statistical methods was performed to investigate trends in total case volume and cases per fellow for common bariatric procedures. RESULTS: From 2012 to 2019, sleeve gastrectomy cases performed by all Fellowship Council fellows nearly doubled from 6,514 to 12,398, compared with a slight increase for gastric bypass, from 8,486 to 9,204. Looking specifically at bariatric fellowships, the mean number of gastric bypass cases per fellow dropped over time, from 91.1 cases (SD = 46.8) in 2012-2013 to 52.6 (SD = 62.1) in 2018-2019. Mean sleeve gastrectomy cases per fellow increased from 54.7 (SD = 31.5) in 2012-2013 to a peak of 98.6 (SD = 64.3) in 2015-2016. Robotic gastric bypasses also increased from 4% of all cases performed in 2012-2013 to 13.3% in 2018-2019. CONCLUSIONS: Bariatric fellowship training has seen a decrease in gastric bypasses, an increase in sleeve gastrectomies, and an increase in robotic surgery completed by each fellow from 2012 to 2019.

A foundation for evaluating the surgical artificial intelligence literature.

With increasing growth in applications of artificial intelligence (AI) in surgery, it has become essential for surgeons to gain a foundation of knowledge to critically appraise the scientific literature, commercial claims regarding products, and regulatory and legal frameworks that govern the development and use of AI. This guide offers surgeons a framework with which to evaluate manuscripts that incorporate the use of AI. It provides a glossary of common terms, an overview of prerequisite knowledge to maximize understanding of methodology, and recommendations on how to carefully consider each element of a manuscript to assess the quality of the data on which an algorithm was trained, the appropriateness of the methodological approach, the potential for reproducibility of the experiment, and the applicability to surgical practice, including considerations on generalizability and scalability.

The impact of generative artificial intelligence on socioeconomic inequalities and policy making.

Generative artificial intelligence (AI) has the potential to both exacerbate and ameliorate existing socioeconomic inequalities. In this article, we provide a state-of-the-art interdisciplinary overview of the potential impacts of generative AI on (mis)information and three information-intensive domains: work, education, and healthcare. Our goal is to highlight how generative AI could worsen existing inequalities while illuminating how AI may help mitigate pervasive social problems. In the information domain, generative AI can democratize content creation and access but may dramatically expand the production and proliferation of misinformation. In the workplace, it can boost productivity and create new jobs, but the benefits will likely be distributed unevenly. In education, it offers personalized learning, but may widen the digital divide. In healthcare, it might improve diagnostics and accessibility, but could deepen pre-existing inequalities. In each section, we cover a specific topic, evaluate existing research, identify critical gaps, and recommend research directions, including explicit trade-offs that complicate the derivation of a priori hypotheses. We conclude with a section highlighting the role of policymaking to maximize generative AI's potential to reduce inequalities while mitigating its harmful effects. We discuss strengths and weaknesses of existing policy frameworks in the European Union, the United States, and the United Kingdom, observing that each fails to fully confront the socioeconomic challenges we have identified. We propose several concrete policies that could promote shared prosperity through the advancement of generative AI. This article emphasizes the need for interdisciplinary collaborations to understand and address the complex challenges of generative AI.

Practical Guide to Machine Learning and Artificial Intelligence in Surgical Education Research.

This Guide to Statistics and Methods gives an overview of artificial intelligence techniques and tools in surgical education research.