Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Opportunities and Applications of Educational Technologies in Surgical Education and Assessment.

BACKGROUND: Surgical education is challenged by continuously increasing clinical content, greater subspecialization, and public scrutiny of access to high quality surgical care. Since the last Blue Ribbon Committee on surgical education, novel technologies have been developed including artificial intelligence and telecommunication. OBJECTIVES AND METHODS: The goals of this Blue Ribbon Sub-Committee were to describe the latest technological advances and construct a framework for applying these technologies to improve the effectiveness and efficiency of surgical education and assessment. An additional goal was to identify implementation frameworks and strategies for centers with different resources and access. All sub-committee recommendations were included in a Delphi consensus process with the entire Blue Ribbon Committee (N=67). RESULTS: Our sub-committee found several new technologies and opportunities that are well poised to improve the effectiveness and efficiency of surgical education and assessment (see Tables 1-3). Our top recommendation was that a Multidisciplinary Surgical Educational Council be established to serve as an oversight body to develop consensus, facilitate implementation, and establish best practices for technology implementation and assessment. This recommendation achieved 93% consensus during the first round of the Delphi process. CONCLUSION: Advances in technology-based assessment, data analytics, and behavioral analysis now allow us to create personalized educational programs based on individual preferences and learning styles. If implemented properly, education technology has the promise of improving the quality and efficiency of surgical education and decreasing the demands on clinical faculty.

Key Issues in Surgical Residency Education: Recommendations of the Blue Ribbon II Committee Residency Education Subcommittee.

In September 2022, a summit was convened by the American Board of Surgery (ABS) to discuss competency-based reform in surgical education. A key output of that summit was the recommendation that the prior work of the Blue Ribbon I Committee convened 20 years earlier be revived. With leadership from the American College of Surgeons (ACS) and the American Surgical Association (ASA) , the Blue Ribbon Committee (BRC) II was subsequently convened. This paper describes the output of the Residency Education Subcommittee of the BRC II Committee. The Subcommittee organized its work around prioritized themes including curriculum, assessment, and transition to practice. Top recommendations, time-based action steps, potential barriers, and required resources were detailed and vetted through group discussion, broader Committee review and critique, and subsequent refinement. Primary concluding emphases included transitioning to a competency-based training model, facilitating dynamically capable curricular reform emphasizing the digital transformation of surgical care, using predictive analytic assessment strategies to optimize training effectiveness and efficiency, and creating mentorship strategies to govern the transition from training to independent practice in an outcomes-accountable fashion. It was recognized that coordinated efforts across existing organizational structures will be required, informed by dataset integration strategies that meaningfully measure educational and related patient outcomes.

LapBot-Safe Chole: validation of an artificial intelligence-powered mobile game app to teach safe cholecystectomy.

BACKGROUND: Gaming can serve as an educational tool to allow trainees to practice surgical decision-making in a low-stakes environment. LapBot is a novel free interactive mobile game application that uses artificial intelligence (AI) to provide players with feedback on safe dissection during laparoscopic cholecystectomy (LC). This study aims to provide validity evidence for this mobile game. METHODS: Trainees and surgeons participated by downloading and playing LapBot on their smartphone. Players were presented with intraoperative LC scenes and required to locate their preferred location of dissection of the hepatocystic triangle. They received immediate accuracy scores and personalized feedback using an AI algorithm ("GoNoGoNet") that identifies safe/dangerous zones of dissection. Player scores were assessed globally and across training experience using non-parametric ANOVA. Three-month questionnaires were administered to assess the educational value of LapBot. RESULTS: A total of 903 participants from 64 countries played LapBot. As game difficulty increased, average scores (p < 0.0001) and confidence levels (p < 0.0001) decreased significantly. Scores were significantly positively correlated with players' case volume (p = 0.0002) and training level (p = 0.0003). Most agreed that LapBot should be incorporated as an adjunct into training programs (64.1%), as it improved their ability to reflect critically on feedback they receive during LC (47.5%) or while watching others perform LC (57.5%). CONCLUSIONS: Serious games, such as LapBot, can be effective educational tools for deliberate practice and surgical coaching by promoting learner engagement and experiential learning. Our study demonstrates that players' scores were correlated to their level of expertise, and that after playing the game, most players perceived a significant educational value.

Structured feedback and operative video debriefing with critical view of safety annotation in training of laparoscopic cholecystectomy: a randomized controlled study.

BACKGROUND: The learning curve in minimally invasive surgery (MIS) is lengthened compared to open surgery. It has been reported that structured feedback and training in teams of two trainees improves MIS training and MIS performance. Annotation of surgical images and videos may prove beneficial for surgical training. This study investigated whether structured feedback and video debriefing, including annotation of critical view of safety (CVS), have beneficial learning effects in a predefined, multi-modal MIS training curriculum in teams of two trainees. METHODS: This randomized-controlled single-center study included medical students without MIS experience (n = 80). The participants first completed a standardized and structured multi-modal MIS training curriculum. They were then randomly divided into two groups (n = 40 each), and four laparoscopic cholecystectomies (LCs) were performed on ex-vivo porcine livers each. Students in the intervention group received structured feedback after each LC, consisting of LC performance evaluations through tutor-trainee joint video debriefing and CVS video annotation. Performance was evaluated using global and LC-specific Objective Structured Assessments of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS) scores. RESULTS: The participants in the intervention group had higher global and LC-specific OSATS as well as global and LC-specific GOALS scores than the participants in the control group (25.5 ± 7.3 vs. 23.4 ± 5.1, p = 0.003; 47.6 ± 12.9 vs. 36 ± 12.8, p < 0.001; 17.5 ± 4.4 vs. 16 ± 3.8, p < 0.001; 6.6 ± 2.3 vs. 5.9 ± 2.1, p = 0.005). The intervention group achieved CVS more often than the control group (1. LC: 20 vs. 10 participants, p = 0.037, 2. LC: 24 vs. 8, p = 0.001, 3. LC: 31 vs. 8, p < 0.001, 4. LC: 31 vs. 10, p < 0.001). CONCLUSIONS: Structured feedback and video debriefing with CVS annotation improves CVS achievement and ex-vivo porcine LC training performance based on OSATS and GOALS scores.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions.

OBJECTIVE: To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting. BACKGROUND: To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown. METHODS: Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. RESULTS: Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy. CONCLUSIONS: Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

Preoperative manometric data can predict symptom improvement after per oral endoscopic myotomy for treatment of achalasia.

BACKGROUND: While per oral endoscopic myotomy (POEM) has been shown to be efficacious in the treatment of achalasia, it can be difficult to predict who will have a robust and durable response. Historically, high lower esophageal sphincter pressures have been shown to predict a worse response to endoscopic therapies such as botox therapy. This study was designed to evaluate if modern preoperative manometric data could predict a response to therapy after POEM. METHODS: This was a retrospective study of 144 patients who underwent a POEM at a single institution by a single surgeon over an 8-year period (2014-2022) who had high-resolution manometry performed preoperatively and had an Eckardt symptom score performed both preoperatively and postoperatively. The achalasia type and integrated relaxation pressures (IRP) were then tested for potential correlation with need for any further achalasia interventions postoperatively as well as the degree of Eckardt score reduction using univariate analysis. RESULTS: The achalasia type on preoperatively manometry was not predictive of need for further interventions or degree of Eckardt score reduction (p = 0.74 and 0.44, respectively). A higher IRP was not predictive of need for further interventions however it was predictive of a greater reduction in postoperative Eckardt scores (p = 0.03) as shown by a nonzero regression slope. CONCLUSION: In this study, achalasia type was not a predictive factor in need for further interventions or degree of symptom relief. While IRP was not predictive of need for further interventions, a higher IRP did predict better symptomatic relief postoperatively. This result is opposite that of other endoscopic treatment modalities. Therefore, patients with higher IRP on high-resolution manometry would likely benefit from myotomy which provides significant symptomatic relief postoperatively.

Symptom duration is not associated with per oral endoscopic myotomy (POEM) failure.

INTRODUCTION: Per oral endoscopic myotomy (POEM) is a relatively novel technique to address achalasia; however, little is known about the efficacy of POEM for patients with long-standing achalasia. We hypothesize that patients with long-standing achalasia prior to intervention will be more recalcitrant to POEM than patients with symptoms for a short duration. METHODS: We performed a retrospective analysis of patients with achalasia who received a POEM at a single institution from 2012 to 2022. Patients were grouped into cohorts based on the time of symptom duration: < 1 year, 1-3 years, 4-10 years, > 10 years. POEM failure was defined as need for repeat intervention, symptom recurrence, and a high postoperative Eckart score. Demographic and clinical data were compared between cohorts. Measures of failure multivariable logistic regression analyzed the association between symptom duration and response to POEM. RESULTS: During the study period, 132 patients met inclusion criteria. Patient age at surgery, sex, BMI, Charleston-Deyo Comorbidity Index, and patients with diabetes with and without end organ complications, connective tissue diseases, and patients with ulcer diseases did not differ among cohorts. Patients who have had symptoms for greater than 10 years had significantly more endoscopic interventions prior to their POEM (30% vs, 60% p = 0.002). Patients in all cohorts experienced the same number of symptoms post-POEM. Manometric measurements did not vary across cohorts after POEM. Symptom recurrence, need for repeat endoscopic intervention, repeat surgical intervention, or repeat POEM also did not vary across cohorts. Having symptoms of achalasia > 10 years did not increase the odds POEM failure on multivariable logistical regression. CONCLUSIONS: These data suggest that longer symptom duration is not associated with increased rates of POEM failure. This is promising as clinicians should not exclude patients for POEM eligibility based on duration of symptoms alone.

Do preoperative endoscopic interventions predict symptom recurrence or reintervention after peroral endoscopic myotomy?

BACKGROUND: Studies assessing outcomes of patients undergoing peroral endoscopic myotomy (POEM) after botulinum injection or dilation have had various results with respect to failure, although this has not been differentiated between lack of clinical response and recurrence. We hypothesize that patients with previous endoscopic intervention(s) are more likely to recur than treatment-naïve patients. METHODS: This is a retrospective cohort study of patients that underwent POEM for achalasia at a single tertiary care center between 2011 and 2022. Patients were excluded if they had previous myotomy (POEM or Heller). The remaining patients were stratified into treatment-naïve patients (TN), those with previous botulinum injection (BTX), those with previous dilatation (BD), and those with both previous endoscopic interventions (BOTH). Primary outcome was recurrence indicated by clinical symptoms or need for repeat endoscopic intervention or surgery after originally having clinical resolution (Eckardt ≤ 3). Multivariate logistic regression using preoperative and intraoperative factors was completed to assess odds of recurrence. RESULTS: A total of 164 patients were included in the analysis, 90 TN, 34 BD, 28 BTX, and 12 BOTH. There were no other significant differences in demographics or in preoperative Eckardt score (p = 0.53). There was no difference in the proportion of patients that had postoperative manometry (p = 0.74), symptom recurrence (p = 0.59), surgical intervention (p = 0.16). BTX (14.3%) and BOTH (16.7%) patients had a higher rate of repeat endoscopic intervention than BD and TN patients (5.9% and 1.1%). In the logistic regression analysis, there was no association among the BTX, BD, or BOTH groups compared to the TN group. No odds ratios achieved statistical significance. CONCLUSIONS: There were no increased likelihood of recurrence with botulinum injection or dilatation prior to POEM, implying that they are similarly good candidates compared to treatment-naïve patients.

Preoperative patient factors and anatomy do not predict who will develop reflux after per oral endoscopic myotomy.

BACKGROUND: Per oral endoscopic myotomy (POEM) has been shown to be an efficacious and safe therapy for the treatment of achalasia. Compared to laparoscopic Heller myotomy however, no antireflux procedure is routinely combined with POEM and therefore the development of symptomatic or silent reflux is of concern. This study was designed to determine if various patient factors and anatomy would predict the development of gastroesophageal reflux disease post-operatively. METHODS: This was a retrospective cohort study of all patients who underwent a POEM at a single institution by a single surgeon over an eight-year period (2014-2022). It has been our practice to obtain a postoperative ambulatory pH test on all patients 6 months after POEM off all acid reducing medications. Patients without a postoperative ambulatory esophageal pH monitoring test were excluded. Age, sex, obesity (BMI > 30), achalasia type, presence of a hiatal hernia, history of prior endoscopic achalasia treatments or myotomy were analyzed using univariate analysis as predictive factors for the development of postoperative GERD (DeMeester score > 14.7 on ambulatory pH monitoring). RESULTS: There were 179 total patients included in the study with 42 patients (23.5%) having undergone postoperative ambulatory pH testing. The majority of patients (137 or 76.5%) were lost to follow up and did not undergo ambulatory pH testing. Twenty-three out of those 42 patients (55%) had evidence of GERD on ambulatory pH testing. Multiple preoperative patient characteristics including demographics, manometric results, EGD findings, and history of prior achalasia interventions did not correlate with the development of post-operative GERD. CONCLUSIONS: Despite the high rate of reflux after POEM, there does not appear to be any reliable preoperative indicators of which patients have a higher risk of developing post-operative GERD after POEM.

An analysis of publications originating from abstracts presented at the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) Meeting.

BACKGROUND: Research presentation has benefits, including CV building, networking, and collaboration. A measurable standard for achievement is publication in a peer-reviewed journal. Expectations regarding the likelihood of publication are unknown for studies presented at a national surgical scientific meeting. This study aims to evaluate predictors of manuscript publication arising from abstracts presented at a national surgical scientific meeting. METHODS: Abstracts presented at the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) Meeting 2019 were reviewed. Identification of published manuscripts was completed using MedLine, Embase, and Google Scholar 28 months after the presentation to allow for time for publication. Factors evaluated for association with publication included author and abstract measures. Descriptive analyses and multivariable statistics were performed. RESULTS: 724 abstracts (160 podiums, 564 posters) were included. Of the podium presentations, 128 (80%) were published in a median of 4 months after the presentation. On univariable and multivariable analyses, there was no association between publication and abstract topic, gender, degree, number of publications, or H-indices of first and senior authors. 154 (27.3%) poster presentations were published with a median of 13 months. On univariable analysis, there was a statistically significant difference regarding the abstract topic (p = 0.015) and senior author degree (p = 0.01) between published and unpublished posters. Multivariable analysis demonstrated that colorectal surgery (OR 2.52; CI 1.02-6.23) and metabolic/obesity (OR 2.53; CI 1.09-5.84) are associated with an increased odd of publication. There was an inverse association with female senior authors (OR 0.53; CI 0.29-0.98), while additional degrees (e.g., doctorate and/or master's degree) of the senior authors were associated with an increased publication rate (OR 1.80; CI 1.00-3.22). CONCLUSION: 80% of podiums but only 27% of posters were ultimately published. While some predictors of poster publication were noted, it is unclear if these are why these projects fail to publish. Future research is warranted to determine if there are effective strategies to increase poster publication rates.

SAGES video acquisition framework-analysis of available OR recording technologies by the SAGES AI task force.

BACKGROUND: Surgical video recording provides the opportunity to acquire intraoperative data that can subsequently be used for a variety of quality improvement, research, and educational applications. Various recording devices are available for standard operating room camera systems. Some allow for collateral data acquisition including activities of the OR staff, kinematic measurements (motion of surgical instruments), and recording of the endoscopic video streams. Additional analysis through computer vision (CV), which allows software to understand and perform predictive tasks on images, can allow for automatic phase segmentation, instrument tracking, and derivative performance-geared metrics. With this survey, we summarize available surgical video acquisition technologies and associated performance analysis platforms. METHODS: In an effort promoted by the SAGES Artificial Intelligence Task Force, we surveyed the available video recording technology companies. Of thirteen companies approached, nine were interviewed, each over an hour-long video conference. A standard set of 17 questions was administered. Questions spanned from data acquisition capacity, quality, and synchronization of video with other data, availability of analytic tools, privacy, and access. RESULTS: Most platforms (89%) store video in full-HD (1080p) resolution at a frame rate of 30 fps. Most (67%) of available platforms store data in a Cloud-based databank as opposed to institutional hard drives. CV powered analysis is featured in some platforms: phase segmentation in 44% platforms, out of body blurring or tool tracking in 33%, and suture time in 11%. Kinematic data are provided by 22% and perfusion imaging in one device. CONCLUSION: Video acquisition platforms on the market allow for in depth performance analysis through manual and automated review. Most of these devices will be integrated in upcoming robotic surgical platforms. Platform analytic supplementation, including CV, may allow for more refined performance analysis to surgeons and trainees. Most current AI features are related to phase segmentation, instrument tracking, and video blurring.

Validation of an artificial intelligence platform for the guidance of safe laparoscopic cholecystectomy.

BACKGROUND: Many surgical adverse events, such as bile duct injuries during laparoscopic cholecystectomy (LC), occur due to errors in visual perception and judgment. Artificial intelligence (AI) can potentially improve the quality and safety of surgery, such as through real-time intraoperative decision support. GoNoGoNet is a novel AI model capable of identifying safe ("Go") and dangerous ("No-Go") zones of dissection on surgical videos of LC. Yet, it is unknown how GoNoGoNet performs in comparison to expert surgeons. This study aims to evaluate the GoNoGoNet's ability to identify Go and No-Go zones compared to an external panel of expert surgeons. METHODS: A panel of high-volume surgeons from the SAGES Safe Cholecystectomy Task Force was recruited to draw free-hand annotations on frames of prospectively collected videos of LC to identify the Go and No-Go zones. Expert consensus on the location of Go and No-Go zones was established using Visual Concordance Test pixel agreement. Identification of Go and No-Go zones by GoNoGoNet was compared to expert-derived consensus using mean F1 Dice Score, and pixel accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). RESULTS: A total of 47 frames from 25 LC videos, procured from 3 countries and 9 surgeons, were annotated simultaneously by an expert panel of 6 surgeons and GoNoGoNet. Mean (± standard deviation) F1 Dice score were 0.58 (0.22) and 0.80 (0.12) for Go and No-Go zones, respectively. Mean (± standard deviation) accuracy, sensitivity, specificity, PPV and NPV for the Go zones were 0.92 (0.05), 0.52 (0.24), 0.97 (0.03), 0.70 (0.21), and 0.94 (0.04) respectively. For No-Go zones, these metrics were 0.92 (0.05), 0.80 (0.17), 0.95 (0.04), 0.84 (0.13) and 0.95 (0.05), respectively. CONCLUSIONS: AI can be used to identify safe and dangerous zones of dissection within the surgical field, with high specificity/PPV for Go zones and high sensitivity/NPV for No-Go zones. Overall, model prediction was better for No-Go zones compared to Go zones. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

Artificial Intelligence for Intraoperative Guidance: Using Semantic Segmentation to Identify Surgical Anatomy During Laparoscopic Cholecystectomy.

OBJECTIVE: The aim of this study was to develop and evaluate the performance of artificial intelligence (AI) models that can identify safe and dangerous zones of dissection, and anatomical landmarks during laparoscopic cholecystectomy (LC). SUMMARY BACKGROUND DATA: Many adverse events during surgery occur due to errors in visual perception and judgment leading to misinterpretation of anatomy. Deep learning, a subfield of AI, can potentially be used to provide real-time guidance intraoperatively. METHODS: Deep learning models were developed and trained to identify safe (Go) and dangerous (No-Go) zones of dissection, liver, gallbladder, and hepatocystic triangle during LC. Annotations were performed by 4 high-volume surgeons. AI predictions were evaluated using 10-fold cross-validation against annotations by expert surgeons. Primary outcomes were intersection- over-union (IOU) and F1 score (validated spatial correlation indices), and secondary outcomes were pixel-wise accuracy, sensitivity, specificity, ± standard deviation. RESULTS: AI models were trained on 2627 random frames from 290 LC videos, procured from 37 countries, 136 institutions, and 153 surgeons. Mean IOU, F1 score, accuracy, sensitivity, and specificity for the AI to identify Go zones were 0.53 (±0.24), 0.70 (±0.28), 0.94 (±0.05), 0.69 (±0.20). and 0.94 (±0.03), respectively. For No-Go zones, these metrics were 0.71 (±0.29), 0.83 (±0.31), 0.95 (±0.06), 0.80 (±0.21), and 0.98 (±0.05), respectively. Mean IOU for identification of the liver, gallbladder, and hepatocystic triangle were: 0.86 (±0.12), 0.72 (±0.19), and 0.65 (±0.22), respectively. CONCLUSIONS: AI can be used to identify anatomy within the surgical field. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

Artificial intelligence prediction of cholecystectomy operative course from automated identification of gallbladder inflammation.

BACKGROUND: Operative courses of laparoscopic cholecystectomies vary widely due to differing pathologies. Efforts to assess intra-operative difficulty include the Parkland grading scale (PGS), which scores inflammation from the initial view of the gallbladder on a 1-5 scale. We investigated the impact of PGS on intra-operative outcomes, including laparoscopic duration, attainment of the critical view of safety (CVS), and gallbladder injury. We additionally trained an artificial intelligence (AI) model to identify PGS. METHODS: One surgeon labeled surgical phases, PGS, CVS attainment, and gallbladder injury in 200 cholecystectomy videos. We used multilevel Bayesian regression models to analyze the PGS's effect on intra-operative outcomes. We trained AI models to identify PGS from an initial view of the gallbladder and compared model performance to annotations by a second surgeon. RESULTS: Slightly inflamed gallbladders (PGS-2) minimally increased duration, adding 2.7 [95% compatibility interval (CI) 0.3-7.0] minutes to an operation. This contrasted with maximally inflamed gallbladders (PGS-5), where on average 16.9 (95% CI 4.4-33.9) minutes were added, with 31.3 (95% CI 8.0-67.5) minutes added for the most affected surgeon. Inadvertent gallbladder injury occurred in 25% of cases, with a minimal increase in gallbladder injury observed with added inflammation. However, up to a 28% (95% CI - 2, 63) increase in probability of a gallbladder hole during PGS-5 cases was observed for some surgeons. Inflammation had no substantial effect on whether or not a surgeon attained the CVS. An AI model could reliably (Krippendorff's α = 0.71, 95% CI 0.65-0.77) quantify inflammation when compared to a second surgeon (α = 0.82, 95% CI 0.75-0.87). CONCLUSIONS: An AI model can identify the degree of gallbladder inflammation, which is predictive of cholecystectomy intra-operative course. This automated assessment could be useful for operating room workflow optimization and for targeted per-surgeon and per-resident feedback to accelerate acquisition of operative skills.

Transfer of virtual reality endoscopy training to live animal colonoscopy: a randomized control trial of proficiency vs. repetition-based training.

BACKGROUND: Low first-time pass rates of the Fundamentals of Endoscopic Surgery (FES) exam stimulated development of virtual reality (VR) simulation curricula for test preparation. This study evaluates the transfer of VR endoscopy training to live porcine endoscopy performance and compares the relative effectiveness of a proficiency-based vs repetition-based VR training curriculum. METHODS: Novice endoscopists completed pretesting including the FES manual skills examination and Global Assessment of GI Endoscopic Skills (GAGES) assessment of porcine upper and lower endoscopy. Participants were randomly assigned one of two curricula: proficiency-based or repetition-based. Following curriculum completion, participants post-tested via repeat FES examination and GAGES porcine endoscopy assessments. The two cohorts pre-to-post-test differences were compared using ANCOVA. RESULTS: Twenty-two residents completed the curricula. There were no differences in demographics or clinical endoscopy experience between the groups. The repetition group spent significantly more time on the simulator (repetition: 242.2 min, SD 48.6) compared to the proficiency group (proficiency: 170.0 min, SD 66.3; p = 0.013). There was a significant improvement in porcine endoscopy (pre: 10.6, SD 2.8, post: 16.6, SD 3.4; p < 0.001) and colonoscopy (pre: 10.4, SD 2.7, post: 16.4, SD 4.2; p < 0.001) GAGES scores as well as FES manual skills performance (pre: 270.9, SD 105.5, post: 477.4, SD 68.9; p < 0.001) for the total cohort. There was no difference in post-test GAGES performance or FES manual skills exam performance between the two groups. Both the proficiency and repetition group had a 100% pass rate on the FES skills exam following VR curriculum completion. CONCLUSION: A VR endoscopy curriculum translates to improved performance in upper and lower endoscopy in a live animal model. VR curricula type did not affect FES manual skills examination or live colonoscopy outcomes; however, a proficiency curriculum is less time-consuming and can provide a structured approach to prepare for both the FES exam and clinical endoscopy.

Development and validity evidence of an objective structured assessment of technical skills score for minimally invasive linear-stapled, hand-sewn intestinal anastomoses: the A-OSATS score.

INTRODUCTION: The aim of this study was to develop a reliable objective structured assessment of technical skills (OSATS) score for linear-stapled, hand-sewn closure of enterotomy intestinal anastomoses (A-OSATS). MATERIALS AND METHODS: The Delphi methodology was used to create a traditional and weighted A-OSATS score highlighting the more important steps for patient outcomes according to an international expert consensus. Minimally invasive novices, intermediates, and experts were asked to perform a minimally invasive linear-stapled intestinal anastomosis with hand-sewn closure of the enterotomy in a live animal model either laparoscopically or robot-assisted. Video recordings were scored by two blinded raters assessing intrarater and interrater reliability and discriminative abilities between novices (n = 8), intermediates (n = 24), and experts (n = 8). RESULTS: The Delphi process included 18 international experts and was successfully completed after 4 rounds. A total of 4 relevant main steps as well as 15 substeps were identified and a definition of each substep was provided. A maximum of 75 points could be reached in the unweighted A-OSATS score and 170 points in the weighted A-OSATS score respectively. A total of 41 anastomoses were evaluated. Excellent intrarater (r = 0.807-0.988, p < 0.001) and interrater (intraclass correlation coefficient = 0.923-0.924, p < 0.001) reliability was demonstrated. Both versions of the A-OSATS correlated well with the general OSATS and discriminated between novices, intermediates, and experts defined by their OSATS global rating scale. CONCLUSION: With the weighted and unweighted A-OSATS score, we propose a new reliable standard to assess the creation of minimally invasive linear-stapled, hand-sewn anastomoses based on an international expert consensus. Validity evidence in live animal models is provided in this study. Future research should focus on assessing whether the weighted A-OSATS exceeds the predictive capabilities of patient outcomes of the unweighted A-OSATS and provide further validity evidence on using the score on different anastomotic techniques in humans.

Using Artificial Intelligence-Enhanced Sensing and Wearable Technology in Sports Medicine and Performance Optimisation.

Wearable technologies are small electronic and mobile devices with wireless communication capabilities that can be worn on the body as a part of devices, accessories or clothes. Sensors incorporated within wearable devices enable the collection of a broad spectrum of data that can be processed and analysed by artificial intelligence (AI) systems. In this narrative review, we performed a literature search of the MEDLINE, Embase and Scopus databases. We included any original studies that used sensors to collect data for a sporting event and subsequently used an AI-based system to process the data with diagnostic, treatment or monitoring intents. The included studies show the use of AI in various sports including basketball, baseball and motor racing to improve athletic performance. We classified the studies according to the stage of an event, including pre-event training to guide performance and predict the possibility of injuries; during events to optimise performance and inform strategies; and in diagnosing injuries after an event. Based on the included studies, AI techniques to process data from sensors can detect patterns in physiological variables as well as positional and kinematic data to inform how athletes can improve their performance. Although AI has promising applications in sports medicine, there are several challenges that can hinder their adoption. We have also identified avenues for future work that can provide solutions to overcome these challenges.

Machine Learning for Surgical Phase Recognition: A Systematic Review.

OBJECTIVE: To provide an overview of ML models and data streams utilized for automated surgical phase recognition. BACKGROUND: Phase recognition identifies different steps and phases of an operation. ML is an evolving technology that allows analysis and interpretation of huge data sets. Automation of phase recognition based on data inputs is essential for optimization of workflow, surgical training, intraoperative assistance, patient safety, and efficiency. METHODS: A systematic review was performed according to the Cochrane recommendations and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. PubMed, Web of Science, IEEExplore, GoogleScholar, and CiteSeerX were searched. Literature describing phase recognition based on ML models and the capture of intraoperative signals during general surgery procedures was included. RESULTS: A total of 2254 titles/abstracts were screened, and 35 full-texts were included. Most commonly used ML models were Hidden Markov Models and Artificial Neural Networks with a trend towards higher complexity over time. Most frequently used data types were feature learning from surgical videos and manual annotation of instrument use. Laparoscopic cholecystectomy was used most commonly, often achieving accuracy rates over 90%, though there was no consistent standardization of defined phases. CONCLUSIONS: ML for surgical phase recognition can be performed with high accuracy, depending on the model, data type, and complexity of surgery. Different intraoperative data inputs such as video and instrument type can successfully be used. Most ML models still require significant amounts of manual expert annotations for training. The ML models may drive surgical workflow towards standardization, efficiency, and objectiveness to improve patient outcome in the future. REGISTRATION PROSPERO: CRD42018108907.