Automated segmentation of phases, steps, and tasks in laparoscopic cholecystectomy using deep learning.

BACKGROUND: Video-based review is paramount for operative performance assessment but can be laborious when performed manually. Hierarchical Task Analysis (HTA) is a well-known method that divides any procedure into phases, steps, and tasks. HTA requires large datasets of videos with consistent definitions at each level. Our aim was to develop an AI model for automated segmentation of phases, steps, and tasks for laparoscopic cholecystectomy videos using a standardized HTA. METHODS: A total of 160 laparoscopic cholecystectomy videos were collected from a publicly available dataset known as cholec80 and from our own institution. All videos were annotated for the beginning and ending of a predefined set of phases, steps, and tasks. Deep learning models were then separately developed and trained for the three levels using a 3D Convolutional Neural Network architecture. RESULTS: Four phases, eight steps, and nineteen tasks were defined through expert consensus. The training set for our deep learning models contained 100 videos with an additional 20 videos for hyperparameter optimization and tuning. The remaining 40 videos were used for testing the performance. The overall accuracy for phases, steps, and tasks were 0.90, 0.81, and 0.65 with the average F1 score of 0.86, 0.76 and 0.48 respectively. Control of bleeding and bile spillage tasks were most variable in definition, operative management, and clinical relevance. CONCLUSION: The use of hierarchical task analysis for surgical video analysis has numerous applications in AI-based automated systems. Our results show that our tiered method of task analysis can successfully be used to train a DL model.

Surgical quality assessment of critical view of safety in 283 laparoscopic cholecystectomy videos by surgical residents and surgeons.

INTRODUCTION: Surgical quality assessment has improved the efficacy and efficiency of surgical training and has the potential to optimize the surgical learning curve. In laparoscopic cholecystectomy (LC), the critical view of safety (CVS) can be assessed with a 6-point competency assessment tool (CAT), a task commonly performed by experienced surgeons. The aim of this study is to determine the capability of surgical residents to perform this assessment. METHODS: Both surgeons and surgical residents assessed unedited LC videos using a 6-point CVS, a CAT, using an online video assessment platform. The CAT consists of the following three criteria: 1. clearance of hepatocystic triangle, 2. cystic plate, and 3. two structures connect to the gallbladder, with a maximum of 2 points available for each criterion. A higher score indicates superior surgical performance. The intraclass correlation coefficient (ICC) was employed to assess the inter-rater reliability between surgeons and surgical residents. RESULTS: In total, 283 LC videos were assessed by 19 surgeons and 31 surgical residents. The overall ICC for all criteria was 0.628. Specifically, the ICC scores were 0.504 for criterion 1, 0.639 for criterion 2, and 0.719 for the criterion involving the two structures connected to the gallbladder. Consequently, only the criterion regarding clearance of the hepatocystic triangle exhibited fair agreement, whereas the other two criteria, as well as the overall scores, demonstrated good agreement. In 71% of cases, both surgeons and surgical residents scored a total score either ranging from 0 to 4 or from 5 to 6. CONCLUSION: Compared to the gold standard, i.e., the surgeons' assessments, surgical residents are equally skilled at assessing critical view of safety (CVS) in laparoscopic cholecystectomy (LC) videos. By incorporating video-based assessments of surgical procedures into their training, residents could potentially enhance their learning pace, which may result in better clinical outcomes.

A multi-national, video-based qualitative study to refine training guidelines for assigning an "unsafe" score in laparoscopic cholecystectomy critical view of safety.

BACKGROUND: The critical view of safety (CVS) was incorporated into a novel 6-item objective procedure-specific assessment for laparoscopic cholecystectomy (LC-CVS OPSA) to enhance focus on safe completion of surgical tasks and advance the American Board of Surgery's entrustable professional activities (EPAs) initiative. To enhance instrument development, a feasibility study was performed to elucidate expert surgeon perspectives regarding "safe" vs. "unsafe" practice. METHODS: A multi-national consortium of 11 expert LC surgeons were asked to apply the LC-CVS OPSA to ten LC videos of varying surgical difficulty using a "safe" vs. "unsafe" scale. Raters were asked to provide written rationale for all "unsafe" ratings and invited to provide additional feedback regarding instrument clarity. A qualitative analysis was performed on written responses to extract major themes. RESULTS: Of the 660 ratings, 238 were scored as "unsafe" with substantial variation in distribution across tasks and raters. Analysis of the comments revealed three major categories of "unsafe" ratings: (a) inability to achieve the critical view of safety (intended outcome), (b) safe task completion but less than optimal surgical technique, and (c) safe task completion but risk for potential future complication. Analysis of reviewer comments also identified the potential for safe surgical practice even when CVS was not achieved, either due to unusual anatomy or severe pathology preventing safe visualization. Based upon findings, modifications to the instructions to raters for the LC-CVS OPSA were incorporated to enhance instrument reliability. CONCLUSIONS: A safety-based LC-CVS OPSA has the potential to significantly improve surgical training by incorporating CVS formally into learner assessment. This study documents the perspectives of expert biliary tract surgeons regarding clear identification and documentation of unsafe surgical practice for LC-CVS and enables the development of training materials to improve instrument reliability. Learnings from the study have been incorporated into rater instructions to enhance instrument reliability.

Laparoscopic cholecystectomy critical view of safety (LC-CVS): a multi-national validation study of an objective, procedure-specific assessment using video-based assessment (VBA).

BACKGROUND: A novel 6-item objective, procedure-specific assessment for laparoscopic cholecystectomy incorporating the critical view of safety (LC-CVS OPSA) was developed to support trainee formative and summative assessments. The LC-CVS OPSA included two retraction items (fundus and infundibulum retraction) and four CVS items (hepatocystic triangle visualization, gallbladder-liver separation, cystic artery identification, and cystic duct identification). The scoring rubric for retraction consisted of poor (frequently outside of defined range), adequate (minimally outside of defined range) and excellent (consistently inside defined range) and for CVS items were "poor-unsafe", "adequate-safe", or "excellent-safe". METHODS: A multi-national consortium of 12 expert LC surgeons applied the OPSA-LC CVS to 35 unique LC videos and one duplicate video. Primary outcome measure was inter-rater reliability as measured by Gwet's AC2, a weighted measure that adjusts for scales with high probability of random agreement. Analysis of the inter-rater reliability was conducted on a collapsed dichotomous scoring rubric of "poor-unsafe" vs. "adequate/excellent-safe". RESULTS: Inter-rater reliability was high for all six items ranging from 0.76 (hepatocystic triangle visualization) to 0.86 (cystic duct identification). Intra-rater reliability for the single duplicate video was substantially higher across the six items ranging from 0.91 to 1.00. CONCLUSIONS: The novel 6-item OPSA LC CVS demonstrated high inter-rater reliability when tested with a multi-national consortium of LC expert surgeons. This brief instrument focused on safe surgical practice was designed to support the implementation of entrustable professional activities into busy surgical training programs. Instrument use coupled with video-based assessments creates novel datasets with the potential for artificial intelligence development including computer vision to drive assessment automation.

Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety.

BACKGROUND: Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric. METHODS: The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2. RESULTS: The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases. CONCLUSIONS: A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.

AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers?

INTRODUCTION: Video-based review of surgical procedures has proven to be useful in training by enabling efficiency in the qualitative assessment of surgical skill and intraoperative decision-making. Current video segmentation protocols focus largely on procedural steps. Although some operations are more complex than others, many of the steps in any given procedure involve an intricate choreography of basic maneuvers such as suturing, knot tying, and cutting. The use of these maneuvers at certain procedural steps can convey information that aids in the assessment of the complexity of the procedure, surgical preference, and skill. Our study aims to develop and evaluate an algorithm to identify these maneuvers. METHODS: A standard deep learning architecture was used to differentiate between suture throws, knot ties, and suture cutting on a data set comprised of videos from practicing clinicians (N = 52) who participated in a simulated enterotomy repair. Perception of the added value to traditional artificial intelligence segmentation was explored by qualitatively examining the utility of identifying maneuvers in a subset of steps for an open colon resection. RESULTS: An accuracy of 84% was reached in differentiating maneuvers. The precision in detecting the basic maneuvers was 87.9%, 60%, and 90.9% for suture throws, knot ties, and suture cutting, respectively. The qualitative concept mapping confirmed realistic scenarios that could benefit from basic maneuver identification. CONCLUSIONS: Basic maneuvers can indicate error management activity or safety measures and allow for the assessment of skill. Our deep learning algorithm identified basic maneuvers with reasonable accuracy. Such models can aid in artificial intelligence-assisted video review by providing additional information that can complement traditional video segmentation protocols.

Outcome prediction in bariatric surgery through video-based assessment.

INTRODUCTION: The relationship between intraoperative surgical performance scores and patient outcomes has not been demonstrated at a single-case level. The GEARS score is a Likert-based scale that quantifies robotic surgical proficiency in 5 domains. Given that even highly skilled surgeons can have variability in their skill among their cases, we hypothesized that at a patient level, higher surgical skill as determined by the GEARS score will predict individual patient outcomes. METHODS: Patients undergoing robotic sleeve gastrectomy between July 2018 and January 2021 at a single-health care system were captured in a prospective database. Bivariate Pearson's correlation was used to compare continuous variables, one-way ANOVA for categorical variables compared with a continuous variable, and chi-square for two categorical variables. Significant variables in the univariable screen were included in a multivariable linear regression model. Two-tailed p-value < 0.05 was considered significant. RESULTS: Of 162 patients included, 9 patients (5.5%) experienced a serious morbidity within 30 days. The average excess weight loss (EWL) was 72 ± 12% at 6 months and 74 ± 15% at 12 months. GEARS score was not significantly correlated with EWL at 6 months (p = 0.349), 12 months (p = 0.468), or serious morbidity (p = 0.848) on unadjusted analysis. After adjusting, total GEARS score was not correlated with serious morbidity (p = 0.914); however, GEARS score did predict EWL at 6 (p < 0.001) and 12 months (p < 0.001). All GEARS subcomponent scores, bimanual dexterity, depth perception, efficiency, force sensitivity, and robotic control were predictive of EWL at 6 months (p < 0.001) and 12 months (p < 0.001) on multivariable analysis. CONCLUSION: For patients undergoing sleeve gastrectomy, surgical skill as assessed by the GEARS score was correlated with EWL, suggesting that better performance of a sleeve gastrectomy can result in improved postoperative weight loss.

New generation evaluations: video-based surgical assessments : A technology update.

BACKGROUND: Surgical skill training, assessment, and feedback are the backbone of surgical training. High-quality skills require expert supervision and evaluation throughout a resource-intensive multi-year training process. As technological barriers to internet access and the ability to save and upload surgical videos continue to improve, video-based assessment technology is emerging as a tool that could reshape surgical training for the next generation of surgeons. Video-based assessment platforms have the potential to allow surgeons from across the globe to upload their surgical videos online and receive high-quality, standardized, and unbiased feedback. They combine visual recordings of a surgeon's operative technique, with standardized grading tools that have the potential to significantly impact surgical training and technical skill acquisition across the world. METHOD: The platforms included in this review are in various stages of development after a thorough discussion with national experts on the SAGES TAVAC (Technology and Value Assessments) Committee. For each VBA program, a description of its platform was given and a literature review was obtained using a PubMed search performed from inception until December 2021. RESULTS: The study reviewed all video-based assessment programs currently available in the market, identified their strengths and weaknesses, and how they can be optimized in future. CONCLUSION: The technological platforms will play a key role in the training and technical skill acquisition of the next generation of surgeons and can have an immense impact on patient care across the world. There is immense potential for all these platforms to grow and become incorporated within the framework of an effective surgical training program.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy.

INTRODUCTION: Self-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies. METHODS AND PROCEDURES: Using a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1-3] and senior trainees (PGY 4-5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties. RESULTS: A total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment. CONCLUSIONS: OPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.

Measuring Quality at the Surgeon Level.

Patients expect high-quality surgical care and increasingly are looking for ways to assess the quality of the surgeon they are seeing, but quality measurement is often more complicated than one might expect. Measurement of individual surgeon quality in a manner that allows for comparison among surgeons is particularly difficult. While the concept of measuring individual surgeon quality has a long history, technology now allows for new and innovative ways to measure and achieve surgical excellence. However, some recent efforts to make surgeon-level quality data publicly available have highlighted the challenges of this work. Through this chapter, the reader will be introduced to a brief history of surgical quality measurement, learn about the current state of quality measurement, and get a glimpse into what the future holds.

Blinded intraoperative skill evaluations avoid gender-based bias.

INTRODUCTION: Gender bias has been identified consistently in written performance evaluations. Qualitative tools may provide a standardized way to evaluate surgical skill and minimize gender bias. We hypothesized that there is no difference in operative time or GEARS scores in robotic hysterectomy for men vs women surgeons. METHODS: Patients undergoing robotic hysterectomies performed between June 2019 and March 2020 at 8 hospitals within the same hospital system were captured into a prospective database. GEARS scores were assigned by crowd-sourced evaluators by a third party blinded to any surgeon- or patient-identifying information. One-way ANOVA was used to compare the mean operative time and GEARS scores for each group, and significant variables were included in a one-way ANCOVA to control for confounders. Two-tailed p-value < 0.05 was considered significant. RESULTS: Seventeen women and 13 men performed a total of 188 hysterectomies; women performed 34 (18%) and men performed 153 (81%). Women surgeons had a higher mean operative time (133 ± 58 vs 86.3 ± 46 min, p = 0.024); after adjustment, there were no significant differences in operative time (p = 0.607). There was no significant difference between the genders in total GEARS score (20.0 ± 0.77 vs 20.2 ± 0.70, p = 0.415) or GEARS subcomponent scores: bimanual dexterity (3.98 ± 0.03 vs 4.00 ± 0.03, p = 0.705); depth perception (4.04 ± 0.04 vs 4.05 ± 0.02, p = 0.799); efficiency (3.79 ± 0.02 vs 3.82 ± 0.02, p = 0.437); force sensitivity (4.01 ± 0.04 vs 4.05 ± 0.05, p = 0.533); or robotic control (4.16 ± 0.03 vs 4.26 ± 0.01, p = 0.079). CONCLUSION: There was no difference in GEARS score between men vs women surgeons performing robotic hysterectomies. Video-based blinded assessment of skills may minimize gender biases when evaluating surgical skill for competency evaluation and credentialing.

A comparative analysis of video-based surgical assessment: is evaluation of the entire critical portion of the operation necessary?

BACKGROUND: Previous studies of video-based operative assessments using crowd sourcing have established the efficacy of non-expert evaluations. Our group sought to establish the equivalence of abbreviating video content for operative assessment. METHODS: A single institution video repository of six core general surgery operations was submitted for evaluation. Each core surgery included three unique surgical performances, totaling 18 unique operative videos. Each video was edited using four different protocols based on the critical portion of the operation: (1) custom edited critical portion (2) condensed critical portion (3) first 20 s of every minute of the critical portion, and (4) first 10 s of every minute of the critical portion. In total, 72 individually edited operative videos were submitted to the C-SATS (Crowd-Sourced Assessment of Technical Skills) platform (C-SATS) for evaluation. Aggregate score for study protocol was compared using the Kruskal-Wallis test. A multivariable, multilevel mixed-effects model was constructed to predict total skill assessment scores. RESULTS: Median video lengths for each protocol were: custom, 6:20 (IQR 5:27-7:28); condensed, 10:35 (8:50-12:06); 10 s, 4:35 (2:11-6:09); and 20 s, 9:09 (4:20-12:14). There was no difference in aggregate median score among the four study protocols: custom, 15.7 (14.4-16.2); condensed, 15.8 (15.2-16.4); 10 s, 15.8 (15.3-16.1); 20 s, 16.0 (15.1-16.3); χ2 = 1.661, p = 0.65. Regression modeling demonstrated a significant, but minimal effect of the 10 s and 20 s editing protocols compared to the custom method on individual video score: condensed, + 0.33 (- 0.05-0.70), p = 0.09; 10 s, + 0.29 (0.04-0.55), p = 0.03; 20 s, + 0.40 (0.15-0.66), p = 0.002. CONCLUSION: A standardized protocol for video editing abbreviated surgical performances yields reproducible assessment of surgical aptitude when assessed by non-experts.

The association between video-based assessment of intraoperative technical performance and patient outcomes: a systematic review.

BACKGROUND: Efforts to improve surgical safety and outcomes have traditionally placed little emphasis on intraoperative performance, partly due to difficulties in measurement. Video-based assessment (VBA) provides an opportunity for blinded and unbiased appraisal of surgeon performance. Therefore, we aimed to systematically review the existing literature on the association between intraoperative technical performance, measured using VBA, and patient outcomes. METHODS: Major databases (Medline, Embase, Cochrane Database, and Web of Science) were systematically searched for studies assessing the association of intraoperative technical performance measured by tools supported by validity evidence with short-term (≤ 30 days) and/or long-term postoperative outcomes. Study quality was assessed using the Newcastle-Ottawa Scale. Results were appraised descriptively as study heterogeneity precluded meta-analysis. RESULTS: A total of 11 observational studies were identified involving 8 different procedures in foregut/bariatric (n = 4), colorectal (n = 4), urologic (n = 2), and hepatobiliary surgery (n = 1). The number of surgeons assessed ranged from 1 to 34; patient sample size ranged from 47 to 10,242. High risk of bias was present in 5 of 8 studies assessing short-term outcomes and 2 of 6 studies assessing long-term outcomes. Short-term outcomes were reported in 8 studies (i.e., morbidity, mortality, and readmission), while 6 reported long-term outcomes (i.e., cancer outcomes, weight loss, and urinary continence). Better intraoperative performance was associated with fewer postoperative complications (6 of 7 studies), reoperations (3 of 4 studies), and readmissions (1 of 4 studies). Long-term outcomes were less commonly investigated, with mixed results. CONCLUSION: Current evidence supports an association between superior intraoperative technical performance measured using surgical videos and improved short-term postoperative outcomes. Intraoperative performance analysis using video-based assessment represents a promising approach to surgical quality-improvement.

Multicentric validation of EndoDigest: a computer vision platform for video documentation of the critical view of safety in laparoscopic cholecystectomy.

BACKGROUND: A computer vision (CV) platform named EndoDigest was recently developed to facilitate the use of surgical videos. Specifically, EndoDigest automatically provides short video clips to effectively document the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). The aim of the present study is to validate EndoDigest on a multicentric dataset of LC videos. METHODS: LC videos from 4 centers were manually annotated with the time of the cystic duct division and an assessment of CVS criteria. Incomplete recordings, bailout procedures and procedures with an intraoperative cholangiogram were excluded. EndoDigest leveraged predictions of deep learning models for workflow analysis in a rule-based inference system designed to estimate the time of the cystic duct division. Performance was assessed by computing the error in estimating the manually annotated time of the cystic duct division. To provide concise video documentation of CVS, EndoDigest extracted video clips showing the 2 min preceding and the 30 s following the predicted cystic duct division. The relevance of the documentation was evaluated by assessing CVS in automatically extracted 2.5-min-long video clips. RESULTS: 144 of the 174 LC videos from 4 centers were analyzed. EndoDigest located the time of the cystic duct division with a mean error of 124.0 ± 270.6 s despite the use of fluorescent cholangiography in 27 procedures and great variations in surgical workflows across centers. The surgical evaluation found that 108 (75.0%) of the automatically extracted short video clips documented CVS effectively. CONCLUSIONS: EndoDigest was robust enough to reliably locate the time of the cystic duct division and efficiently video document CVS despite the highly variable workflows. Training specifically on data from each center could improve results; however, this multicentric validation shows the potential for clinical translation of this surgical data science tool to efficiently document surgical safety.

Intellectual property and data ownership in the age of video recording in the operating room.

BACKGROUND: Recording intraoperative videos has become commonplace during surgery, with applications in video-based assessment, education and research. These videos can be both manually and automatically analyzed for performance analysis. A number of commercial entities providing video acquisition and processing has flourished over the recent years. As these companies expand, a number of medico-legal, licensing, intellectual property and data sharing related questions have been raised. METHODS: We performed a qualitative survey of surgeons, hospital administrators, lawyers and commercial entities offering video recording capabilities for serious issues that the average surgeon who records their videos should consider. To address these concerns, we reviewed relevant legal precedent and currently available contracts. RESULTS: We identified several key medico-legal constraints, including data ownership and storage, FDA compliance, privacy and potential for use in litigation, present the legal background and potential solutions. CONCLUSION: Given the availability of surgical recording and the future of video-based performance analysis, surgeons need to become comfortable with the medico-legal issues and the potential solutions available with national physician-led lobbying.

Can Deep Learning Algorithms Help Identify Surgical Workflow and Techniques?

BACKGROUND: Surgical videos are now being used for performance review and educational purposes; however, broad use is still limited due to time constraints. To make video review more efficient, we implemented Artificial Intelligence (AI) algorithms to detect surgical workflow and technical approaches. METHODS: Participants (N = 200) performed a simulated open bowel repair. The operation included two major phases: (1) Injury Identification and (2) Suture Repair. Accordingly, a phase detection algorithm (MobileNetV2+GRU) was implemented to automatically detect the two phases using video data. In addition, participants were noted to use three different technical approaches when running the bowel: (1) use of both hands, (2) use of one hand and one tool, or (3) use of two tools. To discern the three technical approaches, an object detection (YOLOv3) algorithm was implemented to recognize objects that were commonly used during the Injury Identification phase (hands versus tools). RESULTS: The phase detection algorithm achieved high precision (recall) when segmenting the two phases: Injury Identification (86 ± 9% [81 ± 12%]) and Suture Repair (81 ± 6% [81 ± 16%]). When evaluating three technical approaches in running the bowel, the object detection algorithm achieved high average precisions (Hands [99.32%] and Tools [94.47%]). The three technical approaches showed no difference in execution time (Kruskal-Wallis Test: P= 0.062) or injury identification (not missing an injury) (Chi-squared: P= 0.998). CONCLUSIONS: The AI algorithms showed high precision when segmenting surgical workflow and identifying technical approaches. Automation of these techniques for surgical video databases has great potential to facilitate efficient performance review.

Video-based assessment for laparoscopic fundoplication: initial development of a robust tool for operative performance assessment.

INTRODUCTION: While better technical performance correlates with improved outcomes, there is a lack of procedure-specific tools to perform video-based assessment (VBA). SAGES is developing a series of VBA tools with enough validity evidence to allow reliable measurement of surgeon competence. A task force was established to develop a VBA tool for laparoscopic fundoplication using an evidence-based process that can be replicated for additional procedures. The first step in this process was to seek content validity evidence. METHODS: Forty-two subject matter experts (SME) in laparoscopic fundoplication were interviewed to obtain consensus on procedural steps, identify potential variations in technique, and to generate an inventory of required skills and common errors. The results of these interviews were used to inform creation of a task inventory questionnaire (TIQ) that was delivered to a larger SME group (n = 188) to quantify the criticality and difficulty of the procedural steps, the impact of potential errors associated with each step, the technical skills required to complete the procedure, and the likelihood that future techniques or technologies may change the presence or importance of any of these factors. Results of the TIQ were used to generate a list of steps, skills, and errors with strong validity evidence. RESULTS: Initial SMEs interviewed included fellowship program directors (45%), recent fellows (24%), international surgeons (19%), and highly experienced super SMEs with quality outcomes data (12%). Qualitative analysis of interview data identified 6 main procedural steps (visualization, hiatal dissection, fundus mobilization, esophageal mobilization, hiatal repair, and wrap creation) each with 2-5 sub steps. Additionally, the TIQ identified 5-10 potential errors for each step and 11 key technical skills required to perform the procedure. Based on the TIQ, the mean criticality and difficulty scores for the 11/21 sub steps included in the final scoring rubric is 4.66/5 (5 = absolutely essential for patient outcomes) and 3.53/5 (5 = difficulty level requires significant experience and use of alternative strategies to accomplish consistently), respectively. The mean criticality and frequency scores for the 9/11 technical skills included is 4.51/5 and 4.51/5 (5 = constantly used ≥ 80% of the time), respectively. The mean impact score of the 42/47 errors incorporated into the final rubric is 3.85/5 (5 = significant error that is unrecoverable, or even if recovered, likely to have a negative impact on patient outcome). CONCLUSIONS: A rigorous, multi-method process has documented the content validity evidence for the SAGES video-based assessment tool for laparoscopic fundoplication. Work is ongoing to pilot the assessment tool on recorded fundoplication procedures to establish reliability and further validity evidence.

Understanding and developing procedures for video-based assessment in medical education.

INTRODUCTION: Novel uses of video aim to enhance assessment in health-professionals education. Whilst these uses presume equivalence between video and live scoring, some research suggests that poorly understood variations could challenge validity. We aimed to understand examiners' and students' interaction with video whilst developing procedures to promote its optimal use. METHODS: Using design-based research we developed theory and procedures for video use in assessment, iteratively adapting conditions across simulated OSCE stations. We explored examiners' and students' perceptions using think-aloud, interviews and focus group. Data were analysed using constructivist grounded-theory methods. RESULTS: Video-based assessment produced detachment and reduced volitional control for examiners. Examiners ability to make valid video-based judgements was mediated by the interaction of station content and specifically selected filming parameters. Examiners displayed several judgemental tendencies which helped them manage videos' limitations but could also bias judgements in some circumstances. Students rarely found carefully-placed cameras intrusive and considered filming acceptable if adequately justified. DISCUSSION: Successful use of video-based assessment relies on balancing the need to ensure station-specific information adequacy; avoiding disruptive intrusion; and the degree of justification provided by video's educational purpose. Video has the potential to enhance assessment validity and students' learning when an appropriate balance is achieved.

Validation of Motion Tracking Software for Evaluation of Surgical Performance in Laparoscopic Cholecystectomy.

Motion tracking software for assessing laparoscopic surgical proficiency has been proven to be effective in differentiating between expert and novice performances. However, with several indices that can be generated from the software, there is no set threshold that can be used to benchmark performances. The aim of this study was to identify the best possible algorithm that can be used to benchmark expert, intermediate and novice performances for objective evaluation of psychomotor skills. 12 video recordings of various surgeons were collected in a blinded fashion. Data from our previous study of 6 experts and 23 novices was also included in the analysis to determine thresholds for performance. Video recording were analyzed both by the Kinovea 0.8.15 software and a blinded expert observer using the CAT form. Multiple algorithms were tested to accurately identify expert and novice performances. ½ L + [Formula: see text] A + [Formula: see text] J scoring of path length, average movement and jerk index respectively resulted in identifying 23/24 performances. Comparing the algorithm to CAT assessment yielded in a linear regression coefficient R2 of 0.844. The value of motion tracking software in providing objective clinical evaluation and retrospective analysis is evident. Given the prospective use of this tool the algorithm developed in this study proves to be effective in benchmarking performances for psychomotor skills evaluation.

A software-based tool for video motion tracking in the surgical skills assessment landscape.

BACKGROUND: The use of motion tracking has been proved to provide an objective assessment in surgical skills training. Current systems, however, require the use of additional equipment or specialised laparoscopic instruments and cameras to extract the data. The aim of this study was to determine the possibility of using a software-based solution to extract the data. METHODS: 6 expert and 23 novice participants performed a basic laparoscopic cholecystectomy procedure in the operating room. The recorded videos were analysed using Kinovea 0.8.15 and the following parameters calculated the path length, average instrument movement and number of sudden or extreme movements. RESULTS: The analysed data showed that experts had significantly shorter path length (median 127 cm vs. 187 cm, p = 0.01), smaller average movements (median 0.40 cm vs. 0.32 cm, p = 0.002) and fewer sudden movements (median 14.00 vs. 21.61, p = 0.001) than their novice counterparts. CONCLUSION: The use of software-based video motion tracking of laparoscopic cholecystectomy is a simple and viable method enabling objective assessment of surgical performance. It provides clear discrimination between expert and novice performance.