Development of a Classification System for Live Surgical Feedback.

Importance: Live feedback in the operating room is essential in surgical training. Despite the role this feedback plays in developing surgical skills, an accepted methodology to characterize the salient features of feedback has not been defined. Objective: To quantify the intraoperative feedback provided to trainees during live surgical cases and propose a standardized deconstruction for feedback. Design, Setting, and Participants: In this qualitative study using a mixed methods analysis, surgeons at a single academic tertiary care hospital were audio and video recorded in the operating room from April to October 2022. Urological residents, fellows, and faculty attending surgeons involved in robotic teaching cases during which trainees had active control of the robotic console for at least some portion of a surgery were eligible to voluntarily participate. Feedback was time stamped and transcribed verbatim. An iterative coding process was performed using recordings and transcript data until recurring themes emerged. Exposure: Feedback in audiovisual recorded surgery. Main Outcomes and Measures: The primary outcomes were the reliability and generalizability of a feedback classification system in characterizing surgical feedback. Secondary outcomes included assessing the utility of our system. Results: In 29 surgical procedures that were recorded and analyzed, 4 attending surgeons, 6 minimally invasive surgery fellows, and 5 residents (postgraduate years, 3-5) were involved. For the reliability of the system, 3 trained raters achieved moderate to substantial interrater reliability in coding cases using 5 types of triggers, 6 types of feedback, and 9 types of responses (prevalence-adjusted and bias-adjusted κ range: a 0.56 [95% CI, 0.45-0.68] minimum for triggers to a 0.99 [95% CI, 0.97-1.00] maximum for feedback and responses). For the generalizability of the system, 6 types of surgical procedures and 3711 instances of feedback were analyzed and coded with types of triggers, feedback, and responses. Significant differences in triggers, feedback, and responses reflected surgeon experience level and surgical task being performed. For example, as a response, attending surgeons took over for safety concerns more often for fellows than residents (prevalence rate ratio [RR], 3.97 [95% CI, 3.12-4.82]; P = .002), and suturing involved more errors that triggered feedback than dissection (RR, 1.65 [95% CI, 1.03-3.33]; P = .007). For the utility of the system, different combinations of trainer feedback had associations with rates of different trainee responses. For example, technical feedback with a visual component was associated with an increased rate of trainee behavioral change or verbal acknowledgment responses (RR, 1.11 [95% CI, 1.03-1.20]; P = .02). Conclusions and Relevance: These findings suggest that identifying different types of triggers, feedback, and responses may be a feasible and reliable method for classifying surgical feedback across several robotic procedures. Outcomes suggest that a system that can be generalized across surgical specialties and for trainees of different experience levels may help galvanize novel surgical education strategies.

A Multi-institution Study on the Association of Virtual Reality Skills with Continence Recovery after Robot-assisted Radical Prostatectomy.

BACKGROUND: Virtual reality (VR) simulators are increasingly being used for surgical skills training. It is unclear what skills are best improved via VR, translate to live surgical skills, and influence patient outcomes. OBJECTIVE: To assess surgeons in VR and live surgery using a suturing assessment tool and evaluate the association between technical skills and a clinical outcome. DESIGN, SETTING, AND PARTICIPANTS: This prospective five-center study enrolled participants who completed VR suturing exercises and provided live surgical video. Graders provided skill assessments using the validated End-To-End Assessment of Suturing Expertise (EASE) suturing evaluation tool. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: A hierarchical Poisson model was used to compare skill scores among cohorts and evaluate the association of scores with clinical outcomes. Spearman's method was used to assess correlation between VR and live skills. RESULTS AND LIMITATIONS: Ten novices, ten surgeons with intermediate expertise (median 64 cases, interquartile range [IQR] 6-80), and 26 expert surgeons (median 850 cases, IQR 375-3000) participated in this study. Intermediate and expert surgeons were significantly more likely to have ideal scores in comparison to novices for the subskills needle hold angle, wrist rotation, and wrist rotation needle withdrawal (p < 0.01). For both intermediate and expert surgeons, there was positive correlation between VR and live skills for needle hold angle (p < 0.05). For expert surgeons, there was a positive association between ideal scores for VR needle hold angle and driving smoothness subskills and 3-mo continence recovery (p < 0.05). Limitations include the size of the intermediate surgeon sample and clinical data limited to expert surgeons. CONCLUSIONS: EASE can be used in VR to identify skills to improve for trainee surgeons. Technical skills that influence postoperative outcomes may be assessable in VR. PATIENT SUMMARY: This study provides insights into surgical skills that translate from virtual simulation to live surgery and that have an impact on urinary continence after robot-assisted removal of the prostate. We also highlight the usefulness of virtual reality in surgical education.

Severity Grading Systems for Intraoperative Adverse Events. A Systematic Review of the Literature and Citation Analysis.

INTRODUCTION: The accurate assessment and grading of adverse events (AE) is essential to ensure comparisons between surgical procedures and outcomes. The current lack of a standardized severity grading system may limit our understanding of the true morbidity attributed to AEs in surgery. The aim of this study is to review the prevalence in which intraoperative adverse event (iAE) severity grading systems are used in the literature, evaluate the strengths and limitations of these systems, and appraise their applicability in clinical studies. METHODS: A systematic review was conducted in accordance with Preferred Reporting Items for Systematic Reviews and Meta-analysis guidelines. PubMed, Web of Science, and Scopus were queried to yield all clinical studies reporting the proposal and/or the validation of iAE severity grading systems. Google Scholar, Web of Science, and Scopus were searched separately to identify the articles citing the systems to grade iAEs identified in the first search. RESULTS: Our search yielded 2957 studies, with 7 studies considered for the qualitative synthesis. Five studies considered only surgical/interventional iAEs, while 2 considered both surgical/interventional and anesthesiologic iAEs. Two included studies validated the iAE severity grading system prospectively. A total of 357 citations were retrieved, with an overall self/nonself-citation ratio of 0.17 (53/304). The majority of citing articles were clinical studies (44.1%). The average number of citations per year was 6.7 citations for each classification/severity system, with only 2.05 citations/year for clinical studies. Of the 158 clinical studies citing the severity grading systems, only 90 (56.9%) used them to grade the iAEs. The appraisal of applicability (mean%/median%) was below the 70% threshold in 3 domains: stakeholder involvement (46/47), clarity of presentation (65/67), and applicability (57/56). CONCLUSION: Seven severity grading systems for iAEs have been published in the last decade. Despite the importance of collecting and grading the iAEs, these systems are poorly adopted, with only a few studies per year using them. A uniform globally implemented severity grading system is needed to produce comparable data across studies and develop strategies to decrease iAEs, further improving patient safety.

Automated Capture of Intraoperative Adverse Events Using Artificial Intelligence: A Systematic Review and Meta-Analysis.

Intraoperative adverse events (iAEs) impact the outcomes of surgery, and yet are not routinely collected, graded, and reported. Advancements in artificial intelligence (AI) have the potential to power real-time, automatic detection of these events and disrupt the landscape of surgical safety through the prediction and mitigation of iAEs. We sought to understand the current implementation of AI in this space. A literature review was performed to PRISMA-DTA standards. Included articles were from all surgical specialties and reported the automatic identification of iAEs in real-time. Details on surgical specialty, adverse events, technology used for detecting iAEs, AI algorithm/validation, and reference standards/conventional parameters were extracted. A meta-analysis of algorithms with available data was conducted using a hierarchical summary receiver operating characteristic curve (ROC). The QUADAS-2 tool was used to assess the article risk of bias and clinical applicability. A total of 2982 studies were identified by searching PubMed, Scopus, Web of Science, and IEEE Xplore, with 13 articles included for data extraction. The AI algorithms detected bleeding (n = 7), vessel injury (n = 1), perfusion deficiencies (n = 1), thermal damage (n = 1), and EMG abnormalities (n = 1), among other iAEs. Nine of the thirteen articles described at least one validation method for the detection system; five explained using cross-validation and seven divided the dataset into training and validation cohorts. Meta-analysis showed the algorithms were both sensitive and specific across included iAEs (detection OR 14.74, CI 4.7-46.2). There was heterogeneity in reported outcome statistics and article bias risk. There is a need for standardization of iAE definitions, detection, and reporting to enhance surgical care for all patients. The heterogeneous applications of AI in the literature highlights the pluripotent nature of this technology. Applications of these algorithms across a breadth of urologic procedures should be investigated to assess the generalizability of these data.

Urinary complications after penile inversion vaginoplasty in transgender women Systematic review and meta-analysis.

INTRODUCTION: Penile inversion vaginoplasty (PIV) remains the gold standard technique for vaginoplasty, a gender-affirming feminizing surgery, but has been associated with urinary complications; however, there is little literature synthesizing urinary complications after PIV surgery, and there is a need to compile these complications to counsel patients pre- and postoperatively on managing surgical expectations. In this systematic review, we summarize the prevalence of urinary complications following PIV. METHODS: We searched the MEDLINE, EMBASE, CINAHL, and Scopus databases in July 2020. The primary outcome was the prevalence of urinary and surgical complications in patients after penile inversion vaginoplasty. Pooled prevalence was determined from extrapolated data. ROBINS-I tool was used to assess study quality. The study was prospectively registered on PROSPERO (CRD 42020204139). RESULTS: Of 843 unique records, 27 articles were pooled for synthesis, with 3388 patients in total. Overall patient satisfaction ranged from 80-100%. The most common urological complications included poor/splayed stream (11.7%, 95% confidence interval [CI] 5.7-19.3), meatal stenosis (6.9%, 95% CI 2.7-12.7), and irritative symptoms (frequency, urgency, nocturia) (11.5%, 95% CI 2.6-25.1). Other urinary complications included retention requiring catheterization (5.1%, 95% CI 0.3-13.8), incontinence (8.7%, 95% CI 3.4-15.6), urethral stricture (4.6%, 95% CI 1.2-9.8), and urinary tract infection (5.6%, 95% CI 2.7-9.4). Most pooled studies had moderate risk of bias. CONCLUSIONS: The available evidence suggests that there is a low prevalence of urinary complications following PIV. Overall, there is a need for standardization of data in transgender surgical care to better understand surgical outcomes and improve postoperative management.

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill.

This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and time-series kinematic data. All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). This work is significant since it demonstrates that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. The study demonstrates an algorithm with state-of-the-art performance in surgical skill assessment. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.

Standardized Reporting of Machine Learning Applications in Urology: The STREAM-URO Framework.

The Standardized Reporting of Machine Learning Applications in Urology (STREAM-URO) framework was developed to provide a set of recommendations to help standardize how machine learning studies in urology are reported. This framework serves three purposes: (1) to promote high-quality studies and streamline the peer review process; (2) to enhance reproducibility, comparability, and interpretability of results; and (3) to improve engagement and literacy of machine learning within the urological community.

Can video games enhance surgical skills acquisition for medical students? A systematic review.

OBJECTIVE: To systematically assess literature examining the impact of being a gamer or participating in video-game-based training on surgical skills acquisition amongst medical students. BACKGROUND: Video games and surgical procedures share similar skills such as visuospatial abilities and hand-eye coordination; therefore, video games can be a valuable tool for surgical training amongst medical students. However, comprehensive, up-to-date systematic reviews are necessary to confirm. METHODS: A systematic literature search of PubMed, MEDLINE, and EMBASE was performed in April 2020 with no limits set on the date of publication. Observational and randomized controlled studies were included. Quality and bias were assessed using the Newcastle-Ottawa Scale for nonrandomized studies and the Grading of Recommendations Assessment, Development and Evaluation system for randomized studies. RESULTS: A total of 575 participants from 16 studies were included. The most common surgical skills tested were laparoscopy (n = 283, from 8 studies) and robotic surgery (n = 199, from 5 studies). A history of gaming and video-game-based training were associated with improved metrics in robotic surgery and laparoscopy, respectively. Neither was beneficial in arthroscopy or bronchoscopy. Studies using the Wii U and Underground reported significant improvement in overall laparoscopic performance. CONCLUSION: Video games demonstrate potential as adjunctive training in surgical skill education, with a history of gaming and video-game-based training being beneficial in robotic surgery and laparoscopy, respectively. Methodological heterogeneity amongst included studies limit the ability to make conclusive decisions; thus, future studies with long-term follow-up, larger sample sizes, outcomes stratified by video-game characteristics, and up-to-date technology are necessary.

Quantifying the "Assistant Effect" in Robotic-Assisted Radical Prostatectomy (RARP): Measures of Technical Performance.

PURPOSE: Surgeons are reliant on the bedside assistant during robotic surgeries. Using a modified global rating scale (GRS), we aim to assess the association between an assistant's technical skill on surgeon performance in Robotic-Assisted Radical Prostatectomy (RARP). METHODS: Prospective, intraoperative video from RARP cases at three centers were collected. Baseline demographic and RARP-experience data were collected from participating surgeons and trainees. The dissection of the prostatic pedicle and neurovascular bundle step (NVB) was analyzed. Expert analysts scored the console surgeon performance using the Global Evaluative Assessment of Robotic Skills (GEARS), and the bedside assistant performance using a modified Objective Structured Assessment of Technical Skills (aOSATS). The primary outcome is the association between console surgeon performance, as measured by GEARS, and assistant skill, as measured by aOSATS. Spearman's rho correlations were used to test the relationship between assistant and surgeon technical performance, and a multivariable linear regression model was created to test this association while controlling for patient factors. RESULTS: 92 RARP cases were available for the analysis, comprising 14 console surgeons and 22 different bedside assistants. In only 5 (5.4%) cases, the neurovascular bundle step was completed by a trainee, and in 13 (14.1%) of cases, a staff-level surgeon acted as the bedside assistant. aOSATS score was significantly associated with robotic console experience (P = 0.011), and prior laparoscopic experience (P < 0.001). Assistant aOSATS score showed a weak but significant correlation with surgeon GEARS score during the neurovascular bundle step (spearman's rho = 0.248, P = 0.028). On linear regression, aOSATS remained a significant predictor of console surgeon performance (P = 0.016), after controlling for patient age and BMI, prostate volume, tumor stage, and presence of nerve-sparing. CONCLUSIONS: This is the first study to assess the association between assistant technical skill and surgeon performance in RARP. Additionally, we have provided validity evidence for a modified OSATS global rating scale for training and assessing bedside assistant performance.

Evidence that surgical performance predicts clinical outcomes.

PURPOSE: Assessment of surgeon performance in the operating room has been identified as a direct method of measuring surgical quality. Studies published in urology and other surgical disciplines have investigated this link directly by measuring surgeon and team performance using methodology supported by validity evidence. This article highlights the key findings of these studies and associated underlying concepts. METHODS: ​Seminal literature from urology and related areas of research was used to inform this review of the performance-outcome relationship in surgery. Current efforts to further our understanding of this concept are discussed, including relevant quality improvement and educational interventions that utilize this relationship. RESULTS: Evidence from multiple surgical specialties and procedures has established the association between surgeon skill and clinically significant patient outcomes. Novel methods of measuring performance utilize surgeon kinematics and artificial intelligence techniques to more reliably and objectively quantify surgical performance. CONCLUSIONS: Future directions include the use of this data to create interventions for quality improvement, as well as innovate the credentialing and recertification process for practicing surgeons.

From box ticking to the black box: the evolution of operating room safety.

PURPOSE: Efforts to improve the safety of patients in the operating room have focused on mitigating harm through the standardization of system, team, and human level factors. This article highlights existing and future methods for enhancing safety in the perioperative setting, and the theory and principles that underpin them. METHODS: Evidence surrounding the development and implementation of select surgical safety interventions is discussed. RESULTS: Work in human factors and engineering that has inspired safety interventions such as the WHO Safety Checklist, and more recently operating room recorders, represents a movement away from traditional, retrospective or reactive methods of studying surgical safety, to prospective and proactive ones. CONCLUSIONS: Future work will examine the effectiveness of these interventions for improving patient outcomes and minimizing iatrogenic harm.

Current Educational Interventions for Improving Technical Skills of Urology Trainees in Endourological Procedures: A Systematic Review.

Objective: Endourology continues to grow with the introduction of new technologies into clinical practice. Simulators and training models have been developed to improve comfort and proficiency in endoscopic procedures. The purpose of this systematic review was to examine the current educational interventions utilized to improve the performance of endourology trainees and to critically appraise the strengths and limitations of each. Methods: A search of the Ovid MEDLINE, EMBASE, PsycINFO, and the Cochrane Library databases was performed to identify literature focused on current educational interventions for improving technical skills of trainees in endourologic procedures. The Medical Education Research Study Quality Instrument (MERSQI) was used to evaluate the methodological quality of the abstracted articles. Results: Of the 2236 articles identified, 22 met the inclusion criteria. The types of educational interventions included: bench/wet lab models, virtual reality simulators, and instructional courses. Metrics used to quantify the impact of these interventions include global rating scales, Objective Structured Assessment of Technical Skills (OSATS) scores, and task-specific checklists. The setting of these evaluations comprises both virtual reality simulators and live surgery. Conclusions: In the surgical education literature, simulation-based training and assessment continues to play a prominent role in urologic training. The educational interventions highlighted in this review address various aspects of endourology, from stone management to transurethral resection. Additional work is needed to correlate technical performance in clinical and nonclinical settings with patient outcomes and develop a focused approach to nontechnical skill training.

Psychology and learning: The role of the clinical learning environment.

The clinical learning environment for the postgraduate education of physicians significantly influences the learning process and the outcomes of learning. Two critical aspects of the learning environment, when viewed through a psychology lens are (1) constructs from psychology relevant to learning, such as cognitive load theory and learner self-efficacy; and (2) psychological attributes of the context in which learning occurs such as psychological safety and "Just Culture". In this paper, we address selected psychological aspects of the clinical learning environment, with a particular focus on the establishment and sustainment of psychological safety in the clinical learning environment for physicians. Psychological safety is defined as individuals' perceptions that they can speak out in the learning or working context without consequences for their professional standing or risks to their status on work teams or groups. We close with seven critical strategies for use by educators, learners, health systems leaders, and other stakeholders to contribute to a clinical environment that optimizes learning. These dimensions can also provide avenues for future research to enhance the community's understanding of psychological constructs operating in the clinical learning environment.

A Novel Method of Setting Performance Standards in Surgery Using Patient Outcomes.

OBJECTIVE: To describe a novel, outcome-based method of standard setting that differentiates between clinical outcomes rather than arbitrary educational goals. BACKGROUND: Standard setting methods used in assessments of procedural skill are currently not evidence-driven or outcome-based. This represents a potential obstacle for the broad implementation of these evaluations in summative assessments such as certification and credentialing. METHODS: The concept is based on deriving a receiver operating characteristic curve from a regression model that incorporates measures of intraoperative surgeon performance and confounding patient characteristics. This allows the creation of a performance standard that best predicts a clinically significant outcome of interest. The discovery cohort used to create the predictive model was derived from pilot data that used the Global Evaluative Assessment of Robotic Skill assessment tool to predict patient urinary continence 3 months following robotic-assisted radical prostatectomy. RESULTS: A receiver operating characteristic curve with an area under the curve of 0.75 was created from predicted probability statistic generated by the predictive model. We chose a predicted probability of 0.35, based on an optimal tradeoff in sensitivity and specificity (Youden Index). Rearranging the regression equation, we determined the performance score required to predict a 35%, patient-adjusted probability of postoperative urinary incontinence. CONCLUSIONS: This novel methodology is context, patient, and assessment-specific. Current standard setting methods do not account for the heterogeneity of the clinical environment. Workplace-based assessments in competency-based medical education require standards that are credible to the educator and the trainee. High-stakes assessments must ensure that surgeons have been evaluated to a standard that prioritizes satisfactory patient outcomes and safety.

Variation and Trends in Antidepressant Prescribing for Men Undergoing Treatment for Nonmetastatic Prostate Cancer: A Population-based Cohort Study.

Psychological distress is prevalent among men with prostate cancer (PCa). However, the variation in antidepressant use among individuals throughout the survivorship period is unknown. We sought to examine the variation and trends in receipt of antidepressants after PCa treatment, among patients with nonmetastatic PCa. Using population-based linked administrative data, we identified men ≥66 yr old who underwent surgery (n=4952), radiotherapy (n=4994), or surveillance (n=2136), and these men were matched to general population controls (n=57127). One year prior to PCa treatment, 7.7% of men received an antidepressant prescription, which increased to 10.5% in the year after treatment. In difference-in-differences analysis, adjusted for demographic and health characteristics, men had increased odds of antidepressant receipt up to 5 yr after surgery (odds ratio [OR] 1.49; 95% confidence interval [CI] 1.35-1.64; p≤0.0001) or radiotherapy (OR 1.33; 95% CI 1.21-1.47; p≤0.0001). Men did not have an increased risk of antidepressant receipt up to 5 yr after surveillance (OR 1.15; 95% CI 0.94-1.41; p=0.16). Limitations include the potential for selection bias and misclassification due to the retrospective design of the study and the use of administrative databases. Thus, men with nonmetastatic PCa who initially receive surgery or radiotherapy, but not those who initially undergo surveillance, have an increased risk of antidepressant receipt after treatment. PATIENT SUMMARY: In this report, we examined antidepressant prescription for men after treatment of nonmetastatic prostate cancer across the entire population of men ≥66 yr in Ontario, Canada, from 2002 to 2009. For men diagnosed with nonmetastatic prostate cancer, the risk of antidepressant receipt at 5 yr after treatment was significantly increased after surgery or radiotherapy, but not after surveillance. Providers and patients should consider the psychological effects of prostate cancer treatment during the survivorship period.