Assessment and application of non-technical skills in robotic-assisted surgery: a systematic review.

BACKGROUND: Undeniably, robotic-assisted surgery (RAS) has become very popular in recent decades, but it has introduced challenges to the workflow of the surgical team. Non-technical skills (NTS) have received less emphasis than technical skills in training and assessment. The systematic review aimed to update the evidence on the role of NTS in robotic surgery, specifically focusing on evaluating assessment tools and their utilisation in training and surgical education in robotic surgery. METHODS: A systematic literature search of PubMed, PsycINFO, MEDLINE, and EMBASE was conducted to identify primary articles on NTS in RAS. Messick's validity framework and the Modified Medical Education Research Study Quality Instrument were utilised to evaluate the quality of the validity evidence of the abstracted articles. RESULTS: Seventeen studies were eligible for the final analysis. Communication, environmental factors, anticipation and teamwork were key NTS for RAS. Team-related factors such as ambient noise and chatter, inconveniences due to repeated requests during the procedure and constraints due to poor design of the operating room may harm patient safety during RAS. Three novel rater-based scoring systems and one sensor-based method for assessing NTS in RAS were identified. Anticipation by the team to predict and execute the next move before an explicit verbal command improved the surgeon's situational awareness. CONCLUSION: This systematic review highlighted the paucity of reporting on non-technical skills in robotic surgery with only three bespoke objective assessment tools being identified. Communication, environmental factors, anticipation, and teamwork are the key non-technical skills reported in robotic surgery, and further research is required to investigate their benefits to improve patient safety during robotic surgery.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review.

BACKGROUND: There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula. METHODS: A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias. RESULTS: Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent. CONCLUSIONS: Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

BACKGROUND: Robotic surgery is increasingly used worldwide to treat many different diseases. The robot is controlled by a surgeon, which may give them greater precision and better outcomes for patients. However, surgeons' robotic skills should be assessed properly, to make sure patients are safe, to improve feedback and for exam assessments for certification to indicate competency. This should be done by experts, using assessment tools that have been agreed upon and proven to work. AIM: This review's aim was to find and explain which training and examination tools are best for assessing surgeons' robotic skills and to find out what gaps remain requiring future research. METHOD: This review searched for all available studies looking at assessment tools in robotic surgery and summarized their findings using several different methods. FINDINGS AND CONCLUSION: Two hundred and forty-seven studies were looked at, finding many assessment tools. Further research is needed for operation-specific and automatic assessment tools before they should be used in the clinical setting.

UEG and EAES rapid guideline: Systematic review, meta-analysis, GRADE assessment and evidence-informed European recommendations on TaTME for rectal cancer.

BACKGROUND: Evidence and practice recommendations on the use of transanal total mesorectal excision (TaTME) for rectal cancer are conflicting. OBJECTIVE: We aimed to summarize best evidence and develop a rapid guideline using transparent, trustworthy, and standardized methodology. METHODS: We developed a rapid guideline in accordance with GRADE, G-I-N, and AGREE II standards. The steering group consisted of general surgeons, members of the EAES Research Committee/Guidelines Subcommittee with expertise and experience in guideline development, advanced medical statistics and evidence synthesis, biostatisticians, and a guideline methodologist. The guideline panel consisted of four general surgeons practicing colorectal surgery, a radiologist with expertise in rectal cancer, a radiation oncologist, a pathologist, and a patient representative. We conducted a systematic review and the results of evidence synthesis by means of meta-analyses were summarized in evidence tables. Recommendations were authored and published through an online authoring and publication platform (MAGICapp), with the guideline panel making use of an evidence-to-decision framework and a Delphi process to arrive at consensus. RESULTS: This rapid guideline provides a weak recommendation for the use of TaTME over laparoscopic or robotic TME for low rectal cancer when expertise is available. Furthermore, it details evidence gaps to be addressed by future research and discusses policy considerations. The guideline, with recommendations, evidence summaries, and decision aids in user-friendly formats can also be accessed in MAGICapp: https://app.magicapp.org/#/guideline/4494 . CONCLUSIONS: This rapid guideline provides evidence-informed trustworthy recommendations on the use of TaTME for rectal cancer.

Guideline Assessment Project II: statistical calibration informed the development of an AGREE II extension for surgical guidelines.

OBJECTIVE: To inform the development of an AGREE II extension specifically tailored for surgical guidelines. AGREE II was designed to inform the development, reporting, and appraisal of clinical practice guidelines. Previous research has suggested substantial room for improvement of the quality of surgical guidelines. METHODS: A previously published search in MEDLINE for clinical practice guidelines published by surgical scientific organizations with an international scope between 2008 and 2017, resulted in a total of 67 guidelines. The quality of these guidelines was assessed using AGREE II. We performed a series of statistical analyses (reliability, correlation and Factor Analysis, Item Response Theory) with the objective to calibrate AGREE II for use specifically in surgical guidelines. RESULTS: Reliability/correlation/factor analysis and Item Response Theory produced similar results and suggested that a structure of 5 domains, instead of 6 domains of the original instrument, might be more appropriate. Furthermore, exclusion and re-arrangement of items to other domains was found to increase the reliability of AGREE II when applied in surgical guidelines. CONCLUSIONS: The findings of this study suggest that statistical calibration of AGREE II might improve the development, reporting, and appraisal of surgical guidelines.

Surgical challenges and research priorities in the era of the COVID-19 pandemic: EAES membership survey.

BACKGROUND: Healthcare systems and general surgeons are being challenged by the current pandemic. The European Association for Endoscopic Surgery (EAES) aimed to evaluate surgeons' experiences and perspectives, to identify gaps in knowledge, to record shortcomings in resources and to register research priorities. METHODS: An ad hoc web-based survey of EAES members and affiliates was developed by the EAES Research Committee. The questionnaire consisted of 69 items divided into the following sections: (Ι) demographics, (II) institutional burdens and management strategies, and (III) analysis of resource, knowledge, and evidence gaps. Descriptive statistics were summarized as frequencies, medians, ranges,, and interquartile ranges, as appropriate. RESULTS: The survey took place between March 25th and April 16th with a total of 550 surgeons from 79 countries. Eighty-one percent had to postpone elective cases or suspend their practice and 35% assumed roles not related to their primary expertise. One-fourth of respondents reported having encountered abdominal pathologies in COVID-19-positive patients, most frequently acute appendicitis (47% of respondents). The effect of protective measures in surgical or endoscopic procedures on infected patients, the effect of endoscopic surgery on infected patients, and the infectivity of positive patients undergoing laparoscopic surgery were prioritized as knowledge gaps and research priorities. CONCLUSIONS: Perspectives and priorities of EAES members in the era of the pandemic are hereto summarized. Research evidence is urgently needed to effectively respond to challenges arisen from the pandemic.

Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery.

Importance: Complex surgical interventions are inherently prone to variation yet they are not objectively measured. The reasons for outcome differences following cancer surgery are unclear. Objective: To quantify surgical skill within advanced laparoscopic procedures and its association with histopathological and clinical outcomes. Design, Setting, and Participants: This analysis of data and video from the Australasian Laparoscopic Cancer of Rectum (ALaCaRT) and 2-dimensional/3-dimensional (2D3D) multicenter randomized laparoscopic total mesorectal excision trials, which were conducted at 28 centers in Australia, the United Kingdom, and New Zealand, was performed from 2018 to 2019 and included 176 patients with clinical T1 to T3 rectal adenocarcinoma 15 cm or less from the anal verge. Case videos underwent blinded objective analysis using a bespoke performance assessment tool developed with a 62-international expert Delphi exercise and workshop, interview, and pilot phases. Interventions: Laparoscopic total mesorectal excision undertaken with curative intent by 34 credentialed surgeons. Main Outcomes and Measures: Histopathological (plane of mesorectal dissection, ALaCaRT composite end point success [mesorectal fascial plane, circumferential margin, ≥1 mm; distal margin, ≥1 mm]) and 30-day morbidity. End points were analyzed using surgeon quartiles defined by tool scores. Results: The laparoscopic total mesorectal excision performance tool was produced and shown to be reliable and valid for the specialist level (intraclass correlation coefficient, 0.889; 95% CI, 0.832-0.926; P < .001). A substantial variation in tool scores was recorded (range, 25-48). Scores were associated with the number of intraoperative errors, plane of mesorectal dissection, and short-term patient morbidity, including the number and severity of complications. Upper quartile-scoring surgeons obtained excellent results compared with the lower quartile (mesorectal fascial plane: 93% vs 59%; number needed to treat [NNT], 2.9, P = .002; ALaCaRT end point success, 83% vs 58%; NNT, 4; P = .03; 30-day morbidity, 23% vs 50%; NNT, 3.7; P = .03). Conclusions and Relevance: Intraoperative surgical skill can be objectively and reliably measured in complex cancer interventions. Substantial variation in technical performance among credentialed surgeons is seen and significantly associated with clinical and pathological outcomes.

Competency assessment tool for laparoscopic suturing: development and reliability evaluation.

BACKGROUND: Laparoscopic suturing can be technically challenging and requires extensive training to achieve competency. To date no specific and objective assessment method for laparoscopic suturing and knot tying is available that can guide training and monitor performance in these complex surgical skills. In this study we aimed to develop a laparoscopic suturing competency assessment tool (LS-CAT) and assess its inter-observer reliability. METHODS: We developed a bespoke CAT tool for laparoscopic suturing through a structured, mixed methodology approach, overseen by a steering committee with experience in developing surgical assessment tools. A wide Delphi consultation with over twelve experts in laparoscopic surgery guided the development stages of the tool. Following, subjects with different levels of laparoscopic expertise were included to evaluate this tool, using a simulated laparoscopic suturing task which involved placing of two surgical knots. A research assistant video recorded and anonymised each performance. Two blinded expert surgeons assessed the anonymised videos using the developed LS-CAT. The LS-CAT scores of the two experts were compared to assess the inter-observer reliability. Lastly, we compared the subjects' LS-CAT performance scores at the beginning and end of their learning curve. RESULTS: This study evaluated a novel LS-CAT performance tool, comprising of four tasks. Thirty-six complete videos were analysed and evaluated with the LS-CAT, of which the scores demonstrated excellent inter-observer reliability. Cohen's Kappa analysis revealed good to excellent levels of agreement for almost all tasks of both instrument handling and tissue handling (0.87; 0.77; 0.75; 0.86; 0.85, all with p < 0.001). Subjects performed significantly better at the end of their learning curve compared to their first attempt for all LS-CAT items (all with p < 0.001). CONCLUSIONS: We developed the LS-CAT, which is a laparoscopic suturing grading matrix, with excellent inter-rater reliability and to discriminate between experience levels. This LS-CAT has a potential for wider use to objectively assess laparoscopic suturing skills.

Guideline Assessment Project: Filling the GAP in Surgical Guidelines: Quality Improvement Initiative by an International Working Group.

OBJECTIVE: The aim of the study was to identify clinical practice guidelines published by surgical scientific organizations, assess their quality, and investigate the association between defined factors and quality. The ultimate objective was to develop a framework to improve the quality of surgical guidelines. SUMMARY BACKGROUND DATA: Evidence on the quality of surgical guidelines is lacking. METHODS: We searched MEDLINE for clinical practice guidelines published by surgical scientific organizations with an international scope between 2008 and 2017. We investigated the association between the following factors and guideline quality, as assessed using the AGREE II instrument: number of guidelines published within the study period by a scientific organization, the presence of a guidelines committee, applying the GRADE methodology, consensus project design, and the presence of intersociety collaboration. RESULTS: Ten surgical scientific organizations developed 67 guidelines over the study period. The median overall score using AGREE II tool was 4 out of a maximum of 7, whereas 27 (40%) guidelines were not considered suitable for use. Guidelines produced by a scientific organization with an output of ≥9 guidelines over the study period [odds ratio (OR) 3.79, 95% confidence interval (CI), 1.01-12.66, P = 0.048], the presence of a guidelines committee (OR 4.15, 95% CI, 1.47-11.77, P = 0.007), and applying the GRADE methodology (OR 8.17, 95% CI, 2.54-26.29, P < 0.0001) were associated with higher odds of being recommended for use. CONCLUSIONS: Development by a guidelines committee, routine guideline output, and adhering to the GRADE methodology were found to be associated with higher guideline quality in the field of surgery.

Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England.

OBJECTIVES: To develop, validate, and implement a competency assessment tool (CAT) for technical surgical performance in the context of a summative assessment process for the National Training Programme in Laparoscopic Colorectal Surgery (NTP). BACKGROUND: The NTP is an educational initiative by the National Cancer Action Team in England to safely increase the uptake of laparoscopic colorectal surgery. It is the first competency-based national educational initiative for specialist surgeons (consultants), and performance assessment is an integral part of the program. METHODS: Content validity was sought using expert opinion by semistructured interviews and the Delphi method. For validity and reliability studies, NTP apprentices and experts were asked to submit video-recorded cases. Construct validity was established between delegates who passed the assessment and those who failed. Concurrent validity was tested by comparing scores with error counts as identified by observational clinical human reliability analysis. A fully crossed design, using generalizability theory methods and D-studies, was used for reliability. FINDINGS: Interviews and the Delphi method revealed a list of characteristics for assessment. A hybrid structure combining task-specific and generic items was used to include important characteristics into the assessment format. Fifty-four cases were submitted. Overall reliability reached G(ACI) = 0.803 when using 2 cases and 2 assessors. Experts scored significantly better than apprentices (3.19 vs 2.60; P = 0.004), and apprentices who passed had better scores than those who failed (2.95 vs 2.28; P < 0.001). There was an inverse correlation between CAT scores and observational clinical human reliability analysis error counts (ρ = -0.520, P < 0.001). The combination of both methods reached overall sensitivity of 100%, specificity of 83.3%, a positive predictive value of 93.8%, and a negative predictive value of 100%. CONCLUSIONS: The CAT can reliably assess technical performance in laparoscopic colorectal surgery. The use of CATs to judge specialist technical performance before embarking on independent practice of new procedures is achievable on a national scale and can be adapted by other specialties.