Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review.

BACKGROUND: There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula. METHODS: A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias. RESULTS: Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent. CONCLUSIONS: Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

BACKGROUND: Robotic surgery is increasingly used worldwide to treat many different diseases. The robot is controlled by a surgeon, which may give them greater precision and better outcomes for patients. However, surgeons' robotic skills should be assessed properly, to make sure patients are safe, to improve feedback and for exam assessments for certification to indicate competency. This should be done by experts, using assessment tools that have been agreed upon and proven to work. AIM: This review's aim was to find and explain which training and examination tools are best for assessing surgeons' robotic skills and to find out what gaps remain requiring future research. METHOD: This review searched for all available studies looking at assessment tools in robotic surgery and summarized their findings using several different methods. FINDINGS AND CONCLUSION: Two hundred and forty-seven studies were looked at, finding many assessment tools. Further research is needed for operation-specific and automatic assessment tools before they should be used in the clinical setting.

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review.

INTRODUCTION: There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity. MATERIAL AND METHODS: A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis. RESULTS: A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes. CONCLUSIONS: Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Fertility-sparing surgery of cervical cancer >2 cm (International Federation of Gynecology and Obstetrics 2009 stage IB1-IIA) after neoadjuvant chemotherapy.

OBJECTIVE: To assess the feasibility, safety, oncological, and obstetric outcomes in patients with cervical tumors >2 cm treated with neoadjuvant chemotherapy in preparation for abdominal radical trachelectomy. METHODS: A retrospective analysis of patients with cervical cancer >2 cm (up to 6 cm) was conducted in patients who were selected to receive neoadjuvant chemotherapy before abdominal radical trachelectomy. Surgical and clinical outcomes were examined in relation to radiological and pathological results. In addition, obstetric outcomes were described. The Mann-Whitney U test and Fisher's exact test were performed to compare radiological findings between successful and unsuccessful abdominal radical trachelectomy procedures. International Federation of Gynecology and Obstetrics (FIGO) 2009 staging classification was used for this study. RESULTS: A total of 19 women were treated with neoadjuvant chemotherapy for cervical tumors >2 cm at our institution between May 2006 and July 2018. The median age was 28 years (range 19-36). The distribution of FIGO stages was seven patients stage IB1 (37%), 10 patients stage IB2 (53%), and two patients (10%) stage IIA. Mean clinical tumor size was 4.4 cm (range 3.5-6.0). Histology revealed 74% cases of squamous cell carcinoma. The remaining patients had adenocarcinoma (21%) and only one patient had clear cell adenocarcinoma (5%). Chemotherapy consisted of six weekly cycles of cisplatin (70 mg/m2) and paclitaxel (70 mg/m2). In 15 of the 19 patients (74%) fertility was successfully preserved. In the four patients in whom fertility preservation failed, one patient had stable disease after three cycles and did not meet the criteria for fertility-sparing surgery and three patients had intra- or post-operative indications for adjuvant therapy. Three of the 19 patients (15.7%) had a relapse, two of whom died. One case was in the group of successful abdominal radical trachelectomy. CONCLUSION: Neoadjuvant chemotherapy followed by fertility-sparing surgery may be a feasible and safe option in select patients with cervical tumors >2 cm. Unfavorable prognostic factors are defined as non-responsiveness and non-squamous pathology, which can help in patient selection for fertility-sparing surgery.

Development and evaluation of a societal core robotic surgery accreditation curriculum for the UK.

Standardised proficiency-based progression is the cornerstone of safe robotic skills acquisition, however, is currently lacking within surgical training curricula. Expert consensuses have defined a modular pathway to accredit surgeons. This study aimed to address the lack of a formal, pre-clinical core robotic skills, proficiency-based accreditation curriculum in the UK. Novice robotic participants underwent a four-day pre-clinical core robotic skills curriculum incorporating multimodal assessment. Modifiable-Global Evaluative Assessment of Robotic Skills (M-GEARS), VR-automated performance metrics (APMs) and Objective Clinical Human Reliability Analysis (OCHRA) error methodology assessed performance at the beginning and end of training. Messick's validity concept and a curriculum evaluation model were utilised. Feedback was collated. Proficiency-based progression, benchmarking, tool validity and reliability was assessed through comparative and correlational statistical methods. Forty-seven participants were recruited. Objective assessment of VR and dry models across M-GEARS, APMs and OCHRA demonstrated significant improvements in technical skill (p < 0.001). Concurrent validity between assessment tools demonstrated strong correlation in dry and VR tasks (r = 0.64-0.92, p < 0.001). OCHRA Inter-rater reliability was excellent (r = 0.93, p < 0.001 and 81% matched error events). A benchmark was established with M-GEARS and for the curriculum at 80%. Thirty (63.82%) participants passed. Feedback was 5/5 stars on average, with 100% recommendation. Curriculum evaluation fulfilled all five domains of Messick's validity. Core robotic surgical skills training can be objectively evaluated and benchmarked to provide accreditation in basic robotic skills. A strategy is necessary to enrol standardised curricula into national surgical training at an early stage to ensure patient safety.