Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review.

BACKGROUND: There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula. METHODS: A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias. RESULTS: Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent. CONCLUSIONS: Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

BACKGROUND: Robotic surgery is increasingly used worldwide to treat many different diseases. The robot is controlled by a surgeon, which may give them greater precision and better outcomes for patients. However, surgeons' robotic skills should be assessed properly, to make sure patients are safe, to improve feedback and for exam assessments for certification to indicate competency. This should be done by experts, using assessment tools that have been agreed upon and proven to work. AIM: This review's aim was to find and explain which training and examination tools are best for assessing surgeons' robotic skills and to find out what gaps remain requiring future research. METHOD: This review searched for all available studies looking at assessment tools in robotic surgery and summarized their findings using several different methods. FINDINGS AND CONCLUSION: Two hundred and forty-seven studies were looked at, finding many assessment tools. Further research is needed for operation-specific and automatic assessment tools before they should be used in the clinical setting.

Assessment and application of non-technical skills in robotic-assisted surgery: a systematic review.

BACKGROUND: Undeniably, robotic-assisted surgery (RAS) has become very popular in recent decades, but it has introduced challenges to the workflow of the surgical team. Non-technical skills (NTS) have received less emphasis than technical skills in training and assessment. The systematic review aimed to update the evidence on the role of NTS in robotic surgery, specifically focusing on evaluating assessment tools and their utilisation in training and surgical education in robotic surgery. METHODS: A systematic literature search of PubMed, PsycINFO, MEDLINE, and EMBASE was conducted to identify primary articles on NTS in RAS. Messick's validity framework and the Modified Medical Education Research Study Quality Instrument were utilised to evaluate the quality of the validity evidence of the abstracted articles. RESULTS: Seventeen studies were eligible for the final analysis. Communication, environmental factors, anticipation and teamwork were key NTS for RAS. Team-related factors such as ambient noise and chatter, inconveniences due to repeated requests during the procedure and constraints due to poor design of the operating room may harm patient safety during RAS. Three novel rater-based scoring systems and one sensor-based method for assessing NTS in RAS were identified. Anticipation by the team to predict and execute the next move before an explicit verbal command improved the surgeon's situational awareness. CONCLUSION: This systematic review highlighted the paucity of reporting on non-technical skills in robotic surgery with only three bespoke objective assessment tools being identified. Communication, environmental factors, anticipation, and teamwork are the key non-technical skills reported in robotic surgery, and further research is required to investigate their benefits to improve patient safety during robotic surgery.

The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills.

BACKGROUND: Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment. METHODS: Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed. RESULTS: OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001). CONCLUSION: OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.

Development and evaluation of a societal core robotic surgery accreditation curriculum for the UK.

Standardised proficiency-based progression is the cornerstone of safe robotic skills acquisition, however, is currently lacking within surgical training curricula. Expert consensuses have defined a modular pathway to accredit surgeons. This study aimed to address the lack of a formal, pre-clinical core robotic skills, proficiency-based accreditation curriculum in the UK. Novice robotic participants underwent a four-day pre-clinical core robotic skills curriculum incorporating multimodal assessment. Modifiable-Global Evaluative Assessment of Robotic Skills (M-GEARS), VR-automated performance metrics (APMs) and Objective Clinical Human Reliability Analysis (OCHRA) error methodology assessed performance at the beginning and end of training. Messick's validity concept and a curriculum evaluation model were utilised. Feedback was collated. Proficiency-based progression, benchmarking, tool validity and reliability was assessed through comparative and correlational statistical methods. Forty-seven participants were recruited. Objective assessment of VR and dry models across M-GEARS, APMs and OCHRA demonstrated significant improvements in technical skill (p < 0.001). Concurrent validity between assessment tools demonstrated strong correlation in dry and VR tasks (r = 0.64-0.92, p < 0.001). OCHRA Inter-rater reliability was excellent (r = 0.93, p < 0.001 and 81% matched error events). A benchmark was established with M-GEARS and for the curriculum at 80%. Thirty (63.82%) participants passed. Feedback was 5/5 stars on average, with 100% recommendation. Curriculum evaluation fulfilled all five domains of Messick's validity. Core robotic surgical skills training can be objectively evaluated and benchmarked to provide accreditation in basic robotic skills. A strategy is necessary to enrol standardised curricula into national surgical training at an early stage to ensure patient safety.

A review of minimal access surgery provision and training within the United Kingdom.

When combined with healthcare pressures, the exponential growth of robotic-assisted surgery (RAS) has impacted UK-based training outcomes, including the learning curve to competency. Aim: To ascertain the current provision of RAS and investigate differences in access to minimal access surgical (MAS) facilities and training across the UK. A two-armed electronic survey was conducted. The first arm questioned clinical leads regarding robotic practice and future training provisions. The second investigated trainee and trainers' perceptions of MAS training and facilities. 64% (52/81) of responding trusts utilise a robotic system. The majority (68% [55/81]) have plans to expand or acquire a system within 3 years. 171 responses from 112 UK and Republic of Ireland hospitals were collected for Arm 2. Laparoscopic categories queried whether trainees had access to a formal curriculum, training days and sim-boxes. Most consultants (51.9%) and trainees (51.6%) reported that there was no formal local training curriculum for robotic surgery. Combined responses demonstrated 42.1% (n = 195/463) said "yes", 39.5% (n = 183) "no" and 18.4% (n = 85) "don't know". For combined robotic categories (simulation, training days and operative lists) 28.3% (n = 134/473) responded "yes", 51.6% (n = 244) said "no" and 20.1% (n = 95) said "don't know". This study provides insight into the current provision of robotic-assisted surgery at UK trusts and highlights the need to facilitate regular clinical training and equitable access to MAS simulation within a formal curriculum. This may aid regulation of training in parallel with the expansion of robotic practice and avoid a significant skill acquisition gap and risks to patient safety.

European Robotic Surgery Consensus (ERSC): Protocol for the development of a consensus in robotic training for gastrointestinal surgery trainees.

BACKGROUND: The rapid adoption of robotic surgical systems across Europe has led to a critical gap in training and credentialing for gastrointestinal (GI) surgeons. Currently, there is no existing standardised curriculum to guide robotic training, assessment and certification for GI trainees. This manuscript describes the protocol to achieve a pan-European consensus on the essential components of a comprehensive training programme for GI robotic surgery through a five-stage process. METHODS AND ANALYSIS: In Stage 1, a Steering Committee, consisting of international experts, trainees and educationalists, has been established to lead and coordinate the consensus development process. In Stage 2, a systematic review of existing multi-specialty robotic training curricula will be performed to inform the formulation of key position statements. In Stage 3, a comprehensive survey will be disseminated across Europe to capture the current state of robotic training and identify potential challenges and opportunities for improvement. In Stage 4, an international panel of GI surgeons, trainees, and robotic theatre staff will participate in a three-round Delphi process, seeking ≥ 70% agreement on crucial aspects of the training curriculum. Industry and patient representatives will be involved as external advisors throughout this process. In Stage 5, the robotic training curriculum for GI trainees will be finalised in a dedicated consensus meeting, culminating in the production of an Explanation and Elaboration (E&E) document. REGISTRATION DETAILS: The study protocol has been registered on the Open Science Framework (https://osf.io/br87d/).

Logistical Factors Influencing Cold Ischemia Times in Deceased Donor Kidney Transplants.

BACKGROUND: Prolonged cold ischemia time (CIT) is associated with a significant risk of short- and long-term graft failure in deceased donor kidney transplants across the world. The aim of this prospective longitudinal study was to determine the importance of logistical factors on CIT. METHOD: Data on 1763 transplants were collected prospectively over 14 months from personnel in 16 transplant centers, 19 histocompatibility and immunogenetics laboratories, transport providers, and National Health Service Blood and Transplant. RESULTS: The overall mean CIT was 13.8 hours, with significant center variation (P < 0.0001). Factors that significantly reduced CIT were donation after circulatory death (P = 0.03), shorter transport time (P = 0.0002), use of virtual crossmatch (XM) (P < 0.0001), and use of donor blood for pretransplant XM (P < 0.0001). The CIT for transplants that went ahead with a virtual XM was 3 hours shorter than those requiring a pretransplant XM (P < 0.0001). There was a mean delay of 3 hours in starting transplants despite organ, recipient, and pretransplant XM result being ready, suggesting that theater access contributes significantly to increased CIT. DISCUSSION: This study identifies logistical factors relating to donor, transport, crossmatching, recipient, and theater that impact significantly on CIT in deceased donor renal transplantation, some of which are modifiable; attention should be focussed on addressing all of these.

Global validation of the WSES Sepsis Severity Score for patients with complicated intra-abdominal infections: a prospective multicentre study (WISS Study).

BACKGROUND: To validate a new practical Sepsis Severity Score for patients with complicated intra-abdominal infections (cIAIs) including the clinical conditions at the admission (severe sepsis/septic shock), the origin of the cIAIs, the delay in source control, the setting of acquisition and any risk factors such as age and immunosuppression. METHODS: The WISS study (WSES cIAIs Score Study) is a multicenter observational study underwent in 132 medical institutions worldwide during a four-month study period (October 2014-February 2015). Four thousand five hundred thirty-three patients with a mean age of 51.2 years (range 18-99) were enrolled in the WISS study. RESULTS: Univariate analysis has shown that all factors that were previously included in the WSES Sepsis Severity Score were highly statistically significant between those who died and those who survived (p < 0.0001). The multivariate logistic regression model was highly significant (p < 0.0001, R2 = 0.54) and showed that all these factors were independent in predicting mortality of sepsis. Receiver Operator Curve has shown that the WSES Severity Sepsis Score had an excellent prediction for mortality. A score above 5.5 was the best predictor of mortality having a sensitivity of 89.2 %, a specificity of 83.5 % and a positive likelihood ratio of 5.4. CONCLUSIONS: WSES Sepsis Severity Score for patients with complicated Intra-abdominal infections can be used on global level. It has shown high sensitivity, specificity, and likelihood ratio that may help us in making clinical decisions.