International consensus panel for transurethral resection of bladder tumours metrics: assessment of face and content validity.

OBJECTIVE: To develop performance metrics that objectively define a reference approach to a transurethral resection of bladder tumours (TURBT) procedure, seek consensus on the performance metrics from a group of international experts. METHODS: The characterisation of a reference approach to a TURBT procedure was performed by identifying phases and explicitly defined procedure events (i.e., steps, errors, and critical errors). An international panel of experienced urologists (i.e., Delphi panel) was then assembled to scrutinise the metrics using a modified Delphi process. Based on the panel's feedback, the proposed metrics could be edited, supplemented, or deleted. A voting process was conducted to establish the consensus level on the metrics. Consensus was defined as the panel majority (i.e., >80%) agreeing that the metric definitions were accurate and acceptable. The number of metric units before and after the Delphi meeting were presented. RESULTS: A core metrics group (i.e., characterisation group) deconstructed the TURBT procedure. The reference case was identified as an elective TURBT on a male patient, diagnosed after full diagnostic evaluation with three or fewer bladder tumours of ≤3 cm. The characterisation group identified six procedure phases, 60 procedure steps, 43 errors, and 40 critical errors. The metrics were presented to the Delphi panel which included 15 experts from six countries. After the Delphi, six procedure phases, 63 procedure steps, 47 errors, and 41 critical errors were identified. The Delphi panel achieved a 100% consensus. CONCLUSION: Performance metrics to characterise a reference approach to TURBT were developed and an international panel of experts reached 100% consensus on them. This consensus supports their face and content validity. The metrics can now be used for a proficiency-based progression training curriculum for TURBT.

Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial.

Objective: To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity. Background: GEARS-Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations. Methods: In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves. Results: The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments. Conclusions: Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment.

A cost-effective model for training in Robot-Assisted Sacrocolpopexy.

BACKGROUND: The number of robotically assisted sacrocolpopexy procedures are increasing; therefore, experienced clinicians are needed. Simulation-based cadaver models are challenging in aspects of cost and availability. Therefore, we need to look at alternative and more cost-effective models. OBJECTIVE: The objective of this video was to design a new surgical model for the training of robotic-assisted sacrocolpopexy, which is affordable and accessible. METHODS: We used a whole chicken model to simulate the female pelvic floor. We used Medtronic's Hugo™ RAS system as the robotic console in that procedure. A vaginal cuff was prepared from the proventriculus (stomach), and a Y shaped mesh was secured to the ischium to simulate the sacrocolpopexy procedure. CONCLUSION: This model is easily constructed and in our view is cost-effective. We have demonstrated a new valuable education tool that can serve as a practical simulation model to teach the sacrocolpopexy procedure and to improve trainees' skills. A larger cohort study size is essential to demonstrate the learning curve among young trainees using this simulation model.

Objective assessment of intraoperative skills for robot-assisted partial nephrectomy (RAPN).

RAPN training usually takes place in-vivo and methods vary across countries/institutions. No common system exists to objectively assess trainee capacity to perform RAPN at predetermined performance levels prior to in-vivo practice. The identification of objective performance metrics for RAPN training is a crucial starting point to improve training and surgical outcomes. The authors sought to examine the reliability, construct and discriminative validity of objective intraoperative performance metrics which best characterize the optimal and suboptimal performance of a reference approach for training novice RAPN surgeons. Seven Novice and 9 Experienced RAPN surgeons video recorded one or two independently performed RAPN procedures in the human. The videos were anonymized and two experienced urology surgeons were trained to reliably score RAPN performance, using previously developed metrics. The assessors were blinded to the performing surgeon, hospital and surgeon group. They independently scored surgeon RAPN performance. Novice and Experienced group performance scores were compared for procedure steps completed and errors made. Each group was divided at the median for Total Errors score, and subgroup scores (i.e., Novice HiErrs and LoErrs, Experienced HiErrs and LoErrs) were compared. The mean inter-rater reliability (IRR) for scoring was 0.95 (range 0.84-1). Compared with Novices, Experienced RAPN surgeons made 69% fewer procedural Total Errors. This difference was accentuated when the LoErr Expert RAPN surgeon's performance was compared with the HiErrs Novice RAPN surgeon's performance with an observed 170% fewer Total Errors. GEARS showed poor reliability (Mean IRR = 0.44; range 0.0-0.8), for scoring RAPN surgical performance. The RAPN procedure metrics reliably distinguish Novice and Experienced surgeon performances. They further differentiated performance levels within a group with similar experiences. Reliable and valid metrics will underpin quality-assured novice RAPN surgical training.

Development and Validation of the Metric-Based Assessment of a Robotic Dissection Task on an Avian Model.

INTRODUCTION: The introduction of robot-assisted surgical devices requires the application of objective performance metrics to verify performance levels. OBJECTIVE: To develop and validate (face, content, response process, and construct) the performance metrics for a robotic dissection task using a chicken model. METHODS: In a procedure characterization, we developed the performance metrics (i.e., procedure steps, errors, and critical errors) for a robotic dissection task, using a chicken model. In a modified Delphi panel, 14 experts from four European Union countries agreed on the steps, errors, and critical errors (CEs) of the task. Six experienced surgeons and eight novice urology surgeons performed the robotic dissection task twice on the chicken model. In the Delphi meeting, 100% consensus was reached on five procedure steps, 15 errors and two CEs. Novice surgeons took 20 min to complete the task on trial 1 and 14 min during trial two, whereas experts took 8.2 min and 6.5 min. On average, the Expert Group completed the task 56% faster than the Novice Group and made 46% fewer performance errors. RESULTS: Sensitivity and specificity for procedure errors and time were excellent to good (i.e., 1.0-0.91) but poor (i.e., 0.5) for step metrics. The mean interrater reliability for the assessments by two robotic surgeons was 0.91 (Expert Group inter-rater reliability = 0.92 and Novice Group = 0.9). CONCLUSIONS: We report evidence which supports the demonstration of face, content, and construct validity for a standard and replicable basic robotic dissection task on the chicken model.

Development and validation of the metric-based assessment of a robotic vessel dissection, vessel loop positioning, clip applying and bipolar coagulation task on an avian model.

The evolution of robotic technology and its diffusion does not seem to have been adequately accompanied by the development and implementation of surgeon training programs that ensure skilled and safe device use at the start of the learning curve. The objective of the study is to develop and validate performance metrics for vessel dissection, vessel loop positioning, clip applying and bipolar coagulation using an avian model. Three robotic surgeons and a behavioral scientist characterized the performance metrics of the task according to the proficiency-based progression methodology. Fourteen experienced robotic surgeons from different European countries participated in a modified online Delphi consensus. Eight experienced surgeons and eight novices performed the robotic task twice. In the Delphi meeting, 100% consensus was reached on the performance metrics. Novice surgeons took 26 min to complete the entire task on trial 1 and 20 min on trial 2. Experts took 10.1 min and 9.5 min. On average the Expert Group completed the task 137% faster than the Novice Group. The amount of time to reach the vessel part of the task was also calculated. Novice surgeons took 26 min on trial 1 and 20 min on trial 2. Experts took 5.5 min and 4.8 min. On average the experts reached the vessel 200% faster than the novices. The Expert Group made 155% fewer performance errors than the Novice Group. The mean IRR of video-recorded performance assessments for all metrics was 0.96 (95% confidence intervals (CI) lower = 0.94-upper = 0.98). We report the development and validation for a standard and replicable basic robotic vessel dissection, vessel loop positioning, clip applying and bipolar coagulation task on an avian model. The development of objective performance metrics, based on a transparent and fair methodology (i.e., PBP), is the first fundamental step toward quality assured training. This task developed on the avian model proved to have good results in the validation study.