Development and validation of metrics for a new RAPN training model.

Robot-assisted partial nephrectomy (RAPN) is a complex and index procedure that urologists need to learn how to perform safely. No validated performance metrics specifically developed for a RAPN training model (TM) exist. A Core Metrics Group specifically adapted human RAPN metrics to be used in a newly developed RAPN TM, explicitly defining phases, steps, errors, and critical errors. A modified Delphi meeting concurred on the face and content validation of the new metrics. One hundred percent consensus was achieved by the Delphi panel on 8 Phases, 32 Steps, 136 Errors and 64 Critical Errors. Two trained assessors evaluated recorded video performances of novice and expert RAPN surgeons executing an emulated RAPN in the newly developed TM. There were no differences in procedure Steps completed by the two groups. Experienced RAPN surgeons made 34% fewer Total Errors than the Novice group. Performance score for both groups was divided at the median score using Total Error scores, into HiError and LoError subgroups. The LowErrs Expert RAPN surgeons group made 118% fewer Total Errors than the Novice HiErrs group. Furthermore, the LowErrs Expert RAPN surgeons made 77% fewer Total Errors than the HiErrs Expert RAPN surgeons. These results established construct and discriminative validity of the metrics. The authors described a novel RAPN TM and its associated performance metrics with evidence supporting their face, content, construct, and discriminative validation. This report and evidence support the implementation of a simulation-based proficiency-based progression (PBP) training program for RAPN.

Current Standards for Training in Robot-assisted Surgery and Endourology: A Systematic Review.

BACKGROUND AND OBJECTIVE: Different training programs have been developed to improve trainee outcomes in urology. However, evidence on the optimal training methodology is sparse. Our aim was to provide a comprehensive description of the training programs available for urological robotic surgery and endourology, assess their validity, and highlight the fundamental elements of future training pathways. METHODS: We systematically reviewed the literature using PubMed/Medline, Embase, and Web of Science databases. The validity of each training model was assessed. The methodological quality of studies on metrics and curricula was graded using the MERSQI scale. The level of evidence (LoE) and level of recommendation for surgical curricula were awarded using the educational Oxford Centre for Evidence-Based Medicine classification. KEY FINDINGS AND LIMITATIONS: A total of 75 studies were identified. Many simulators have been developed to aid trainees in mastering skills required for both robotic and endourology procedures, but only four demonstrated predictive validity. For assessment of trainee proficiency, we identified 18 in robotics training and six in endourology training; however, the majority are Likert-type scales. Although proficiency-based progression (PBP) curricula demonstrated superior outcomes to traditional training in preclinical settings, only four of six (67%) in robotics and three of nine (33%) in endourology are PBP-based. Among these, the Fundamentals of Robotic Surgery and the SIMULATE curricula have the highest LoE (level 1b). The lack of a quantitative synthesis is the main limitation of our study. CONCLUSIONS AND CLINICAL IMPLICATIONS: Training curricula that integrate simulators and PBP methodology have been introduced to standardize trainee outcomes in robotics and endourology. However, evidence regarding their educational impact remains restricted to preclinical studies. Efforts should be made to expand these training programs to different surgical procedures and assess their clinical impact. PATIENT SUMMARY: Simulation-based training and programs in which progression is based on proficiency represent the new standard of quality for achieving surgical proficiency in urology. Studies have demonstrated the educational impact of these approaches. However, there are still no standardized training pathways for several urology procedures.

Proposal of a standardized training curriculum for open and robot-assisted kidney transplantation.

Standardized training programs for open (OKT) and robot-assisted kidney transplantation (RAKT) remain unmet clinical needs. To fill this gap, we designed a modified Delphi Consensus aiming to propose the first structured surgical curricula for both OKT and RAKT, involving a multispecialty international panel of experts. Two web-based surveys were built drafting two separate series of statements (for OKT and RAKT, respectively). The two surveys were delivered to a panel of 63 experts (for OKT) and 21 experts (for RAKT), selected considering their experience, academic profile and involvement in international associations. Consensus was defined as ≥75% agreement between the responders. Overall, 25/63 (40%) and 14/21 (67%) participated in the first round of the Delphi Consensus for the development of the structured curriculum for OKT and RAKT, respectively. During the second round, the experts reached consensus on all statements, as well as on the domains composing the structure of the training pathway: 1) eligibility criteria for trainees and host centres; 2) theoretical training; 3) simulation; 4) real case observation; and 5) modular training in the operating theatre. To the best of our knowledge, this is the first proposal of a dedicated curriculum for OKT and RAKT grounded on insights from international experts of different specialties. Our proposed curriculum provides the foundation to standardize the education and training in KT across Europe and beyond.

Surgical Phase Duration in Robot-Assisted Partial Nephrectomy: A Surgical Data Science Exploration for Clinical Relevance.

(1) Background: Surgical phases form the basic building blocks for surgical skill assessment, feedback, and teaching. The phase duration itself and its correlation with clinical parameters at diagnosis have not yet been investigated. Novel commercial platforms provide phase indications but have not been assessed for accuracy yet. (2) Methods: We assessed 100 robot-assisted partial nephrectomy videos for phase durations based on previously defined proficiency metrics. We developed an annotation framework and subsequently compared our annotations to an existing commercial solution (Touch Surgery, Medtronic™). We subsequently explored clinical correlations between phase durations and parameters derived from diagnosis and treatment. (3) Results: An objective and uniform phase assessment requires precise definitions derived from an iterative revision process. A comparison to a commercial solution shows large differences in definitions across phases. BMI and the duration of renal tumor identification are positively correlated, as are tumor complexity and both tumor excision and renorrhaphy duration. (4) Conclusions: The surgical phase duration can be correlated with certain clinical outcomes. Further research should investigate whether the retrieved correlations are also clinically meaningful. This requires an increase in dataset sizes and facilitation through intelligent computer vision algorithms. Commercial platforms can facilitate this dataset expansion and help unlock the full potential, provided that the phase annotation details are disclosed.

Artificial Intelligence in Surgical Training for Kidney Cancer: A Systematic Review of the Literature.

The prevalence of renal cell carcinoma (RCC) is increasing due to advanced imaging techniques. Surgical resection is the standard treatment, involving complex radical and partial nephrectomy procedures that demand extensive training and planning. Furthermore, artificial intelligence (AI) can potentially aid the training process in the field of kidney cancer. This review explores how artificial intelligence (AI) can create a framework for kidney cancer surgery to address training difficulties. Following PRISMA 2020 criteria, an exhaustive search of PubMed and SCOPUS databases was conducted without any filters or restrictions. Inclusion criteria encompassed original English articles focusing on AI's role in kidney cancer surgical training. On the other hand, all non-original articles and articles published in any language other than English were excluded. Two independent reviewers assessed the articles, with a third party settling any disagreement. Study specifics, AI tools, methodologies, endpoints, and outcomes were extracted by the same authors. The Oxford Center for Evidence-Based Medicine's evidence levels were employed to assess the studies. Out of 468 identified records, 14 eligible studies were selected. Potential AI applications in kidney cancer surgical training include analyzing surgical workflow, annotating instruments, identifying tissues, and 3D reconstruction. AI is capable of appraising surgical skills, including the identification of procedural steps and instrument tracking. While AI and augmented reality (AR) enhance training, challenges persist in real-time tracking and registration. The utilization of AI-driven 3D reconstruction proves beneficial for intraoperative guidance and preoperative preparation. Artificial intelligence (AI) shows potential for advancing surgical training by providing unbiased evaluations, personalized feedback, and enhanced learning processes. Yet challenges such as consistent metric measurement, ethical concerns, and data privacy must be addressed. The integration of AI into kidney cancer surgical training offers solutions to training difficulties and a boost to surgical education. However, to fully harness its potential, additional studies are imperative.

Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial.

Objective: To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity. Background: GEARS-Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations. Methods: In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves. Results: The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments. Conclusions: Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment.

Effect of Metrics-Based Simulation Training to Proficiency on Procedure Quality and Errors Among Novice Cardiac Device Implanters: The IMPROF Randomized Trial.

Importance: In cardiac device implant training, there is no common system to objectively assess trainees' ability to perform tasks at predetermined performance levels before in vivo practice; therefore, patients are potentially exposed to risks related to operators' early learning curve. Objective: To assess the effect on implant performance quality of novel metrics-based simulation training to proficiency (proficiency-based progression [PBP]) vs traditional simulation-based training (SBT). Design, Setting, and Participants: In this prospective randomized trial, conducted between March 8, 2022 and November 24, 2022, novice implanters were randomized (blinded) 1:1 to participate in an SBT curriculum (procedural knowledge e-learning and in-person simulation training) at an international skills training center, with proficiency demonstration requirements at each training stage for advancing (PBP approach) or without the requirements. Ultimately, trainees performed a cardiac resynchronization therapy (CRT) implant using virtual reality simulation. The procedure was video-recorded and subsequently scored using previously validated metrics by 2 independent assessors blinded to group. Physicians who had already implanted more than 20 pacemakers or defibrillators and fewer than 200 CRT systems as the first operator were eligible. Thirty-two implanters from 10 countries voluntarily enrolled in the training program and were randomized; 30 (15 per group) started and completed training. Data analysis was performed from November 27 to December 22, 2022. Intervention: Training with PBP vs SBT. Main Outcome and Measures: The primary outcome comprised 4 objectively assessed performance metrics derived from the video-recordings: number of procedural steps completed, errors, critical errors, and all errors combined. Results: Baseline experience of the 30 participants (19 [63%] male; mean [SD] number of years in implant practice, 2.0 [1.8]; median [IQR] number of implanted pacemakers or defibrillators, 47.5 [30.0-115.0]; median [IQR] number of implanted CRT systems, 3.0 [1.25-10.0]) was similar between study groups. Compared with the SBT group, the PBP group completed 27% more procedural steps (median [IQR], 31 [30-32] vs 24 [22-27]; P < .001) and made 73% fewer errors (median [IQR], 2 [1-3] vs 7 [5-8]; P < .001), 84% fewer critical errors (median [IQR], 1 [0-1] vs 3 [3-5]; P < .001), and 77% fewer all errors combined (errors plus critical errors) (median [IQR], 3 [1-3] vs 11 [8-12]; P < .001); 14 of the 15 PBP trainees (93%) demonstrated the predefined target performance level vs 0 of the 15 SBT trainees. Conclusions and Relevance: In this randomized trial, the PBP approach to novice implanter training generated superior objectively assessed performance vs SBT. If implemented broadly and systematically, PBP training may ensure safe and effective performance standards before trainees proceed to (supervised) in vivo practice. Future studies are needed to verify implications on procedure-related patient complications. Trial Registration: ClinicalTrials.gov Identifier: NCT05952908.

Developing procedural metrics for training in class II posterior composite resin restorations: Outcomes from an international Delphi panel.

OBJECTIVE: The objective of this study was to develop and operationally define procedural performance metrics for the Class II posterior composite resin restoration procedure and to obtain face and content validity through a consensus meeting. METHODS: A Metrics Group consisting of 4 experienced Consultants in Restorative Dentistry, an experienced member of staff from the Restorative Dentistry department in CUDSH and a senior behavioural scientist and education/training expert deconstructed the performance of the Class II posterior composite resin restoration and defined performance metrics. At a modified Delphi meeting, 20 experts in the field of Restorative Dentistry from 11 different Dental Institutions critiqued these metrics and their operational definitions before reaching consensus. RESULTS: Initially performance metrics consisting of 15 Phases, 45 steps, 42 errors and 34 critical errors were identified that characterize the performance of the Class II posterior resin composite procedure. During the Delphi panel these were modified and consensus was reached on 15 Phases (with a change to the initial sequence) with 46 Steps (1 added, 13 modified), 37 Errors (2 added, 1 deleted, 6 reclassified as Critical Error), and 43 Critical Errors (9 added). Consensus on the resulting metrics was obtained and face and content validity verified. CONCLUSIONS: It is possible to develop and objectively define performance metrics that comprehensively characterize a Class II posterior composite resin restoration. It is also possible to achieve consensus on the metrics from a Delphi panel of experts and to confirm the face and content validity of those procedure metrics.

Development of objective performance metrics for ultrasound-guided internal jugular vein cannulation on behalf of the College of Anaesthesiologists of Ireland and observation of scores amongst novice and experienced operators.

BACKGROUND: Ultrasound-guided, internal jugular venous (IJV) cannulation is a core technical skill for anaesthesiologists and intensivists. OBJECTIVES: At a modified Delphi panel meeting, to define and reach consensus on a set of objective ultrasound-guided IJV cannulation performance metrics on behalf of the College of Anaesthesiologists of Ireland (CAI). To use these metrics to objectively score video recordings of novice and experienced anaesthesiologists. DESIGN: An observational study. SETTING: CAI, March to June 2016 and four CAI training hospitals, November 2016 to July 2019. PARTICIPANTS: Metric development group: two CAI national directors of postgraduate training (specialist anaesthesiolgists), a behavioural scientist, a specialist intensivist and a senior CAI trainee. Scoring by two blinded assessors of video recordings of novice ( n  = 11) and experienced anaesthesiologists ( n  = 15) ultrasound-guided IJV cannulations. MAIN OUTCOME MEASURES: A set of agreed CAI objective performance metrics, that is, steps, errors, and critical errors characterising ultrasound-guided IJV cannulation. The difference in performance scores between novice and experienced anaesthesiologists as determined by skill level defined as being below or above the median total error score (errors plus critical errors): that is, low error (LoErr) and high error (HiErr), respectively. RESULTS: The study identified 47 steps, 18 errors and 13 critical errors through six phases.Variability was observed in the range of total error scores for both novice (1 to 3) and experienced (0 to 4.5) anaesthesiologists. This resulted in two further statistically different subgroups (LoErr and HiErr) for both novice ( P  = 0.011) and experienced practitioners ( P  < 0.000). The LoErr-experienced group performed the best in relation to steps, errors and total errors. Critical errors were only observed in the experienced group. CONCLUSION: A set of valid, reliable objective performance metrics has been developed for ultrasound-guided IJV cannulation. Considerable skill variability underlines the need to develop a CAI simulation-training programme using these metrics.

Objective assessment of intraoperative skills for robot-assisted partial nephrectomy (RAPN).

RAPN training usually takes place in-vivo and methods vary across countries/institutions. No common system exists to objectively assess trainee capacity to perform RAPN at predetermined performance levels prior to in-vivo practice. The identification of objective performance metrics for RAPN training is a crucial starting point to improve training and surgical outcomes. The authors sought to examine the reliability, construct and discriminative validity of objective intraoperative performance metrics which best characterize the optimal and suboptimal performance of a reference approach for training novice RAPN surgeons. Seven Novice and 9 Experienced RAPN surgeons video recorded one or two independently performed RAPN procedures in the human. The videos were anonymized and two experienced urology surgeons were trained to reliably score RAPN performance, using previously developed metrics. The assessors were blinded to the performing surgeon, hospital and surgeon group. They independently scored surgeon RAPN performance. Novice and Experienced group performance scores were compared for procedure steps completed and errors made. Each group was divided at the median for Total Errors score, and subgroup scores (i.e., Novice HiErrs and LoErrs, Experienced HiErrs and LoErrs) were compared. The mean inter-rater reliability (IRR) for scoring was 0.95 (range 0.84-1). Compared with Novices, Experienced RAPN surgeons made 69% fewer procedural Total Errors. This difference was accentuated when the LoErr Expert RAPN surgeon's performance was compared with the HiErrs Novice RAPN surgeon's performance with an observed 170% fewer Total Errors. GEARS showed poor reliability (Mean IRR = 0.44; range 0.0-0.8), for scoring RAPN surgical performance. The RAPN procedure metrics reliably distinguish Novice and Experienced surgeon performances. They further differentiated performance levels within a group with similar experiences. Reliable and valid metrics will underpin quality-assured novice RAPN surgical training.

International Expert Consensus on Metric-based Characterization of Robot-assisted Partial Nephrectomy.

BACKGROUND: Robot-assisted partial nephrectomy (RAPN) training usually takes place in vivo, and methods vary across countries/institutions. No common system exists to objectively assess trainee ability to perform RAPN at predetermined performance levels prior to in vivo practice. The identification of objective performance metrics for RAPN training is a crucial starting point to improve training and surgical outcomes. OBJECTIVE: We sought to identify objective performance metrics that best characterize a reference approach to RAPN, and obtain face and content validity from procedure experts through a modified Delphi meeting. DESIGN, SETTING, AND PARTICIPANTS: During a series of online meetings, a core metrics team of three RAPN experts and a senior behavioral scientist performed a detailed task deconstruction of a transperitoneal left-sided RAPN procedure. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: Based on published guidelines, manufacturers' instructions, and unedited videos of RAPN, the team identified performance metrics that constitute an optimal approach for training purposes. The metrics were then subjected to an in-person modified international Delphi panel meeting with 19 expert surgeons. RESULTS AND LIMITATIONS: Eleven procedure phases, with 64 procedure steps, 43 errors, and 39 critical errors, were identified. After the modified Delphi process, the international expert panel added 13 metrics (two steps), six were deleted, and three were modified; 100% panel consensus on the resulting metrics was obtained. Limitations are that the metrics are applicable only to left-sided RAPN cases and some might have been excluded. CONCLUSIONS: Performance metrics that accurately characterize RAPN procedure were developed by a core group of experts. The metrics were then presented to and endorsed by an international panel of very experienced peers. Reliable and valid metrics underpin effective, quality-assured, structured surgical training for RAPN. PATIENT SUMMARY: We organize a meeting among robot-assisted partial nephrectomy (RAPN) experts to identify and reach consensus on objective performance metrics for RAPN training. The metrics are a crucial starting point to improve and quality assure surgical training and patients' clinical outcomes.

Metric-Based Simulation Training to Proficiency for Endovascular Thrombectomy in Ischemic Stroke.

Ischemic stroke is one of the leading causes of death and long-term disability in the West. Mechanical revascularization techniques are considered the standard of care for large vessel occlusive stroke. Traditional apprenticeship models involve doctors training their skills on patients. Simulation platforms have long been recognized as an alternative to this. There has however been very little robust assessment of the training outcomes achieved on some of these platforms. At best, these simulations increase understanding of the procedural process and may help improve some technical skills; at worst they may instill bad habits and poor technique. The prerequisite of any simulation process must be to teach what to do, with which devices, in the correct sequence as well as what not to do. It should provide valid metric-based feedback to the trainee that is objective, transparent, and fair for formative and summative performance feedback. It should deliver a training program that measures the performance progress of trainees against expert benchmarks-benchmarks that represent an evidence-based peer-reviewed standard. In this paper, we present a perspective for PBP training for thrombectomy based on our experience with the process of procedure characterization, metric validation, and early experience of using this approach for proficiency training. Patient outcomes are not only determined by optimal performance in the Angio Suite but also by an efficient patient procedure pathway. There will be value in utilizing the PBP training standard not only for the procedure itself but also for the constituent elements of the stroke pathway to further improve treatment outcomes for ischemic stroke patients.

Discrimination, reliability, sensitivity, and specificity of metric-based assessment of an unstable pertrochanteric 31A2 intramedullary nailing procedure performed by experienced and novice surgeons.

INTRODUCTION: Identifying objective performance metrics for surgical training in orthopedic surgery is imperative for effective training and patient safety. The objective of this study was to determine if an internationally agreed, metric-based objective assessment of video recordings of an unstable pertrochanteric 31A2 intramedullary nailing procedure distinguished between the performance of experienced and novice orthopedic surgeons. MATERIALS AND METHODS: Previously agreed procedure metrics (i.e., 15 phases of the procedure, 75 steps, 88 errors, and 28 sentinel errors) for a closed reduction and standard cephalomedullary nail fixation with a single cephalic element of an unstable pertrochanteric 31A2 fracture. Experienced surgeons trained to assess the performance metrics with an interrater reliability (IRR) > 0.8 assessed 14 videos from 10 novice surgeons (orthopaedic residents/trainees) and 20 videos from 14 experienced surgeons (orthopaedic surgeons) blinded to group and procedure order. RESULTS: The mean IRR of procedure assessments was 0.97. No statistically significant differences were observed between the two groups for Procedure Steps, Errors, Sentinel Errors, and Total Errors. A small number of Experienced surgeons made a similar number of Total Errors as the weakest performing Novices. When the scores of each group were divided at the median Total Error score, large differences were observed between the Experienced surgeons who made the fewest errors and the Novices making the most errors (p < 0.001). Experienced surgeons who made the most errors made significantly more than their Experienced peers (p < 0.003) and the best performing Novices (p < 0.001). Error metrics assessed with Area Under the Curve demonstrated good to excellent Sensitivity and Specificity (0.807-0.907). DISCUSSION: Binary performance metrics previously agreed by an international Delphi meeting discriminated between the objectively assessed video-recorded performance of Experienced and Novice orthopedic surgeons when group scores were sub-divided at the median for Total Errors. Error metrics discriminated best and also demonstrated good to excellent Sensitivity and Specificity. Some very experienced surgeons performed similar to the Novice group surgeons that made most errors. CONCLUSIONS: The procedure metrics used in this study reliably distinguish Novice and Experienced orthopaedic surgeons' performance and will underpin quality-assured novice training.

Intraoperative robotic-assisted low anterior rectal resection performance assessment using procedure-specific binary metrics and a global rating scale.

BACKGROUND: This study aimed to evaluate the use of binary metric-based (proficiency-based progression; PBP) performance assessments and global evaluative assessment of robotic skills (GEARS) of a robotic-assisted low anterior rectal resection (RA-LAR) procedure. METHOD: A prospective study of video analysis of RA-LAR procedures was carried out using the PBP metrics with binary parameters previously developed, and GEARS. Recordings were collected from five novice surgeons (≤30 RA-LAR previously performed) and seven experienced surgeons (>30 RA-LAR previously performed). Two consultant colorectal surgeons were trained to be assessors in the use of PBP binary parameters to evaluate the procedure phases, surgical steps, errors, and critical errors in male and female patients and GEARS scores. Novice and experienced surgeons were categorized and assessed using PBP metrics and GEARS; mean scores obtained were compared for statistical purpose. Also, the inter-rater reliability (IRR) of these assessment tools was evaluated. RESULTS: Twenty unedited recordings of RA-LAR procedures were blindly assessed. Overall, using PBP metric-based assessment, a subgroup of experienced surgeons made more errors (20 versus 16, P = 0.158) and critical errors (9.2 versus 7.8, P = 0.417) than the novice group, although not significantly. However, during the critical phase of RA-LAR, experienced surgeons made significantly fewer errors than the novice group (95% CI of the difference, Lower = 0.104 - Upper = 5.155, df = 11.9, t = 2.23, p = 0.042), and a similar pattern was observed for critical errors. The PBP metric and GEARS assessment tools distinguished between the objectively assessed performance of experienced and novice colorectal surgeons performing RA-LAR (total error scores with PBP metrics, P = 0.019-0.008; GEARS scores, P = 0.029-0.025). GEARS demonstrated poor IRR (mean IRR 0.49) and weaker discrimination between groups (15-41 per cent difference). PBP binary metrics demonstrated good IRR (mean 0.94) and robust discrimination particularly for total error scores (58-64 per cent). CONCLUSIONS: PBP binary metrics seem to be useful for metric-based training for surgeons learning RA-LAR procedures.

Development and Validation of the Metric-Based Assessment of a Robotic Dissection Task on an Avian Model.

INTRODUCTION: The introduction of robot-assisted surgical devices requires the application of objective performance metrics to verify performance levels. OBJECTIVE: To develop and validate (face, content, response process, and construct) the performance metrics for a robotic dissection task using a chicken model. METHODS: In a procedure characterization, we developed the performance metrics (i.e., procedure steps, errors, and critical errors) for a robotic dissection task, using a chicken model. In a modified Delphi panel, 14 experts from four European Union countries agreed on the steps, errors, and critical errors (CEs) of the task. Six experienced surgeons and eight novice urology surgeons performed the robotic dissection task twice on the chicken model. In the Delphi meeting, 100% consensus was reached on five procedure steps, 15 errors and two CEs. Novice surgeons took 20 min to complete the task on trial 1 and 14 min during trial two, whereas experts took 8.2 min and 6.5 min. On average, the Expert Group completed the task 56% faster than the Novice Group and made 46% fewer performance errors. RESULTS: Sensitivity and specificity for procedure errors and time were excellent to good (i.e., 1.0-0.91) but poor (i.e., 0.5) for step metrics. The mean interrater reliability for the assessments by two robotic surgeons was 0.91 (Expert Group inter-rater reliability = 0.92 and Novice Group = 0.9). CONCLUSIONS: We report evidence which supports the demonstration of face, content, and construct validity for a standard and replicable basic robotic dissection task on the chicken model.