International consensus panel for transurethral resection of bladder tumours metrics: assessment of face and content validity.

OBJECTIVE: To develop performance metrics that objectively define a reference approach to a transurethral resection of bladder tumours (TURBT) procedure, seek consensus on the performance metrics from a group of international experts. METHODS: The characterisation of a reference approach to a TURBT procedure was performed by identifying phases and explicitly defined procedure events (i.e., steps, errors, and critical errors). An international panel of experienced urologists (i.e., Delphi panel) was then assembled to scrutinise the metrics using a modified Delphi process. Based on the panel's feedback, the proposed metrics could be edited, supplemented, or deleted. A voting process was conducted to establish the consensus level on the metrics. Consensus was defined as the panel majority (i.e., >80%) agreeing that the metric definitions were accurate and acceptable. The number of metric units before and after the Delphi meeting were presented. RESULTS: A core metrics group (i.e., characterisation group) deconstructed the TURBT procedure. The reference case was identified as an elective TURBT on a male patient, diagnosed after full diagnostic evaluation with three or fewer bladder tumours of ≤3 cm. The characterisation group identified six procedure phases, 60 procedure steps, 43 errors, and 40 critical errors. The metrics were presented to the Delphi panel which included 15 experts from six countries. After the Delphi, six procedure phases, 63 procedure steps, 47 errors, and 41 critical errors were identified. The Delphi panel achieved a 100% consensus. CONCLUSION: Performance metrics to characterise a reference approach to TURBT were developed and an international panel of experts reached 100% consensus on them. This consensus supports their face and content validity. The metrics can now be used for a proficiency-based progression training curriculum for TURBT.

Surgical skill simulation training to proficiency reduces procedural errors among novice cardiac device implanters: a randomized study.

AIMS: In cardiac device implantation, having both surgical skills and ability to manipulate catheter/lead/wire is crucial. Few cardiologists, however, receive formal surgical training prior to implanting. Skills are mostly acquired directly on-the-job and surgical technique varies across institutions; suboptimal approaches may increase complications. We investigated how novel proficiency-based progression (PBP) simulation training impacts the surgical quality of implantations, compared to traditional simulation (SIM) training. METHODS AND RESULTS: In this international prospective study, novice implanters were randomized (blinded) 1:1 to participate in a simulation-based procedure training curriculum, with proficiency demonstration requirements for advancing (PBP approach) or without (SIM). Ultimately, trainees performed the surgical tasks of an implant on a porcine tissue that was video-recorded and then scored by two independent assessors (blinded to group), using previously validated performance metrics. Primary outcomes were the number of procedural Steps Completed, Critical Errors, Errors (non-critical), and All Errors Combined. Thirty novice implanters from 10 countries participated. Baseline experiences were similar between groups. Compared to SIM-trained, the PBP-trained group completed on average 11% more procedural Steps (P < 0.001) and made 61.2% fewer Critical Errors (P < 0.001), 57.1% fewer Errors (P = 0.140), and 60.7% fewer All Errors Combined (P = 0.001); 11/15 (73%) PBP trainees demonstrated the predefined target performance level vs. 3/15 SIM trainees (20%) in the video-recorded performance. CONCLUSION: Proficiency-based progression training produces superior objectively assessed novice operators' surgical performance in device implantation compared with traditional (simulation) training. Systematic PBP incorporation into formal academic surgical skills training is recommended before in vivo device practice. Future studies will quantify PBP training's effect on surgery-related device complications.

Analysis of training pathway to reach expert performance levels based on proficiency-based progression in robotic-assisted minimally invasive esophagectomy (RAMIE).

BACKGROUND: Robotic-assisted minimally invasive esophagectomy (RAMIE) was first introduced in 2003 and has since then shown to significantly improve the postoperative course. Previous studies have shown that a structured training pathway based on proficiency-based progression using individual skill levels as measures of reach of competence can enhance surgical performance. The aim of this study was to evaluate and help understand our pathway to reach surgical expert levels using a proficiency-based approach introducing RAMIE at our German high-volume center. METHODS: All patients undergoing RAMIE performed by two experienced surgeons for esophageal cancer since the introduction of the robotic technique in 2017 was included in this analysis. Intraoperative outcomes and postoperative outcomes were included in the analysis. The cumulative sum method was used to analyze how many cases are needed to reach expert levels for different performance characteristics and skill sets during robotic-assisted minimally invasive esophagectomy. RESULTS: From 06/2017 to 03/2022, a total of 154 patients underwent RAMIE at our facility and were included in the analysis. An advancement in performance level was observed for total operating time after 70 cases and for thoracic operative time after 79 cases. Lymph node yield showed an increase up until case 60 in the CUSUM analysis. Length of hospital stay stabilized after case 55. The CCI score inflection point was at case 55 in both CUSUM and regression analyses. Anastomotic leak rate stabilized at case 38 and showed another inflection point after case 83. CONCLUSION: Our data and analysis showed the progression from proficient to expert performance levels during the implementation of RAMIE at a European high-volume center. Further analysis of surgeons, especially with a different training status has yet to reveal if the caseloads found in this study are universally applicable. However, skill acquisition and respective measures of such are diverse and as a great range of number of cases was observed, we believe that the learning curve and ascent in performance levels cannot be defined by one parameter alone.

Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial.

OBJECTIVE: To determine whether proficiency-based progression (PBP) training leads to better robotic surgical performance compared to traditional training (TT), given that the value of PBP training for learning robotic surgical skills is unclear. MATERIALS AND METHODS: The PROVESA trial is a multicentric, prospective, randomized and blinded clinical study comparing PBP training with TT for robotic suturing and knot-tying anastomosis skills. A total of 36 robotic surgery-naïve junior residents were recruited from 16 training sites and 12 residency training programmes. Participants were randomly allocated to metric-based PBP training or the current standard of care TT, and compared at the end of training. The primary outcome was percentage of participants reaching the predefined proficiency benchmark. Secondary outcomes were the numbers of procedure steps and errors made. RESULTS: Of the group that received TT, 3/18 reached the proficiency benchmark versus 12/18 of the PBP group (i.e. the PBP group were ~10 times as likely to demonstrate proficiency [P = 0.006]). The PBP group demonstrated a 51% reduction in number of performance errors from baseline to the final assessment (18.3 vs 8.9). The TT group demonstrated a marginal improvement (15.94 vs 15.44) in errors made. CONCLUSIONS: The PROVESA trial is the first prospective randomized controlled trial on basic skills training in robotic surgery. Implementation of a PBP training methodology resulted in superior surgical performance for robotic suturing and knot-tying anastomosis performance. Compared to TT, better surgical quality could be obtained by implementing PBP training for basic skills in robotic surgery.

[Impact of simulation-based training in endourology: A systematic review of the literature]. / Impact de la formation fondée sur la simulation en endo-urologie : revue systématique de la littérature.

INTRODUCTION: The impact of simulation on the acquisition of surgical skills as well as their transfer to the operating room is still debated. The objective was to assess these two specific points, focusing on the field of endourology. METHODS: A systematic review of the literature, following the PRIMA statement, was performed using Medline database through September 2021 without time limit. Studies focusing on the impact of simulators on the acquisition of surgical technical skills as well as their transfer to the operating room in the field of endourology were included. The endourological procedures were: cystoscopy, ureteroscopy, percutaneous nephrolithotomy, endoscopic treatment of benign prostatic hyperplasia, endoscopic bladder resection. RESULTS: Among the 11,442 publications identified, fifty-two studies were included in the analysis. The majority reported an improvement in procedure time of the requested tasks and dexterity of participants, regardless of the type of simulator and procedure. The level of evidence of included studies was often low. Few studies evaluated the transfer of acquired skills from the simulator to the operating room. CONCLUSION: This review showed the positive impact of simulation on the acquisition of technical skills in endourology. However, in order to include proficiency-based progression in the curriculum of trainees, some parameters such as the choice of reference simulators, choice of tasks, and method of validation of acquired skills must be validated in a consensual manner to offer a quality training.

Evolving robotic surgery training and improving patient safety, with the integration of novel technologies.

INTRODUCTION: Robot-assisted surgery is becoming increasingly adopted by multiple surgical specialties. There is evidence of inherent risks of utilising new technologies that are unfamiliar early in the learning curve. The development of standardised and validated training programmes is crucial to deliver safe introduction. In this review, we aim to evaluate the current evidence and opportunities to integrate novel technologies into modern digitalised robotic training curricula. METHODS: A systematic literature review of the current evidence for novel technologies in surgical training was conducted online and relevant publications and information were identified. Evaluation was made on how these technologies could further enable digitalisation of training. RESULTS: Overall, the quality of available studies was found to be low with current available evidence consisting largely of expert opinion, consensus statements and small qualitative studies. The review identified that there are several novel technologies already being utilised in robotic surgery training. There is also a trend towards standardised validated robotic training curricula. Currently, the majority of the validated curricula do not incorporate novel technologies and training is delivered with more traditional methods that includes centralisation of training services with wet laboratories that have access to cadavers and dedicated training robots. CONCLUSIONS: Improvements to training standards and understanding performance data have good potential to significantly lower complications in patients. Digitalisation automates data collection and brings data together for analysis. Machine learning has potential to develop automated performance feedback for trainees. Digitalised training aims to build on the current gold standards and to further improve the 'continuum of training' by integrating PBP training, 3D-printed models, telementoring, telemetry and machine learning.

Training in robotic surgery, replicating the airline industry. How far have we come?

INTRODUCTION: As the role of robot-assisted surgery continues to expand, development of standardised and validated training programmes is becoming increasingly important. We aim to compare current robotic training curricula with training in aviation, to evaluate current similarities and to provide insight into how healthcare can further learn from replicating initiatives in aviation training. METHODS: A systematic literature review of the current evidence was conducted online and relevant publications and information were identified. Evaluation and comparison between training in robotic surgery and the aviation industry was performed. RESULTS: There are significant similarities between modern robotic training curricula and pilot training. Both undergo basic training before proceeding to advanced training. Aviation training methods include classroom instruction, e-learning and practical training, in both the aircraft and flight simulation training devices. Both surgeon and pilot training include technical and procedural instruction as well as training in non-technical skills such as crisis management, decision making, leadership and communication. However, there is more regulation in aviation, with international standards for training curricula, simulation devices and instructors/trainers that are legally binding. Continuous learning with re-qualification with benchmarked high stakes tests are also mandatory throughout a pilot's and instructor's career. CONCLUSION: Robotic surgeons and pilots roles have many fundamental similarities. Both work with expensive and complex technology requiring high levels of skills, within working environments with high physiological and psychological stress levels. Whilst many initiatives in aviation training have already been replicated in surgical training there remain considerable differences in regulation. Adopting established and proven aviation methods of assessment and regulation could help robotic surgical training become more efficient, more effective and ultimately safer.

European expert consensus on a structured approach to training robotic-assisted low anterior resection using performance metrics.

AIM: The aim was to develop and operationally define 'performance metrics' that characterize a reference approach to robotic-assisted low anterior resection (RA-LAR) and to obtain face and content validity through a consensus meeting. METHOD: Three senior colorectal surgeons with robotic experience and a senior behavioural scientist formed the Metrics Group. We used published guidelines, training materials, manufacturers' instructions and unedited videos of RA-LAR to deconstruct the operation into defined, measurable components - performance metrics (i.e. procedure phases, steps, errors and critical errors). The performance metrics were then subjected to detailed critique by 18 expert colorectal surgeons in a modified Delphi process. RESULTS: Performance metrics for RA-LAR had 15 procedure phases, 128 steps, 89 errors and 117 critical errors in women, 88 errors and 118 critical errors in men. After the modified Delphi process the final performance metrics consisted of 14 procedure phases, 129 steps, 88 errors and 115 critical errors in women, 87 errors and 116 critical errors in men. After discussion by the Delphi panel, all procedure phases received unanimous consensus apart from phase I (patient positioning and preparation, 83%) and phase IV (docking, 94%). CONCLUSION: A robotic rectal operation can be broken down into procedure phases, steps, with errors and critical errors, known as performance metrics. The face and content of these metrics have been validated by a large group of expert robotic colorectal surgeons from Europe. We consider the metrics essential for the development of a structured training curriculum and standardized procedural assessment for RA-LAR.

Acute surgical wound-dressing procedure: Description of the steps involved in the development and validation of an observational metric.

The aim of this study was to develop an observational metric that could be used to assess the performance of a practitioner in completing an acute surgical wound-dressing procedure using aseptic non-touch technique (ANTT). A team of clinicians, academics, and researchers came together to develop an observational metric using an iterative six-stage process, culminating in a Delphi panel meeting. A scoping review of the literature provided a background empirical perspective relating to wound-dressing procedure performance. Video recordings of acute surgical wound-dressing procedures performed by nurses in clinical (n = 11) and simulated (n = 3) settings were viewed repeatedly and were iteratively deconstructed by the metric development group. This facilitated the identification of the discrete component steps, potential errors, and sentinel (serious) errors, which characterise a wound dressing procedure and formed part of the observational metric. The ANTT wound-dressing observational metric was stress tested for clarity, the ability to be scored, and interrater reliability, calculated during a further phase of video analysis. The metric was then subjected to a process of cyclical evaluation by a Delphi panel (n = 21) to obtain face and content validity of the metric. The Delphi panel deliberation verified the face and content validity of the metric. The final metric has three phases, 31 individual steps, 18 errors, and 27 sentinel errors. The metric is a tool that identifies the standard to be attained in the performance of acute surgical wound dressings. It can be used as both an adjunct to an educational programme and as a tool to assess a practitioner's performance of a wound-dressing procedure in both simulated and clinical practice contexts.

Current Standards for Training in Robot-assisted Surgery and Endourology: A Systematic Review.

BACKGROUND AND OBJECTIVE: Different training programs have been developed to improve trainee outcomes in urology. However, evidence on the optimal training methodology is sparse. Our aim was to provide a comprehensive description of the training programs available for urological robotic surgery and endourology, assess their validity, and highlight the fundamental elements of future training pathways. METHODS: We systematically reviewed the literature using PubMed/Medline, Embase, and Web of Science databases. The validity of each training model was assessed. The methodological quality of studies on metrics and curricula was graded using the MERSQI scale. The level of evidence (LoE) and level of recommendation for surgical curricula were awarded using the educational Oxford Centre for Evidence-Based Medicine classification. KEY FINDINGS AND LIMITATIONS: A total of 75 studies were identified. Many simulators have been developed to aid trainees in mastering skills required for both robotic and endourology procedures, but only four demonstrated predictive validity. For assessment of trainee proficiency, we identified 18 in robotics training and six in endourology training; however, the majority are Likert-type scales. Although proficiency-based progression (PBP) curricula demonstrated superior outcomes to traditional training in preclinical settings, only four of six (67%) in robotics and three of nine (33%) in endourology are PBP-based. Among these, the Fundamentals of Robotic Surgery and the SIMULATE curricula have the highest LoE (level 1b). The lack of a quantitative synthesis is the main limitation of our study. CONCLUSIONS AND CLINICAL IMPLICATIONS: Training curricula that integrate simulators and PBP methodology have been introduced to standardize trainee outcomes in robotics and endourology. However, evidence regarding their educational impact remains restricted to preclinical studies. Efforts should be made to expand these training programs to different surgical procedures and assess their clinical impact.

Development and evaluation of a societal core robotic surgery accreditation curriculum for the UK.

Standardised proficiency-based progression is the cornerstone of safe robotic skills acquisition, however, is currently lacking within surgical training curricula. Expert consensuses have defined a modular pathway to accredit surgeons. This study aimed to address the lack of a formal, pre-clinical core robotic skills, proficiency-based accreditation curriculum in the UK. Novice robotic participants underwent a four-day pre-clinical core robotic skills curriculum incorporating multimodal assessment. Modifiable-Global Evaluative Assessment of Robotic Skills (M-GEARS), VR-automated performance metrics (APMs) and Objective Clinical Human Reliability Analysis (OCHRA) error methodology assessed performance at the beginning and end of training. Messick's validity concept and a curriculum evaluation model were utilised. Feedback was collated. Proficiency-based progression, benchmarking, tool validity and reliability was assessed through comparative and correlational statistical methods. Forty-seven participants were recruited. Objective assessment of VR and dry models across M-GEARS, APMs and OCHRA demonstrated significant improvements in technical skill (p < 0.001). Concurrent validity between assessment tools demonstrated strong correlation in dry and VR tasks (r = 0.64-0.92, p < 0.001). OCHRA Inter-rater reliability was excellent (r = 0.93, p < 0.001 and 81% matched error events). A benchmark was established with M-GEARS and for the curriculum at 80%. Thirty (63.82%) participants passed. Feedback was 5/5 stars on average, with 100% recommendation. Curriculum evaluation fulfilled all five domains of Messick's validity. Core robotic surgical skills training can be objectively evaluated and benchmarked to provide accreditation in basic robotic skills. A strategy is necessary to enrol standardised curricula into national surgical training at an early stage to ensure patient safety.

Effectiveness of a proficiency-based progression e-learning approach to training in communication in the context of clinically deteriorating patients: a multi-arm randomised controlled trial.

OBJECTIVE: To determine the effectiveness of proficiency-based progression (PBP) e-learning in training in communication concerning clinically deteriorating patients. DESIGN: Single-centre multi-arm randomised double-blind controlled trial with three parallel arms. RANDOMISATION, SETTING AND PARTICIPANTS: A computer-generated program randomised and allocated 120 final year medical students in an Irish University into three trial groups. INTERVENTION: Each group completed the standard Identification, Situation, Background, Assessment, Recommendation communication e-learning; group 1 Heath Service Executive course group (HSE) performed this alone; group 2 (PBP) performed additional e-learning using PBP scenarios with expert-determined proficiency benchmarks composed of weighted marking schemes of steps, errors and critical errors cut-offs; group 3 (S) (self-directed, no PBP) performed additional e-learning with identical scenarios to (PBP) without PBP. MAIN OUTCOME MEASURES: Primary analysis was based on 114 students, comparing ability to reach expert-determined predefined proficiency benchmark in standardised low-fidelity simulation assessment, before and after completion of each group's e-learning requirements. Performance was recorded and scored by two independent blinded assessors. RESULTS: Post-intervention, proficiency in each group in the low-fidelity simulation environment improved with statistically significant difference in proficiency between groups (p<0.001). Proficiency was highest in (PBP) (81.1%, 30/37). Post hoc pairwise comparisons revealed statistically significant differences between (PBP) and self-directed (S) (p<0.001) and (HSE) (p<0.001). No statistically significant difference existed between (S) and (HSE) (p=0.479). Changes in proficiency from pre-intervention to post-intervention were significantly different between the three groups (p=0.001). Post-intervention, an extra 67.6% (25/37) in (PBP) achieved proficiency in the low-fidelity simulation. Post hoc pairwise comparisons revealed statistically significant differences between (PBP) and both (S) (p=0.020) and (HSE) (p<0.001). No statistically significant difference was found between (S) and (HSE) (p=0.156). CONCLUSIONS: PBP e-learning is a more effective way to train in communication concerning clinically deteriorating patients than standard e-learning or e-learning without PBP. TRIAL REGISTRATION NUMBER: NCT02937597.

Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial.

Objective: To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity. Background: GEARS-Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations. Methods: In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves. Results: The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments. Conclusions: Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment.

Virtual classroom proficiency-based progression for robotic surgery training (VROBOT): a randomised, prospective, cross-over, effectiveness study.

Robotic surgery training has lacked evidence-based standardisation. We aimed to determine the effectiveness of adjunctive interactive virtual classroom training (VCT) in concordance with the self-directed Fundamentals of Robotic Surgery (FRS) curriculum. The virtual classroom is comprised of a studio with multiple audio-visual inputs to which participants can connect remotely via the BARCO weConnect platform. Eleven novice surgical trainees were randomly allocated to two training groups (A and B). In week 1, both groups completed a robotic skills induction. In week 2, Group A received training with the FRS curriculum and adjunctive VCT; Group B only received access to the FRS curriculum. In week 3, the groups received the alternate intervention. The primary outcome was measured using the validated robotic-objective structured assessment of technical skills (R-OSAT) at the end of week 2 (time-point 1) and 3 (time-point 2). All participants completed the training curriculum and were included in the final analyses. At time-point 1, Group A achieved a statistically significant greater mean proficiency score compared to Group B (44.80 vs 35.33 points, p = 0.006). At time-point 2, there was no significant difference in mean proficiency score in Group A from time-point 1. In contrast, Group B, who received further adjunctive VCT showed significant improvement in mean proficiency by 9.67 points from time-point 1 (95% CI 5.18-14.15, p = 0.003). VCT is an effective, accessible training adjunct to self-directed robotic skills training. With the steep learning curve in robotic surgery training, VCT offers interactive, expert-led learning and can increase training effectiveness and accessibility.

Does quality assured eLearning provide adequate preparation for robotic surgical skills; a prospective, randomized and multi-center study.

PURPOSE: In particular after the onset of the COVID-19 pandemic, there was a precipitous rush to implement virtual and online learning strategies in surgery and medicine. It is essential to understand whether this approach is sufficient and adequate to allow the development of robotic basic surgical skills. The main aim of the authors was to verify if the quality assured eLearning is sufficient to prepare individuals to perform a basic surgical robotic task. METHODS: A prospective, randomized and multi-center study was conducted in September 2020 in the ORSI Academy, International surgical robotic training center. Forty-seven participants, with no experience but a special interest in robotic surgery, were matched and randomized into four groups who underwent a didactic preparation with different formats before carrying out a robotic suturing and anastomosis task. Didactic preparation methods ranged from a complete eLearning path to peer-reviewed published manuscripts describing the suturing, knot tying and task assessment metrics. RESULTS: The primary outcome was the percentage of trainees who demonstrated the quantitatively defined proficiency benchmark after learning to complete an assisted but unaided robotic vesico-urethral anastomosis task. The quantitatively defined benchmark was based on the objectively assessed performance (i.e., procedure steps completed, errors and critical errors) of experienced robotic surgeons for a proficiency-based progression (PBP) training course. None of the trainees in this study demonstrated the proficiency benchmarks in completing the robotic surgery task. CONCLUSIONS: PBP-based e-learning methodology is an effective training method avoiding critical errors in the suturing and knotting task. Quality assured online learning is insufficient preparation for robotic suturing and knot tying anastomosis skills. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT04541615.

Metric-Based Simulation Training to Proficiency for Endovascular Thrombectomy in Ischemic Stroke.

Ischemic stroke is one of the leading causes of death and long-term disability in the West. Mechanical revascularization techniques are considered the standard of care for large vessel occlusive stroke. Traditional apprenticeship models involve doctors training their skills on patients. Simulation platforms have long been recognized as an alternative to this. There has however been very little robust assessment of the training outcomes achieved on some of these platforms. At best, these simulations increase understanding of the procedural process and may help improve some technical skills; at worst they may instill bad habits and poor technique. The prerequisite of any simulation process must be to teach what to do, with which devices, in the correct sequence as well as what not to do. It should provide valid metric-based feedback to the trainee that is objective, transparent, and fair for formative and summative performance feedback. It should deliver a training program that measures the performance progress of trainees against expert benchmarks-benchmarks that represent an evidence-based peer-reviewed standard. In this paper, we present a perspective for PBP training for thrombectomy based on our experience with the process of procedure characterization, metric validation, and early experience of using this approach for proficiency training. Patient outcomes are not only determined by optimal performance in the Angio Suite but also by an efficient patient procedure pathway. There will be value in utilizing the PBP training standard not only for the procedure itself but also for the constituent elements of the stroke pathway to further improve treatment outcomes for ischemic stroke patients.

Discrimination, reliability, sensitivity, and specificity of metric-based assessment of an unstable pertrochanteric 31A2 intramedullary nailing procedure performed by experienced and novice surgeons.

INTRODUCTION: Identifying objective performance metrics for surgical training in orthopedic surgery is imperative for effective training and patient safety. The objective of this study was to determine if an internationally agreed, metric-based objective assessment of video recordings of an unstable pertrochanteric 31A2 intramedullary nailing procedure distinguished between the performance of experienced and novice orthopedic surgeons. MATERIALS AND METHODS: Previously agreed procedure metrics (i.e., 15 phases of the procedure, 75 steps, 88 errors, and 28 sentinel errors) for a closed reduction and standard cephalomedullary nail fixation with a single cephalic element of an unstable pertrochanteric 31A2 fracture. Experienced surgeons trained to assess the performance metrics with an interrater reliability (IRR) > 0.8 assessed 14 videos from 10 novice surgeons (orthopaedic residents/trainees) and 20 videos from 14 experienced surgeons (orthopaedic surgeons) blinded to group and procedure order. RESULTS: The mean IRR of procedure assessments was 0.97. No statistically significant differences were observed between the two groups for Procedure Steps, Errors, Sentinel Errors, and Total Errors. A small number of Experienced surgeons made a similar number of Total Errors as the weakest performing Novices. When the scores of each group were divided at the median Total Error score, large differences were observed between the Experienced surgeons who made the fewest errors and the Novices making the most errors (p < 0.001). Experienced surgeons who made the most errors made significantly more than their Experienced peers (p < 0.003) and the best performing Novices (p < 0.001). Error metrics assessed with Area Under the Curve demonstrated good to excellent Sensitivity and Specificity (0.807-0.907). DISCUSSION: Binary performance metrics previously agreed by an international Delphi meeting discriminated between the objectively assessed video-recorded performance of Experienced and Novice orthopedic surgeons when group scores were sub-divided at the median for Total Errors. Error metrics discriminated best and also demonstrated good to excellent Sensitivity and Specificity. Some very experienced surgeons performed similar to the Novice group surgeons that made most errors. CONCLUSIONS: The procedure metrics used in this study reliably distinguish Novice and Experienced orthopaedic surgeons' performance and will underpin quality-assured novice training.

Proficiency-based progression training: implementing a novel approach to training for epidural analgesia in labour.

BACKGROUND: Epidural insertion is a challenging anaesthetic procedural skill to learn and may require up to 75 attempts to achieve competency. Proficiency-based progression (PbP) training based on unambiguously defined metrics was associated with a 53% reduction in epidural failure rate. The aim of this observational study was to examine the feasibility of implementation of innovative PbP training for labour epidural insertion performed by novices in a busy tertiary hospital. METHODS: All trainees who were scheduled to commence their obstetric anaesthesia training were invited to participate. Novices undertook intensive PbP training with one-to-one supervision by an anaesthetist trained in PbP. Trainees proceeded to the clinical phase only after attaining the pre-defined proficiency benchmark. All subsequent attempts at labour epidural catheter placement were evaluated. RESULTS: All 12 novice trainees who were scheduled for their initial exposure to obstetric anaesthesia completed PbP training in epidural catheter insertion successfully. The average duration of the training courses was 70 (SD 11) min. Trainee characteristics were broadly similar. They performed a total of 180 labour epidural catheter placements with an overall epidural failure rate of 12.2% (22/180). The proportion of supervisor takeover was 6% (11/179). The incidence of complications was 4% (8/180) and difficulty in epidural catheter insertion due to patient factors was 16% (29/180). Patient satisfaction rates were 80% (satisfied or very satisfied), with 20% unsatisfied with their experience of epidural insertion. CONCLUSION: In our experience, PbP training in epidural placement is feasible within existing departmental resources in a busy tertiary teaching hospital setting.

A validation study of intraoperative performance metrics for training novice cardiac resynchronization therapy implanters.

AIMS: Pacing/cardiac resynchronization therapy (CRT) implant training currently lacks a common system to objectively assess trainee ability to perform required tasks at predetermined performance levels. The purpose of this study was to primarily examine construct validity and reliability, secondarily discriminative validity of novel intraoperative performance metrics, developed for a reference approach to training novice CRT implanters. METHODS: Fifteen novice and eleven experienced CRT implanters performed a 3-lead implant procedure on a virtual reality simulator. Performances were video-recorded, then independently scored using predefined metrics endorsed by an international panel of experts. First, Novice and Experienced group scores were compared for steps completed and errors made. Secondly, each group was split in two around the median score of the group and subgroup scores were compared. RESULTS: The mean number of scored metrics per performance was 108 and the inter-rater reliability for scoring was 0.947. Compared with novices, experienced implanters completed more procedural Steps correctly (mean 87% vs. 73%, p = 0.001), made fewer procedural Errors (6.3 vs. 11.2, p = 0.005), Critical Errors (1.8 vs. 4.4, p = 0.004), and total errors (8.1 vs. 15.6, p = 0.002). Furthermore, the differences between the two Novice subgroups were 25% for steps completed correctly and 94% for total errors made (p < 0.001); the differences between the two Experienced subgroups were respectively 16% and 191% (p < 0.001). CONCLUSIONS: The procedure metrics used in this study reliably distinguish novice and experienced CRT implanters' performances. The metrics further differentiated performance levels within a group with similar experience. These performance metrics will underpin quality-assured novice implanter training.

Metrics Development for Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis With and Without Spondylolisthesis by an International Expert Panel.

STUDY DESIGN: Prospective study. OBJECTIVES: To develop, operationally define, and seek consensus from procedure experts on the metrics that best characterize a reference approach to the performance of a minimally invasive unilateral laminotomy for bilateral decompression (ULBD) for lumbar spinal stenosis. METHODS: A Metrics Group consisting of 3 experienced spine surgeons (2 neurosurgeons, 1 orthopedic surgeon), each with over 25 years of clinical practice, and an educational expert formed the Metrics Group that characterized a lumbar decompression surgery for spinal stenosis as a "reference" procedure. In a modified Delphi panel, 26 spine surgeons from 14 countries critiqued these metrics and their operational definitions before reaching consensus. RESULTS: Performance metrics consisting of 6 phases with 42 steps, 21 errors, and 17 sentinel errors were identified that characterize the procedure. During the peer review, these were evaluated, modified, and agreed. CONCLUSIONS: Surgical procedures can be broken down into elemental tasks necessary for the safe and effective completion of a reference approach to a specified surgical procedure. Spinal experts from 16 countries reached consensus on performance metrics for the procedure. This metric-based characterization can be used in a training curriculum and also for assessment of training and performance in clinical practice.