International consensus panel for transurethral resection of bladder tumours metrics: assessment of face and content validity.

OBJECTIVE: To develop performance metrics that objectively define a reference approach to a transurethral resection of bladder tumours (TURBT) procedure, seek consensus on the performance metrics from a group of international experts. METHODS: The characterisation of a reference approach to a TURBT procedure was performed by identifying phases and explicitly defined procedure events (i.e., steps, errors, and critical errors). An international panel of experienced urologists (i.e., Delphi panel) was then assembled to scrutinise the metrics using a modified Delphi process. Based on the panel's feedback, the proposed metrics could be edited, supplemented, or deleted. A voting process was conducted to establish the consensus level on the metrics. Consensus was defined as the panel majority (i.e., >80%) agreeing that the metric definitions were accurate and acceptable. The number of metric units before and after the Delphi meeting were presented. RESULTS: A core metrics group (i.e., characterisation group) deconstructed the TURBT procedure. The reference case was identified as an elective TURBT on a male patient, diagnosed after full diagnostic evaluation with three or fewer bladder tumours of ≤3 cm. The characterisation group identified six procedure phases, 60 procedure steps, 43 errors, and 40 critical errors. The metrics were presented to the Delphi panel which included 15 experts from six countries. After the Delphi, six procedure phases, 63 procedure steps, 47 errors, and 41 critical errors were identified. The Delphi panel achieved a 100% consensus. CONCLUSION: Performance metrics to characterise a reference approach to TURBT were developed and an international panel of experts reached 100% consensus on them. This consensus supports their face and content validity. The metrics can now be used for a proficiency-based progression training curriculum for TURBT.

Development of objective performance metrics for ultrasound-guided internal jugular vein cannulation on behalf of the College of Anaesthesiologists of Ireland and observation of scores amongst novice and experienced operators.

BACKGROUND: Ultrasound-guided, internal jugular venous (IJV) cannulation is a core technical skill for anaesthesiologists and intensivists. OBJECTIVES: At a modified Delphi panel meeting, to define and reach consensus on a set of objective ultrasound-guided IJV cannulation performance metrics on behalf of the College of Anaesthesiologists of Ireland (CAI). To use these metrics to objectively score video recordings of novice and experienced anaesthesiologists. DESIGN: An observational study. SETTING: CAI, March to June 2016 and four CAI training hospitals, November 2016 to July 2019. PARTICIPANTS: Metric development group: two CAI national directors of postgraduate training (specialist anaesthesiolgists), a behavioural scientist, a specialist intensivist and a senior CAI trainee. Scoring by two blinded assessors of video recordings of novice ( n  = 11) and experienced anaesthesiologists ( n  = 15) ultrasound-guided IJV cannulations. MAIN OUTCOME MEASURES: A set of agreed CAI objective performance metrics, that is, steps, errors, and critical errors characterising ultrasound-guided IJV cannulation. The difference in performance scores between novice and experienced anaesthesiologists as determined by skill level defined as being below or above the median total error score (errors plus critical errors): that is, low error (LoErr) and high error (HiErr), respectively. RESULTS: The study identified 47 steps, 18 errors and 13 critical errors through six phases.Variability was observed in the range of total error scores for both novice (1 to 3) and experienced (0 to 4.5) anaesthesiologists. This resulted in two further statistically different subgroups (LoErr and HiErr) for both novice ( P  = 0.011) and experienced practitioners ( P  < 0.000). The LoErr-experienced group performed the best in relation to steps, errors and total errors. Critical errors were only observed in the experienced group. CONCLUSION: A set of valid, reliable objective performance metrics has been developed for ultrasound-guided IJV cannulation. Considerable skill variability underlines the need to develop a CAI simulation-training programme using these metrics.

Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial.

OBJECTIVE: To determine whether proficiency-based progression (PBP) training leads to better robotic surgical performance compared to traditional training (TT), given that the value of PBP training for learning robotic surgical skills is unclear. MATERIALS AND METHODS: The PROVESA trial is a multicentric, prospective, randomized and blinded clinical study comparing PBP training with TT for robotic suturing and knot-tying anastomosis skills. A total of 36 robotic surgery-naïve junior residents were recruited from 16 training sites and 12 residency training programmes. Participants were randomly allocated to metric-based PBP training or the current standard of care TT, and compared at the end of training. The primary outcome was percentage of participants reaching the predefined proficiency benchmark. Secondary outcomes were the numbers of procedure steps and errors made. RESULTS: Of the group that received TT, 3/18 reached the proficiency benchmark versus 12/18 of the PBP group (i.e. the PBP group were ~10 times as likely to demonstrate proficiency [P = 0.006]). The PBP group demonstrated a 51% reduction in number of performance errors from baseline to the final assessment (18.3 vs 8.9). The TT group demonstrated a marginal improvement (15.94 vs 15.44) in errors made. CONCLUSIONS: The PROVESA trial is the first prospective randomized controlled trial on basic skills training in robotic surgery. Implementation of a PBP training methodology resulted in superior surgical performance for robotic suturing and knot-tying anastomosis performance. Compared to TT, better surgical quality could be obtained by implementing PBP training for basic skills in robotic surgery.

Development and Validation of the Metric-Based Assessment of a Robotic Dissection Task on an Avian Model.

INTRODUCTION: The introduction of robot-assisted surgical devices requires the application of objective performance metrics to verify performance levels. OBJECTIVE: To develop and validate (face, content, response process, and construct) the performance metrics for a robotic dissection task using a chicken model. METHODS: In a procedure characterization, we developed the performance metrics (i.e., procedure steps, errors, and critical errors) for a robotic dissection task, using a chicken model. In a modified Delphi panel, 14 experts from four European Union countries agreed on the steps, errors, and critical errors (CEs) of the task. Six experienced surgeons and eight novice urology surgeons performed the robotic dissection task twice on the chicken model. In the Delphi meeting, 100% consensus was reached on five procedure steps, 15 errors and two CEs. Novice surgeons took 20 min to complete the task on trial 1 and 14 min during trial two, whereas experts took 8.2 min and 6.5 min. On average, the Expert Group completed the task 56% faster than the Novice Group and made 46% fewer performance errors. RESULTS: Sensitivity and specificity for procedure errors and time were excellent to good (i.e., 1.0-0.91) but poor (i.e., 0.5) for step metrics. The mean interrater reliability for the assessments by two robotic surgeons was 0.91 (Expert Group inter-rater reliability = 0.92 and Novice Group = 0.9). CONCLUSIONS: We report evidence which supports the demonstration of face, content, and construct validity for a standard and replicable basic robotic dissection task on the chicken model.

A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes.

OBJECTIVE: To analyze all published prospective, randomized, and blinded clinical studies on the proficiency-based progression (PBP) training using objective performance metrics. BACKGROUND: The benefit of PBP methodology to learning clinical skills in comparison to conventional training is not settled. METHODS: Search of PubMed, Cochrane library's Central, EMBASE, MEDLINE, and Scopus databases, from inception to 1st March 2020. Two independent reviewers extracted the data. The Medical Education Research Study Quality Instrument (MERSQI) was used to assess the methodological quality of included studies. Results were pooled using biased corrected standardized mean difference and ratio-of-means. Summary effects were evaluated using a series of fixed and random effects models. The primary outcome was the number of procedural errors performed comparing PBP and non-PBP-based training pathways. Secondary outcomes were the number of procedural steps completed and the time to complete the task/procedure. RESULTS: From the initial pool of 468 studies, 12 randomized clinical studies with a total of 239 participants were included in the analysis. In comparison to the non-PBP training, ratio-of-means results showed that PBP training reduced the number of performance errors by 60% (P < 0.001) and procedural time by 15% (P = 0.003) and increased the number of steps performed by 47% (P < 0.001). CONCLUSIONS AND RELEVANCE: Our systematic review and meta-analysis confirms that PBP training in comparison to conventional or quality assured training improved trainees' performances, by decreasing procedural errors and procedural time, while increasing the number of correct steps taken when compared to standard simulation-based training.

Objective assessment of intraoperative skills for robot-assisted radical prostatectomy (RARP): results from the ERUS Scientific and Educational Working Groups Metrics Initiative.

OBJECTIVE: To develop and seek consensus from procedure experts on the metrics that best characterise a reference robot-assisted radical prostatectomy (RARP) and determine if the metrics distinguished between the objectively assessed RARP performance of experienced and novice urologists, as identifying objective performance metrics for surgical training in robotic surgery is imperative for patient safety. MATERIALS AND METHODS: In Study 1, the metrics, i.e. 12 phases of the procedure, 81 steps, 245 errors and 110 critical errors for a reference RARP were developed and then presented to an international Delphi panel of 19 experienced urologists. In Study 2, 12 very experienced surgeons (VES) who had performed >500 RARPs and 12 novice urology surgeons performed a RARP, which was video recorded and assessed by two experienced urologists blinded as to subject and group. Percentage agreement between experienced urologists for the Delphi meeting and Mann-Whitney U- and Kruskal-Wallis tests were used for construct validation of the newly identified RARP metrics. RESULTS: At the Delphi panel, consensus was reached on the appropriateness of the metrics for a reference RARP. In Study 2, the results showed that the VES performed ~4% more procedure steps and made 72% fewer procedure errors than the novices (P = 0.027). Phases VIIa and VIIb (i.e. neurovascular bundle dissection) best discriminated between the VES and novices. LIMITATIONS: VES whose performance was in the bottom half of their group demonstrated considerable error variability and made five-times as many errors as the other half of the group (P = 0.006). CONCLUSIONS: The international Delphi panel reached high-level consensus on the RARP metrics that reliably distinguished between the objectively scored procedure performance of VES and novices. Reliable and valid performance metrics of RARP are imperative for effective and quality assured surgical training.

Development and validation of the objective assessment of robotic suturing and knot tying skills for chicken anastomotic model.

BACKGROUND: To improve patient safety, there is an imperative to develop objective performance metrics for basic surgical skills training in robotic surgery. OBJECTIVE: To develop and validate (face, content, and construct) the performance metrics for robotic suturing and knot tying, using a chicken anastomotic model. DESIGN, SETTING AND PARTICIPANTS: Study 1: In a procedure characterization, we developed the performance metrics (i.e., procedure steps, errors, and critical errors) for robotic suturing and knot tying, using a chicken anastomotic model. In a modified Delphi panel of 13 experts from four EU countries, we achieved 100% consensus on the five steps, 18 errors and four critical errors (CE) of the task. Study 2: Ten experienced surgeons and nine novice urology surgeons performed the robotic suturing and knot tying chicken anastomotic task. The mean inter-rater reliability for the assessments by two experienced robotic surgeons was 0.92 (95% CI, 0.9-0.95). Novices took 18.5 min to complete the task and experts took 8.2 min. (p = 0.00001) and made 74% more objectively assessed performance errors than the experts (p = 0.000343). CONCLUSIONS: We demonstrated face, content, and construct validity for a standard and replicable basic anastomotic robotic suturing and knot tying task on a chicken model. Validated, objective, and transparent performance metrics of a robotic surgical suturing and knot tying tasks are imperative for effective and quality assured surgical training.

Validation of phlebotomy performance metrics developed as part of a proficiency-based progression initiative to mitigate wrong blood in tube.

AIMS: The purpose of this study was to (1) characterise the procedure of phlebotomy, deconstruct it into its constituent parts and develop a performance metric for the purpose of training healthcare professionals in a large teaching hospital and to (2) evaluate the construct validity of the phlebotomy metric and establish a proficiency benchmark. METHOD: By engaging with a multidisciplinary team with a wide range of experience of preanalytical errors in phlebotomy and observing video recordings of the procedure performed in the actual working environment, we defined a performance metric. This was brought to a modified Delphi meeting, where consensus was reached by an expert panel. To demonstrate construct validity, we used the metric to objectively assess the performance of novices and expert practitioners. RESULTS: A phlebotomy metric consisting of 11 phases and 77 steps was developed. The mean inter-rater reliability was 0.91 (min 0.83, max 0.95). The expert group completed more steps of the procedure (72 vs 69), made fewer errors (19 vs 13, p=0.014) and fewer critical errors (1 Vs 4, p=0.002) than the novice group. CONCLUSIONS: The metrics demonstrated construct validity and the proficiency benchmark was established with a minimum observation of 69 steps, with no critical errors and no more than 13 errors in total.

The effect of augmented reality on the accuracy and learning curve of external ventricular drain placement.

OBJECTIVE: The traditional freehand technique for external ventricular drain (EVD) placement is most frequently used, but remains the primary risk factor for inaccurate drain placement. As this procedure could benefit from image guidance, the authors set forth to demonstrate the impact of augmented-reality (AR) assistance on the accuracy and learning curve of EVD placement compared with the freehand technique. METHODS: Sixteen medical students performed a total of 128 EVD placements on a custom-made phantom head, both before and after receiving a standardized training session. They were guided by either the freehand technique or by AR, which provided an anatomical overlay and tailored guidance for EVD placement through inside-out infrared tracking. The outcome was quantified by the metric accuracy of EVD placement as well as by its clinical quality. RESULTS: The mean target error was significantly impacted by either AR (p = 0.003) or training (p = 0.02) in a direct comparison with the untrained freehand performance. Both untrained (11.9 ± 4.5 mm) and trained (12.2 ± 4.7 mm) AR performances were significantly better than the untrained freehand performance (19.9 ± 4.2 mm), which improved after training (13.5 ± 4.7 mm). The quality of EVD placement as assessed by the modified Kakarla scale (mKS) was significantly impacted by AR guidance (p = 0.005) but not by training (p = 0.07). Both untrained and trained AR performances (59.4% mKS grade 1 for both) were significantly better than the untrained freehand performance (25.0% mKS grade 1). Spatial aptitude testing revealed a correlation between perceptual ability and untrained AR-guided performance (r = 0.63). CONCLUSIONS: Compared with the freehand technique, AR guidance for EVD placement yielded a higher outcome accuracy and quality for procedure novices. With AR, untrained individuals performed as well as trained individuals, which indicates that AR guidance not only improved performance but also positively impacted the learning curve. Future efforts will focus on the translation and evaluation of AR for EVD placement in the clinical setting.

A Proficiency-Based Progression Simulation Training Curriculum to Acquire the Skills Needed in Performing Arthroscopic Bankart and Rotator Cuff Repairs-Implementation and Impact.

PURPOSE: To investigate the impact of a proficiency-based progression (PBP) curriculum employed to teach trainees in the skills needed to demonstrate proficiency for an arthroscopic Bankart repair (ABR) and an arthroscopic rotator cuff repair (ARCR) by objectively comparing pre- and immediate postcourse performances. METHODS: In a prospective study, 16 arthroscopy/sports medicine fellows and 2 senior residents (complete group: N = 18) were randomly assigned to perform a precourse cadaveric ABR (Bankart subgroup: N = 6), ARCR (cuff subgroup: N = 6), or basic skills on a shoulder simulator (N = 6). After completing a PBP training curriculum, all 18 registrants performed both an ABR and ARCR scored in real time by trained raters using previously validated metrics. RESULTS: The Bankart subgroup made 58% fewer objectively assessed errors at the completion of the course than at baseline (P = .004, confidence interval -1.449 to -0.281), and performance variability was substantially reduced (standard deviation = 5.89 vs 2.81). The cuff subgroup also made 58% fewer errors (P = .001, confidence interval -1.376 to 0.382) and showed a similar reduction in performance variability (standard deviation = 5.42 vs 2.1). Only one subject's precourse baseline performance met the proficiency benchmark compared with 89% and 83% of the all registrants on the final ABR and ARCR cadaveric assessments, respectively. CONCLUSIONS: The results of this study reject the null hypothesis. They demonstrate that the implementation of a PBP simulation curriculum to train the skills necessary to perform arthroscopic Bankart and rotator cuff repairs results in a large and statistically significant improvement in the trainee's ability to meet the 2 related performance benchmarks. Proficiency was demonstrated by 89% and 83% of the trainees for an ABR and an ARCR, respectively, in a two- and one-half day course. CLINICAL RELEVANCE: Surgical training employing a PBP curriculum is efficient, effective, and has the potential to improve patient safety.

Live Observational Objective Assessment of Operative Performance in a Cadaveric Model is Equivalent to Delayed Video-Based Assessment.

PURPOSE: The purpose of our study was to compare real-time, live observational scoring with delayed retrospective video review of operative performance and to determine whether the evaluation method affected the attainment of proficiency benchmarks. METHODS: Sixteen arthroscopy/sports medicine fellows and 2 senior residents completed training to perform arthroscopic Bankart repairs (ABRs) and arthroscopic rotator cuff repairs (ARCRs) using a proficiency-based progression curriculum. Each final operative performance for 15 randomly selected ABRs and 13 ARCRs performed on cadavers were scored live (observation during the operative performance) and on delayed video review (6-8 weeks) by 1 of 15 trained raters using validated metric-based (step and error) assessment tools. The inter-rater reliability (IRR) of live versus video review by a single rater was calculated, and changes to the trainee's attainment of the proficiency benchmarks were noted. The correlation coefficient (r) and the R2 were also calculated for the paired scores from the randomly selected performances. RESULTS: No significant differences in the observed IRR agreement or the attainment of the proficiency benchmarks were found when comparing live to video assessment for either ABR or ARCR. The correlation coefficients r and R2 were considerably lower than the agreement coefficient (IRR) for rotator cuff steps (e.g., R2 = 0.74 vs. IRR = 0.97, P = 0.001); Bankart errors (R2 = 0.73 vs. IRR = 0.98, P = 0.006); and rotator cuff errors (R2 = 0.48 vs. IRR = 0.98, P = 0.0002). CONCLUSIONS: Real-time live and delayed video-based scoring of operative performance are essentially equivalent for the metric-based assessments of operative performance in ABRs and ARCRs. When the IRR agreement coefficient was compared with the correlation coefficients, the former was found to have greater homogeneity and measurement precision. CLINICAL RELEVANCE: Metric-based live scoring is reliable and accurate for operative performance assessment, including high-stakes evaluations.

Proving the Effectiveness of the Fundamentals of Robotic Surgery (FRS) Skills Curriculum: A Single-blinded, Multispecialty, Multi-institutional Randomized Control Trial.

OBJECTIVE: To demonstrate the noninferiority of the fundamentals of robotic surgery (FRS) skills curriculum over current training paradigms and identify an ideal training platform. SUMMARY BACKGROUND DATA: There is currently no validated, uniformly accepted curriculum for training in robotic surgery skills. METHODS: Single-blinded parallel-group randomized trial at 12 international American College of Surgeons (ACS) Accredited Education Institutes (AEI). Thirty-three robotic surgery experts and 123 inexperienced surgical trainees were enrolled between April 2015 and November 2016. Benchmarks (proficiency levels) on the 7 FRS Dome tasks were established based on expert performance. Participants were then randomly assigned to 4 training groups: Dome (n = 29), dV-Trainer (n = 30), and DVSS (n = 32) that trained to benchmarks and control (n = 32) that trained using locally available robotic skills curricula. The primary outcome was participant performance after training based on task errors and duration on 5 basic robotic tasks (knot tying, continuous suturing, cutting, dissection, and vessel coagulation) using an avian tissue model (transfer-test). Secondary outcomes included cognitive test scores, GEARS ratings, and robot familiarity checklist scores. RESULTS: All groups demonstrated significant performance improvement after skills training (P < 0.01). Participating residents and fellows performed tasks faster (DOME and DVSS groups) and with fewer errors than controls (DOME group; P < 0.01). Inter-rater reliability was high for the checklist scores (0.82-0.97) but moderate for GEARS ratings (0.40-0.67). CONCLUSIONS: We provide evidence of effectiveness for the FRS curriculum by demonstrating better performance of those trained following FRS compared with controls on a transfer test. We therefore argue for its implementation across training programs before surgeons apply these skills clinically.

Arthroscopic Rotator Cuff Repair Metrics: Establishing Face, Content, and Construct Validity in a Cadaveric Model.

PURPOSE: To create and determine face validity and content validity of arthroscopic rotator cuff repair (ARCR) performance metrics, to confirm construct validity of the metrics coupled with a cadaveric shoulder, and to establish a performance benchmark for the procedure on a cadaveric shoulder. METHODS: Five experienced arthroscopic shoulder surgeons created step, error, and sentinel error metrics for an ARCR. Fourteen shoulder arthroscopy faculty members from the Arthroscopy Association of North America formed the modified Delphi panel to assess face and content validity. Eight Arthroscopy Association of North America shoulder arthroscopy faculty members (experienced group) were compared with 9 postgraduate year 4 or 5 orthopaedic residents (novice group) in their ability to perform an ARCR. Instructions were given to perform a diagnostic arthroscopy and a 2-anchor, 4-simple suture repair of a 2-cm supraspinatus tear. The procedure was videotaped in its entirety and independently scored in blinded fashion by trained, paired reviewers. RESULTS: Delphi panel consensus for 42 steps and 66 potential errors was obtained. Overall performance assessment showed a mean inter-rater reliability of 0.93. Novice surgeons completed 17% fewer steps (32.1 vs 37.5, P = .001) and enacted 2.5 times more errors than the experienced group (6.21 vs 2.5, P = .012). Fifty percent of the experienced group members and none of the novice group members achieved the proficiency benchmark of a minimum of 37 steps completed with 3 or fewer errors. CONCLUSIONS: Face validity and content validity for the ARCR metrics, along with construct validity for the metrics and cadaveric shoulder, were verified. A proficiency benchmark was established based on the mean performance of an experienced group of arthroscopic shoulder surgeons. CLINICAL RELEVANCE: Validated procedural metrics combined with the use of a cadaveric shoulder can be used to accurately assess the performance of an ARCR.

Acute surgical wound-dressing procedure: Description of the steps involved in the development and validation of an observational metric.

The aim of this study was to develop an observational metric that could be used to assess the performance of a practitioner in completing an acute surgical wound-dressing procedure using aseptic non-touch technique (ANTT). A team of clinicians, academics, and researchers came together to develop an observational metric using an iterative six-stage process, culminating in a Delphi panel meeting. A scoping review of the literature provided a background empirical perspective relating to wound-dressing procedure performance. Video recordings of acute surgical wound-dressing procedures performed by nurses in clinical (n = 11) and simulated (n = 3) settings were viewed repeatedly and were iteratively deconstructed by the metric development group. This facilitated the identification of the discrete component steps, potential errors, and sentinel (serious) errors, which characterise a wound dressing procedure and formed part of the observational metric. The ANTT wound-dressing observational metric was stress tested for clarity, the ability to be scored, and interrater reliability, calculated during a further phase of video analysis. The metric was then subjected to a process of cyclical evaluation by a Delphi panel (n = 21) to obtain face and content validity of the metric. The Delphi panel deliberation verified the face and content validity of the metric. The final metric has three phases, 31 individual steps, 18 errors, and 27 sentinel errors. The metric is a tool that identifies the standard to be attained in the performance of acute surgical wound dressings. It can be used as both an adjunct to an educational programme and as a tool to assess a practitioner's performance of a wound-dressing procedure in both simulated and clinical practice contexts.

Simulation Research in Gastrointestinal and Urologic Care-Challenges and Opportunities: Summary of a National Institute of Diabetes and Digestive and Kidney Diseases and National Institute of Biomedical Imaging and Bioengineering Workshop.

: A workshop on "Simulation Research in Gastrointestinal and Urologic Care: Challenges and Opportunities" was held at the National Institutes of Health in June 2016. The purpose of the workshop was to examine the extent to which simulation approaches have been used by skilled proceduralists (not trainees) caring for patients with gastrointestinal and urologic diseases. The current status of research findings in the use and effectiveness of simulation applications was reviewed, and numerous knowledge gaps and research needs were identified by the faculty and the attendees. The paradigm of "deliberate practice," rather than mere repetition, and the value of coaching by experts was stressed by those who have adopted simulation in music and sports. Models that are most useful for the adoption of simulation by expert clinicians have yet to be fully validated. Initial studies on the impact of simulation on safety and error reduction have demonstrated its value in the training domain, but the role of simulation as a strategy for increased procedural safety remains uncertain in the world of the expert practitioner. Although the basic requirements for experienced physicians to acquire new skills have been explored, the widespread availability of such resources is an unrealized goal, and there is a need for well-designed outcome studies to establish the role of simulation in improving the quality of health care.

Visual spatial ability for surgical trainees: implications for learning endoscopic, laparoscopic surgery and other image-guided procedures.

BACKGROUND: In image-guided procedures, a high level of visual spatial ability may be an advantage for surgical trainees. We assessed the visual spatial ability of surgical trainees. METHODS: Two hundred and thirty-nine surgical trainees and 61 controls were tested on visual spatial ability using 3 standardised tests, the Card Rotation, Cube Comparison and Map-Planning Tests. RESULTS: Two hundred and twenty-one, 236 and 236 surgical trainees and 61 controls completed the Card Rotation test, Cube Comparison test and Map-Planning test, respectively. Two percent of surgical trainees performed statistically significantly worse than their peers on card rotation and map-planning test, > 1% on Cube Comparison test. Surgical trainees performed statistically significantly better than controls on all tests. CONCLUSIONS: Two percent of surgical trainees performed statistically significantly worse than their peers on visual spatial ability. The implication of this finding is unclear, further research is required that can look at the learning and educational portfolios of these trainees who perform poorly on visual spatial ability, and ascertain if they are struggling to learn skills for image-guided procedures.

Inter-rater Reliability for Metrics Scored in a Binary Fashion-Performance Assessment for an Arthroscopic Bankart Repair.

PURPOSE: To determine the inter-rater reliability (IRR) of a procedure-specific checklist scored in a binary fashion for the evaluation of surgical skill and whether it meets a minimum level of agreement (≥0.8 between 2 raters) required for high-stakes assessment. METHODS: In a prospective randomized and blinded fashion, and after detailed assessment training, 10 Arthroscopy Association of North America Master/Associate Master faculty arthroscopic surgeons (in 5 pairs) with an average of 21 years of surgical experience assessed the video-recorded 3-anchor arthroscopic Bankart repair performance of 44 postgraduate year 4 or 5 residents from 21 Accreditation Council for Graduate Medical Education orthopaedic residency training programs from across the United States. RESULTS: No paired scores of resident surgeon performance evaluated by the 5 teams of faculty assessors dropped below the 0.8 IRR level (mean = 0.93; range 0.84-0.99; standard deviation = 0.035). A comparison between the 5 assessor groups with 1 factor analysis of variance showed that there was no significant difference between the groups (P = .205). Pearson's product-moment correlation coefficient revealed a strong and statistically significant negative correlation, that is, -0.856 (P < .000), indicating that as intra-operative error rate scores increased, the IRR decreased. CONCLUSIONS: Arthroscopy Association of North America shoulder faculty raters from across the United States showed high levels of IRR in the assessment of an arthroscopic 3-anchor Bankart repair procedure. All paired assessments were above the 0.8 level and the mean IRR of all resident assessments was 0.93, indicating that they could be used for high-stakes decisions. CLINICAL RELEVANCE: With the move toward outcomes-based performance evaluation for graduate medical education, high-stakes assessments of surgical skill will require robust, reliable measurement tools that are able to withstand challenge. Surgical checklists employing metrics scored in a binary fashion meet the need and can show a high (>80%) IRR.