Live Observational Objective Assessment of Operative Performance in a Cadaveric Model is Equivalent to Delayed Video-Based Assessment.

PURPOSE: The purpose of our study was to compare real-time, live observational scoring with delayed retrospective video review of operative performance and to determine whether the evaluation method affected the attainment of proficiency benchmarks. METHODS: Sixteen arthroscopy/sports medicine fellows and 2 senior residents completed training to perform arthroscopic Bankart repairs (ABRs) and arthroscopic rotator cuff repairs (ARCRs) using a proficiency-based progression curriculum. Each final operative performance for 15 randomly selected ABRs and 13 ARCRs performed on cadavers were scored live (observation during the operative performance) and on delayed video review (6-8 weeks) by 1 of 15 trained raters using validated metric-based (step and error) assessment tools. The inter-rater reliability (IRR) of live versus video review by a single rater was calculated, and changes to the trainee's attainment of the proficiency benchmarks were noted. The correlation coefficient (r) and the R2 were also calculated for the paired scores from the randomly selected performances. RESULTS: No significant differences in the observed IRR agreement or the attainment of the proficiency benchmarks were found when comparing live to video assessment for either ABR or ARCR. The correlation coefficients r and R2 were considerably lower than the agreement coefficient (IRR) for rotator cuff steps (e.g., R2 = 0.74 vs. IRR = 0.97, P = 0.001); Bankart errors (R2 = 0.73 vs. IRR = 0.98, P = 0.006); and rotator cuff errors (R2 = 0.48 vs. IRR = 0.98, P = 0.0002). CONCLUSIONS: Real-time live and delayed video-based scoring of operative performance are essentially equivalent for the metric-based assessments of operative performance in ABRs and ARCRs. When the IRR agreement coefficient was compared with the correlation coefficients, the former was found to have greater homogeneity and measurement precision. CLINICAL RELEVANCE: Metric-based live scoring is reliable and accurate for operative performance assessment, including high-stakes evaluations.

A Proficiency-Based Progression Simulation Training Curriculum to Acquire the Skills Needed in Performing Arthroscopic Bankart and Rotator Cuff Repairs-Implementation and Impact.

PURPOSE: To investigate the impact of a proficiency-based progression (PBP) curriculum employed to teach trainees in the skills needed to demonstrate proficiency for an arthroscopic Bankart repair (ABR) and an arthroscopic rotator cuff repair (ARCR) by objectively comparing pre- and immediate postcourse performances. METHODS: In a prospective study, 16 arthroscopy/sports medicine fellows and 2 senior residents (complete group: N = 18) were randomly assigned to perform a precourse cadaveric ABR (Bankart subgroup: N = 6), ARCR (cuff subgroup: N = 6), or basic skills on a shoulder simulator (N = 6). After completing a PBP training curriculum, all 18 registrants performed both an ABR and ARCR scored in real time by trained raters using previously validated metrics. RESULTS: The Bankart subgroup made 58% fewer objectively assessed errors at the completion of the course than at baseline (P = .004, confidence interval -1.449 to -0.281), and performance variability was substantially reduced (standard deviation = 5.89 vs 2.81). The cuff subgroup also made 58% fewer errors (P = .001, confidence interval -1.376 to 0.382) and showed a similar reduction in performance variability (standard deviation = 5.42 vs 2.1). Only one subject's precourse baseline performance met the proficiency benchmark compared with 89% and 83% of the all registrants on the final ABR and ARCR cadaveric assessments, respectively. CONCLUSIONS: The results of this study reject the null hypothesis. They demonstrate that the implementation of a PBP simulation curriculum to train the skills necessary to perform arthroscopic Bankart and rotator cuff repairs results in a large and statistically significant improvement in the trainee's ability to meet the 2 related performance benchmarks. Proficiency was demonstrated by 89% and 83% of the trainees for an ABR and an ARCR, respectively, in a two- and one-half day course. CLINICAL RELEVANCE: Surgical training employing a PBP curriculum is efficient, effective, and has the potential to improve patient safety.

Arthroscopic Rotator Cuff Repair Metrics: Establishing Face, Content, and Construct Validity in a Cadaveric Model.

PURPOSE: To create and determine face validity and content validity of arthroscopic rotator cuff repair (ARCR) performance metrics, to confirm construct validity of the metrics coupled with a cadaveric shoulder, and to establish a performance benchmark for the procedure on a cadaveric shoulder. METHODS: Five experienced arthroscopic shoulder surgeons created step, error, and sentinel error metrics for an ARCR. Fourteen shoulder arthroscopy faculty members from the Arthroscopy Association of North America formed the modified Delphi panel to assess face and content validity. Eight Arthroscopy Association of North America shoulder arthroscopy faculty members (experienced group) were compared with 9 postgraduate year 4 or 5 orthopaedic residents (novice group) in their ability to perform an ARCR. Instructions were given to perform a diagnostic arthroscopy and a 2-anchor, 4-simple suture repair of a 2-cm supraspinatus tear. The procedure was videotaped in its entirety and independently scored in blinded fashion by trained, paired reviewers. RESULTS: Delphi panel consensus for 42 steps and 66 potential errors was obtained. Overall performance assessment showed a mean inter-rater reliability of 0.93. Novice surgeons completed 17% fewer steps (32.1 vs 37.5, P = .001) and enacted 2.5 times more errors than the experienced group (6.21 vs 2.5, P = .012). Fifty percent of the experienced group members and none of the novice group members achieved the proficiency benchmark of a minimum of 37 steps completed with 3 or fewer errors. CONCLUSIONS: Face validity and content validity for the ARCR metrics, along with construct validity for the metrics and cadaveric shoulder, were verified. A proficiency benchmark was established based on the mean performance of an experienced group of arthroscopic shoulder surgeons. CLINICAL RELEVANCE: Validated procedural metrics combined with the use of a cadaveric shoulder can be used to accurately assess the performance of an ARCR.