A Proposed Blueprint for Operative Performance Training, Assessment, and Certification.

OBJECTIVE: The aim of this study was to propose an evidence-based blueprint for training, assessment, and certification of operative performance for surgical trainees. SUMMARY BACKGROUND DATA: Operative skill is a critical aspect of surgical performance. High-quality assessment of operative skill therefore has profound implications for training, accreditation, certification, and the public trust of the profession. Current methods of operative skill assessment for surgeons rely heavily on global assessment strategies across a very broad domain of procedures. There is no mechanism to assure technical competence for individual procedures. The science and scalability of operative skill assessment has progressed significantly in recent decades, and can inform a much more meaningful strategy for competency-based assessment of operative skill than has been previously achieved. METHODS: The present article reviews the current status and science of operative skill assessment and proposes a template for competency-based assessment which could be used to update training, accreditation, and certification processes. The proposal is made in reference to general surgery but is more generally applicable to other procedural specialties. RESULTS: Streamlined, routine assessment of every procedure performed by surgical trainees is feasible and would enable a more competency-based educational paradigm. In light of the constraints imposed by both clinical volume and assessment bias, trainees should be expected to become proficient and be measured against a mastery learning standard only for the most important and highest-frequency procedures. For less frequently observed procedures, performance can be compared to a norm-referenced standard and, to provide an overall trajectory of performance, analyzed in aggregate. Key factors in implementing this approach are the number of evaluations, the number of raters, the timeliness of evaluation, and evaluation items. CONCLUSIONS: A competency-based operative skill assessment can be incorporated into surgical training, assessment, and certification. The time has come to develop a systematic approach to this issue as a means of demonstrating professional standards worthy of the public trust.

How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?

OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

Sex Differences in Faculty Rank Among Academic Surgeons in the United States in 2014.

OBJECTIVE: The aim of this study was to evaluate sex differences in full professorship among a comprehensive, contemporary cohort of US academic surgeons. SUMMARY OF BACKGROUND DATA: Previous work demonstrates that women are less likely than men to be full professors in academic medicine, and in certain surgical subspecialties. Whether sex differences in academic rank exist across all surgical fields, and after adjustment for confounders, is not known. METHODS: A comprehensive list of surgeons with faculty appointments at US medical schools in 2014 was obtained from Association of American Medical Colleges (AAMC) faculty roster and linked to a comprehensive physician database from Doximity, an online physician networking website, which contained the following data for all physicians: sex, age, years since residency, publication number (total and first/last author), clinical trials participation, National Institutes of Health grants, and surgical subspecialty. A 20% sample of 2013 Medicare payments for care was added to this dataset. Multivariable regression models were used to estimate sex differences in full professorship, adjusting for these variables and medical school-specific fixed effects. RESULTS: Among 11,549 surgeon faculty at US medical schools in 2014, 1692 (14.7%) were women. Women comprised 19.4% of assistant professors (1072/5538), 13.8% of associate professors (404/2931), and 7.0% of full professors (216/3080). After multivariable analysis, women were less likely to be full professors than men (adjusted odds ratio: 0.76, 95% confidence interval: 0.6-0.9). CONCLUSION: Among surgical faculty at US medical schools in 2014, women were less likely than men to be full professors after adjustment for multiple factors known to impact faculty rank.

Intraoperative Adverse Events in Abdominal Surgery: What Happens in the Operating Room Does Not Stay in the Operating Room.

OBJECTIVE: We sought to assess the impact of intraoperative adverse events (iAEs) on 30-day postoperative mortality, 30-day postoperative morbidity, and postoperative length of stay (LOS) among patients undergoing abdominal surgery. We hypothesized that iAEs would be associated with significant increases in each outcome. SUMMARY OF BACKGROUND DATA: The relationship between iAEs and postoperative clinical outcomes remains largely unknown. METHODS: The 2007 to 2012 institutional ACS-NSQIP and administrative databases for abdominal surgeries were matched then screened for iAEs using the Agency for Healthcare Research and Quality's 15 Patient Safety Indicator, "Accidental Puncture/Laceration". Each chart flagged during the initial screen was then manually reviewed to confirm whether an iAE occurred. Univariate then multivariable logistic regression models were constructed to assess the independent impact of iAEs on 30-day mortality, 30-day morbidity, and prolonged (≥7 days) postoperative LOS, controlling for preoperative/intraoperative variables (eg, age, comorbidities, ASA, wound classification), procedure type (eg, laparoscopic vs open, intestinal, foregut, hepatopancreaticobiliary vs abdominal wall procedure), and complexity (eg, adhesions; relative value units). Propensity score analyses were conducted with each iAE patient matched with 5 non-iAE patients. Sensitivity analyses were performed. RESULTS: A total of 9288 cases were included; 183 had iAEs. Most iAEs consisted of bowel (44%) or vessel (29%) injuries and were addressed intraoperatively (92%). In multivariable analyses, iAEs were independently associated with increased 30-day mortality [OR = 3.19, 95% confidence interval (CI) 1.52-6.71, P = 0.002], 30-day morbidity (OR = 2.68, 95% CI 1.89-3.81, P < 0.001), and prolonged postoperative LOS (OR = 1.85, 95% CI 1.27-2.70, P = 0.001). Postoperative complications associated with iAEs included deep/organ-space surgical site infection (OR = 1.94, 95% CI 1.20-3.14), P = 0.007), sepsis (OR = 2.14, 95% CI 1.32-3.47, P = 0.002), pneumonia (OR = 2.18, 95% CI 1.11-4.26, P = 0.023), and failure to wean ventilator (OR = 3.88, 95% CI 2.17-6.95, P < 0.001). Propensity score matching confirmed these findings, as did multiple sensitivity analyses. CONCLUSIONS: iAEs are independently associated with substantial increases in postoperative mortality, morbidity, and prolonged LOS. Quality improvement efforts should focus on iAE prevention, mitigation of harm after iAEs occur, and risk/severity-adjusted iAE tracking and benchmarking.

Readiness of US General Surgery Residents for Independent Practice.

OBJECTIVE: This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy. BACKGROUND: The American Board of Surgery has designated 132 procedures as being "Core" to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate. There is growing concern that not all residents achieve that standard. Lack of operative autonomy may play a role. METHODS: Attendings in 14 General Surgery programs were trained to use a) the 5-level System for Improving and Measuring Procedural Learning (SIMPL) Performance scale to assess resident readiness for independent practice and b) the 4-level Zwisch scale to assess the level of guidance (ie, autonomy) they provided to residents during specific procedures. Ratings were collected immediately after cases that involved a categorical GS resident. Data were analyzed using descriptive statistics and supplemented with Bayesian ordinal model-based estimation. RESULTS: A total of 444 attending surgeons rated 536 categorical residents after 10,130 procedures. Performance: from the first to the last year of training, the proportion of Performance ratings for Core procedures (n = 6931) at "Practice Ready" or above increased from 12.3% to 77.1%. The predicted probability that a typical trainee would be rated as Competent after performing an average Core procedure on an average complexity patient during the last week of residency training is 90.5% (95% CI: 85.7%-94%). This falls to 84.6% for more complex patients and to less than 80% for more difficult Core procedures. Autonomy: for all procedures, the proportion of Zwisch ratings indicating meaningful autonomy ("Passive Help" or "Supervision Only") increased from 15.1% to 65.7% from the first to the last year of training. For the Core procedures performed by residents in their final 6 months of training (cholecystectomy, inguinal/femoral hernia repair, appendectomy, ventral hernia repair, and partial colectomy), the proportion of Zwisch ratings (n = 357) indicating near-independence ("Supervision Only") was 33.3%. CONCLUSIONS: US General Surgery residents are not universally ready to independently perform Core procedures by the time they complete residency training. Progressive resident autonomy is also limited. It is unknown if the amount of autonomy residents do achieve is sufficient to ensure readiness for the entire spectrum of independent practice.

Derivation and Validation of a Novel Physiological Emergency Surgery Acuity Score (PESAS).

BACKGROUND: We present a novel and abbreviated Physiological Emergency Surgery Acuity Score (PESAS) that assesses the severity of disease at presentation in patients undergoing Emergency Surgery (ES). METHODS: Using the 2011 ACS-NSQIP database, we identified all patients who underwent "emergent" surgery. The following methodology was designed: (1) identification of independent predictors of 30-day mortality that are markers of acuity; (2) derivation of PESAS based on the relative impact (i.e., odds ratio) of each predictor; and (3) measurement of the c-statistic. The PESAS was validated using the 2012 ACS-NSQIP database. RESULTS: From 24,702 ES cases, a 15-point score was derived. This included 10 components with a range of 0 and 15 points. Its c-statistic was 0.80. Mortality gradually increased from 1.7 to 40.6 to 100% at scores of 0, 8, and 15, respectively. In the validation phase, PESAS c-statistic remained stable. CONCLUSION: PESAS is a novel score that assesses the acuity of disease at presentation in ES patients and strongly correlates with postoperative mortality risk. PESAS could prove useful for preoperative counseling and for risk-adjusted benchmarking.

Evaluation of Oral and Maxillofacial Surgery Residents' Operative Skills: Feasibility and Engagement Study Using SIMPL Software for a Mobile Phone.

PURPOSE: There are no universally accepted tools to evaluate operative skills of surgical residents in a timely fashion. The purpose of this study was to determine the feasibility of using a smartphone application, SIMPL (System for Improving and Measuring Procedural Learning), developed by a multi-institutional research collaborative, to achieve a high rate of timely operative evaluations and resident communication and to collect performance data. The authors hypothesized that these goals would be achieved because the process is convenient and efficient. MATERIALS AND METHODS: This was a prospective feasibility and engagement study using SIMPL to evaluate residents' operative skills. SIMPL requires the attending surgeon to answer 3 multiple-choice questions: 1) What level of help (Zwisch Scale) was required by the trainee? 2) What was the level of performance? 3) How complex was the case? The evaluator also can dictate a narrative. The sample was composed of 3 faculty members and 3 volunteer senior residents. Predictor variables were the surgeons, trainees, and procedures performed. Outcome variables included number and percentage of procedures performed by faculty-and-resident pairs assessed, time required to complete assessments, time lapsed to submission, percentage of assessments with narratives, and residents' response rates. RESULTS: From March through June 2016, 151 procedures were performed in the operating room by the faculty-and-resident teams. There were 107 assessments submitted (71%). Resident response (self-assessment) to faculty evaluations was 81%. Recorded time to complete assessments (n = 75 of 107) was shorter than 2 minutes. The time lapsed to submission was shorter than 72 hours (100%). Dictations were submitted for 35 evaluations (33%). Data for the type of help, performance, and complexity of cases were collected for each resident. CONCLUSIONS: SIMPL facilitates timely intraoperative evaluations of surgical skills, engagement by faculty and residents, and collection of detailed procedural data. Additional prospective trials to assess this tool further are planned.

Surgical resident involvement in foot and ankle surgery.

BACKGROUND: Surgical resident participation in the operating room is necessary for education and progression toward safe and independent practice. However, the impact of resident involvement on patient outcomes in foot and ankle surgery is unknown. METHODS: The American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) database (2005-2012) was used to identify common foot and ankle procedures (by Current Procedural Taxonomy (CPT) code) performed by orthopedic surgeons. Resident participation was determined using the NSQIP-collected variable 'pgy'; cases missing the pgy variable were excluded. Multivariate regression models were constructed to determine an association between resident involvement and 30-day morbidity (total, medical, and surgical complications) and 30-day mortality, when controlling for patient demographics, comorbidities, American Society for Anesthesiologist (ASA) status, body mass index (BMI), and smoking status. RESULTS: A total of 13,685 cases were analyzed for 24 common foot and ankle operations. Overall mortality rate was 3.60%. Overall complication rate was 16.9%; 10.9% had medical and 8.3% had surgical complications. Residents were involved in 55.6% of cases. In unadjusted analyses, resident cases were less likely to be emergent, but were performed on more complicated patients (i.e. higher comorbidity burden, higher ASA scores). Resident cases had increased total morbidity (18.8% vs. 14.6%, p<0.001), medical complications (12.5% vs. 9.0%, p<0.001), and surgical complications (8.7% vs. 7.7%, p=0.03), but similar mortality frequency (3.8% vs. 3.3%, p=0.2). In multivariable analyses, resident cases did not correlate with 30-day mortality, 30-day total morbidity, or 30-day surgical complications; resident cases were, however, associated with increased medical complications [Odds Ratio (OR) 1.18 (95% Confidence Interval (CI) 1.02-1.37, p=0.03)] and longer length of stay [Coeff 2.38 (1.68-3.09), p<0.001]. Subgroup analyses of orthopedic-only cases demonstrated no statistical association between resident involvement and mortality, total morbidity, or medical complications; a decrease in surgical complications was observed for open reduction internal fixation cases [OR 0.23 (0.06-0.82), p=0.02]. CONCLUSIONS: Resident involvement in foot and ankle surgery is not associated with changes in 30-day mortality, 30-day total morbidity, or 30-day surgical complication rates. Residents operate on more medically complex patients who experience higher medical complication rates and longer postoperative length of stay; however, the cause and directionality of this relationship remains to be determined. Efforts to improve the quality of foot and ankle surgery with resident involvement should target reductions in post-operative medical complications. LEVEL OF EVIDENCE: Prognostic study, Level II.

"Idealized" vs. "True" learning curves: the case of laparoscopic liver resection.

BACKGROUND: Learning curves are believed to resemble an "idealized" model, in which continuous improvement occurs until a plateau is reached. We hypothesized that this "idealized" model would not adequately describe the learning process for a complex surgical technique, specifically laparoscopic liver resection (LLR). METHODS: We analyzed the first 150 LLRs performed by a surgeon with expertise in hepatobiliary/laparoscopic surgery but with no previous LLR experience. We divided the procedures performed in 5 consecutive groups of 30 procedures, then compared groups in terms of complications, operative time, length of stay, and estimated blood loss. RESULTS: We observed an increase in operative complexity (3.3% major operations in Group 1 vs. 23.3% in Group 5, p = 0.05). Complications decreased from Group 1 to Group 2 (20%-3%), but increased again as more complex procedures were performed (3% in Group 2-13% in Group 5). Similar improvement/regression patterns were observed for operative time and EBL. DISCUSSION: The "true" learning curve for LLR is more appropriately described as alternating periods of improvement and regression until mastery is achieved. Surgeons should understand the true learning curves of procedures they perform, recognizing and mitigating the increased risk they assume by taking on more complex procedures.

Operating Room Times For Teaching and Nonteaching Cases are Converging: Less Time for Learning?

OBJECTIVE: To compare differences in operating room (OR) times between teaching and nonteaching cases across calendar years. We hypothesize that time devoted to intraoperative resident education is decreasing, therefore, OR times for teaching and nonteaching cases will be converging. BACKGROUND: Teaching cases take longer than similar nonteaching cases, in part due to intraoperative resident education. Pressures to improve OR efficiency and patient safety may threaten resident education and leave less time for intraoperative learning; however, the magnitude of impact is unknown. SETTING/PARTICIPANTS: National Surgical Quality Improvement Program (NSQIP) deidentified national databases from 2006 to 2012, queried for 30 most common General surgery procedures and case teaching status (i.e., teaching vs. nonteaching cases). DESIGN: The NSQIP database was retrospectively reviewed to identify the 30 most common General Surgery procedures. Teaching cases included all operations in which a resident participated. Multivariable regression analyses were constructed to determine the impact of resident involvement on OR times, controlling for year, resident participation, procedure, and patient demographics and comorbidities. Difference-in-difference analysis was performed to assess OR time differences between teaching and nonteaching cases across calendar years and within subpopulations. RESULTS: A total of 693,223 cases met inclusion criteria. Average overall OR times were 98.89 minutes (teaching) vs. 74.22 minutes (nonteaching), with a difference of 24.67 minutes (95% confidence interval [CI] 24.34-24.99 minutes, p < 0.001). In multivariable analyses, the difference between teaching and nonteaching cases was 21.94 minutes (95% CI = 21.11-22.76) in 2006 and 13.95 minutes (95% CI = 10.62-17.28) in 2012, with a difference-in-difference of 7.99 minutes per case. A similar trend was observed across individual PGYs and several individual procedures. CONCLUSIONS: OR times for teaching and nonteaching cases converged by approximately 8 minutes per general surgery procedure during the 7-year study period, representing a 36% reduction in the difference between groups. We must seek to better understand the source of this convergence, and in doing so ensure to preserve and enhance the intraoperative learning experience of surgical trainees.

The reliability of resident self-evaluation of operative performance.

BACKGROUND: Self-assessment is critical to professional self-regulation yet many trainees may not reliably self-evaluate. We examine the gap between resident and faculty perceptions of trainee operative performance and contributing factors. METHODS: Surgery resident and faculty evaluations of trainee performance were collected from 14 academic institutions using smartphone-based performance assessments. Differences in resident/faculty ratings evaluating the same procedure were analyzed using descriptive statistics and Bayesian mixed models. RESULTS: Of 7382 evaluations, 46% trainees and faculty performance ratings were discrepant (r = 0.47), with 80% residents rating themselves lower than faculty in those cases. This gap existed regardless of case complexity and widened as trainees gained experience. Trainees who overrated themselves had the lowest mean performance scores from faculty. CONCLUSION: Half of residents perceived their performance differently from faculty, and this difference widened for senior residents. Future focus should be to provide opportunity for trainees to improve skills to reliably assess themselves before graduation.

Using smartphones for trainee performance assessment: A SIMPL case study.

Workplace-based assessments are used by raters to evaluate observed performance of trainees in actual clinical practice. These types of assessments are of growing interest, especially because observed performance is prioritized within the larger competency-based medical educational movement. Implementation of workplace-based assessments has, however, been challenging. This article describes the motivations and implications for workplace-based assessments that leverage smartphone technology. It does so in reference to an app called SIMPL (System for Improving and Measuring Procedural Learning) in order to highlight some of the challenges and benefits one might encounter during implementation of similar systems.

Design and Impact of a Novel Surgery-Specific Second Victim Peer Support Program.

BACKGROUND: Surgeons are prone to feelings of sadness, guilt, and anxiety when involved in major adverse events. We aimed to create and evaluate a second victim peer support program for surgeons and surgical trainees. STUDY DESIGN: The second victim peer support program was an intervention performed in the Department of Surgery at a tertiary care academic medical center. Surgical attendings and trainees participated as peer supporters or affected peers. In this article, we describe the design of the program and its 1-year impact, which was evaluated through the number of interventions attempted and realized and feedback received from all participants using an anonymous qualitative and quantitative survey. RESULTS: The program was established using the following 5 steps: creation of a conceptual framework, choice of peer supporters, training of peer supporters, multifaceted identification of major adverse events, and design of a systematic intervention plan. In 1 year, the program had 47 interventions distributed evenly between attendings and trainees; 19% of affected peers opted out of receiving support. Most participants expressed satisfaction with the program's confidentiality, the safe/trusting environment it provided, and the timeliness of the intervention (89%, 73%, and 83%, respectively); 81% suggested that the program had a positive impact on the department's "safety and support" culture and would recommend the program to a colleague. Several areas for improvement were identified, including the need to improve identification of events requiring outreach, and the desire for increased awareness of the program throughout the department. CONCLUSIONS: We successfully designed, implemented, and assessed the impact of the first surgery-specific peer support program in the US. Our 1-year experience suggests that the program is highly used and well received, albeit with opportunities for improvement.

Trainee Autonomy in Minimally Invasive General Surgery in the United States: Establishing a National Benchmark.

OBJECTIVE: Minimally invasive surgery (MIS) is an integral component of General Surgery training and practice. Yet, little is known about how much autonomy General Surgery residents achieve in MIS procedures, and whether that amount is sufficient. This study aims to establish a contemporary benchmark for trainee autonomy in MIS procedures. We hypothesize that trainees achieve progressive autonomy, but fail to achieve meaningful autonomy in a substantial percentage of MIS procedures prior to graduation. SETTING/PARTICIPANTS: Fifty General Surgery residency programs in the United States, from September 1, 2015 to March 19, 2020. All Categorical General Surgery Residents and Attending Surgeons within these programs were eligible. DESIGN: Data were collected prospectively from attending surgeons and categorical General Surgery residents. Trainee autonomy was assessed using the 4-level Zwisch scale (Show and Tell, Active Help, Passive Help, and Supervision Only) on a smartphone application (SIMPL). MIS procedures included all laparoscopic, thoracoscopic, endoscopic, and endovascular/percutaneous procedures performed by residents during the study. Primary outcomes of interest were "meaningful autonomy" rates (i.e., scores in the top 2 categories of the Zwisch scale) by postgraduate year (PGY), and "progressive autonomy" (i.e., differences in autonomy between PGYs) in MIS procedures, as rated by attending surgeons. Primary outcomes were determined with descriptive statistics, one-way analysis of variance (ANOVA) and Z-tests. Secondary analyses compared (i) progressive autonomy between common MIS procedures, and (ii) progressive autonomy in MIS vs. non-MIS procedures. RESULTS: A total of 106,054 evaluations were performed across 50 General Surgery residency programs, of which 38,985 (37%) were for MIS procedures. Attendings performed 44,842 (42%) of all evaluations, including 16,840 (43%) of MIS evaluations, while residents performed the rest. Overall, meaningful autonomy in MIS procedures increased from 14.1% (PGY1s) to 75.9% (PGY5s), with significant (p < 0.001) increases between each PGY level. Meaningful autonomy rates were higher in the MIS vs. non-MIS group [57.2% vs. 48.0%, p < 0.001], and progressed more rapidly in MIS vs. non-MIS, (p < 0.05). The 7 most common MIS procedures accounted for 83.5% (n = 14,058) of all MIS evaluations. Among PGY5s performing these procedures, meaningful autonomy rates (%) were: laparoscopic appendectomy (95%); laparoscopic cholecystectomy (93%); diagnostic laparoscopy (87%); upper/lower endoscopy (85%); laparoscopic hernia repair (72%); laparoscopic partial colectomy (58%); and laparoscopic sleeve gastrectomy (45%). CONCLUSIONS: US General Surgery residents receive progressive autonomy in MIS procedures, and appear to progress more rapidly in MIS versus non-MIS procedures. However, residents fail to achieve meaningful autonomy in nearly 25% of MIS cases in their final year of residency, with higher rates of meaningful autonomy only achieved in a small subset of basic MIS procedures.

Mind the Gap: The Autonomy Perception Gap in the Operating Room by Surgical Residents and Faculty.

OBJECTIVE: Examine the concordance of perceived operative autonomy between attendings and resident trainees. DESIGN: Faculty and trainees rated trainee operative autonomy using the 4-level Zwisch scale over a variety of cases and training years. The respective ratings were then compared to explore the effects of experience, gender, case complexity, trainee, trainer, and other covariates to perceived autonomy. SETTING: This study was conducted over 14 general surgery programs in the United States, members of the Procedural Learning and Safety Collaborative. PARTICIPANTS: Participants included faculty and categorical trainees from 14 general surgery programs. RESULTS: A total of 8681 observations was obtained. The sample included 619 unique residents and 457 different attendings. A total of 598 distinct procedures was performed. In 60% of the cases, the autonomy ratings between trainees and attendings were concordant, with only 3.5% of cases discrepant by more than 1 level. An autonomy perception gap was modeled based on the discrepancy between the trainee and attending Zwisch ratings for the same case. The mean Zwisch score expected for a trainee was lower than the attending across all scenarios. Trainees were more likely to perceive relatively more autonomy in the second half of the year. The autonomy perception gap decreased with increasing case complexity. As trainees gained experience, the perception gap increased with trainees underestimating autonomy. CONCLUSIONS: Trainees and attendings generally demonstrated concordance on autonomy perception scores. However, in 40% of cases, a perception gap exists between trainee and attending with the trainee generally underestimating autonomy. The gap worsens as the trainee progresses through residency. This perception gap suggests that attendings and trainees could be better aligned on teaching goals and expectations.

Concordance Between Expert and Nonexpert Ratings of Condensed Video-Based Trainee Operative Performance Assessment.

OBJECTIVE: We examined the impact of video editing and rater expertise in surgical resident evaluation on operative performance ratings of surgical trainees. DESIGN: Randomized independent review of intraoperative video. SETTING: Operative video was captured at a single, tertiary hospital in Boston, MA. PARTICIPANTS: Six common general surgery procedures were video recorded of 6 attending-trainee dyads. Full-length and condensed versions (n = 12 videos) were then reviewed by 13 independent surgeon raters (5 evaluation experts, 8 nonexperts) using a crossed design. Trainee performance was rated using the Operative Performance Rating Scale, System for Improving and Measuring Procedural Learning (SIMPL) Performance scale, the Zwisch scale, and ten Cate scale. These ratings were then standardized before being compared using Bayesian mixed models with raters and videos treated as random effects. RESULTS: Editing had no effect on the Operative Performance Rating Scale Overall Performance (-0.10, p = 0.30), SIMPL Performance (0.13, p = 0.71), Zwisch (-0.12, p = 0.27), and ten Cate scale (-0.13, p = 0.29). Additionally, rater expertise (evaluation expert vs. nonexpert) had no effect on the same scales (-0.16 (p = 0.32), 0.18 (p = 0.74), 0.25 (p = 0.81), and 0.25 (p = 0.17). CONCLUSIONS: There is little difference in operative performance assessment scores when raters use condensed videos or when raters who are not experts in surgical resident evaluation are used. Future validation studies of operative performance assessment scales may be facilitated by using nonexpert surgeon raters viewing videos condensed using a standardized protocol.