A Competency-based Laparoscopic Cholecystectomy Curriculum Significantly Improves General Surgery Residents' Operative Performance and Decreases Skill Variability: Cohort Study.

OBJECTIVE: To demonstrate the feasibility of implementing a CBE curriculum within a general surgery residency program and to evaluate its effectiveness in improving resident skill. SUMMARY OF BACKGROUND DATA: Operative skill variability affects residents and practicing surgeons and directly impacts patient outcomes. CBE can decrease this variability by ensuring uniform skill acquisition. We implemented a CBE LC curriculum to improve resident performance and decrease skill variability. METHODS: PGY-2 residents completed the curriculum during monthly rotations starting in July 2017. Once simulator proficiency was reached, residents performed elective LCs with a select group of faculty at 3 hospitals. Performance at curriculum completion was assessed using LC simulation metrics and intraoperative operative performance rating system scores and compared to both baseline and historical controls, comprised of rising PGY-3s, using a 2-sample Wilcoxon rank-sum test. PGY-2 group's performance variability was compared with PGY-3s using Levene robust test of equality of variances; P < 0.05 was considered significant. RESULTS: Twenty-one residents each performed 17.52 ± 4.15 consecutive LCs during the monthly rotation. Resident simulated and operative performance increased significantly with dedicated training and reached that of more experienced rising PGY-3s (n = 7) but with significantly decreased variability in performance ( P = 0.04). CONCLUSIONS: Completion of a CBE rotation led to significant improvements in PGY-2 residents' LC performance that reached that of PGY-3s and decreased performance variability. These results support wider implementation of CBE in resident training.

Career Advancement for Surgeon-Educators:Findings from a Modified Delphi Process.

OBJECTIVES: Reward and recognition of surgical education as an academic activity remains a highly variable process between institutions. The goal of this study is to provide expert consensus definition of an academic surgical educator, with focus on criteria for academic promotion. STUDY DESIGN AND SETTING: Following IRB approval, a Web-based modified Delphi process was used to generate prioritized academic promotion criteria for surgical educators. PARTICIPANTS AND SETTING: Participants were recruited nationally from a pool of senior academic surgeons who are members of the Society of University Surgeons and the Society of Surgical Chairs. RESULTS: Following a three-round modified Delphi process, the top domains of educational activity for promotion to associate professor and professor were scholarship, teaching, and administration; mentorship was also a priority category for promotion to professor. The top three activities described for promotion to Associate Professor were active participation in conferences/ departmental educational activities for medical students and residents; educational portfolio demonstrating commitment to activities as an educator; and clinical teaching excellence at their home institution. The three activities most highly scored items for promotion to Professor were mentorship of junior surgical educators; active participation in conferences/ departmental educational activities for medical students and residents; and a record of teaching excellence at the medical student and resident levels. CONCLUSIONS: These findings demonstrate a progression from teacher to scholar to leader across a surgical educator's career, with each level incorporating and building upon the prior activities. Identification of categories and criteria may meaningfully inform best practices to be incorporated into the career development and promotion processes for surgeons on an educator academic pathway.

A Proposed Blueprint for Operative Performance Training, Assessment, and Certification.

OBJECTIVE: The aim of this study was to propose an evidence-based blueprint for training, assessment, and certification of operative performance for surgical trainees. SUMMARY BACKGROUND DATA: Operative skill is a critical aspect of surgical performance. High-quality assessment of operative skill therefore has profound implications for training, accreditation, certification, and the public trust of the profession. Current methods of operative skill assessment for surgeons rely heavily on global assessment strategies across a very broad domain of procedures. There is no mechanism to assure technical competence for individual procedures. The science and scalability of operative skill assessment has progressed significantly in recent decades, and can inform a much more meaningful strategy for competency-based assessment of operative skill than has been previously achieved. METHODS: The present article reviews the current status and science of operative skill assessment and proposes a template for competency-based assessment which could be used to update training, accreditation, and certification processes. The proposal is made in reference to general surgery but is more generally applicable to other procedural specialties. RESULTS: Streamlined, routine assessment of every procedure performed by surgical trainees is feasible and would enable a more competency-based educational paradigm. In light of the constraints imposed by both clinical volume and assessment bias, trainees should be expected to become proficient and be measured against a mastery learning standard only for the most important and highest-frequency procedures. For less frequently observed procedures, performance can be compared to a norm-referenced standard and, to provide an overall trajectory of performance, analyzed in aggregate. Key factors in implementing this approach are the number of evaluations, the number of raters, the timeliness of evaluation, and evaluation items. CONCLUSIONS: A competency-based operative skill assessment can be incorporated into surgical training, assessment, and certification. The time has come to develop a systematic approach to this issue as a means of demonstrating professional standards worthy of the public trust.

Trainee Autonomy in Minimally Invasive General Surgery in the United States: Establishing a National Benchmark.

OBJECTIVE: Minimally invasive surgery (MIS) is an integral component of General Surgery training and practice. Yet, little is known about how much autonomy General Surgery residents achieve in MIS procedures, and whether that amount is sufficient. This study aims to establish a contemporary benchmark for trainee autonomy in MIS procedures. We hypothesize that trainees achieve progressive autonomy, but fail to achieve meaningful autonomy in a substantial percentage of MIS procedures prior to graduation. SETTING/PARTICIPANTS: Fifty General Surgery residency programs in the United States, from September 1, 2015 to March 19, 2020. All Categorical General Surgery Residents and Attending Surgeons within these programs were eligible. DESIGN: Data were collected prospectively from attending surgeons and categorical General Surgery residents. Trainee autonomy was assessed using the 4-level Zwisch scale (Show and Tell, Active Help, Passive Help, and Supervision Only) on a smartphone application (SIMPL). MIS procedures included all laparoscopic, thoracoscopic, endoscopic, and endovascular/percutaneous procedures performed by residents during the study. Primary outcomes of interest were "meaningful autonomy" rates (i.e., scores in the top 2 categories of the Zwisch scale) by postgraduate year (PGY), and "progressive autonomy" (i.e., differences in autonomy between PGYs) in MIS procedures, as rated by attending surgeons. Primary outcomes were determined with descriptive statistics, one-way analysis of variance (ANOVA) and Z-tests. Secondary analyses compared (i) progressive autonomy between common MIS procedures, and (ii) progressive autonomy in MIS vs. non-MIS procedures. RESULTS: A total of 106,054 evaluations were performed across 50 General Surgery residency programs, of which 38,985 (37%) were for MIS procedures. Attendings performed 44,842 (42%) of all evaluations, including 16,840 (43%) of MIS evaluations, while residents performed the rest. Overall, meaningful autonomy in MIS procedures increased from 14.1% (PGY1s) to 75.9% (PGY5s), with significant (p < 0.001) increases between each PGY level. Meaningful autonomy rates were higher in the MIS vs. non-MIS group [57.2% vs. 48.0%, p < 0.001], and progressed more rapidly in MIS vs. non-MIS, (p < 0.05). The 7 most common MIS procedures accounted for 83.5% (n = 14,058) of all MIS evaluations. Among PGY5s performing these procedures, meaningful autonomy rates (%) were: laparoscopic appendectomy (95%); laparoscopic cholecystectomy (93%); diagnostic laparoscopy (87%); upper/lower endoscopy (85%); laparoscopic hernia repair (72%); laparoscopic partial colectomy (58%); and laparoscopic sleeve gastrectomy (45%). CONCLUSIONS: US General Surgery residents receive progressive autonomy in MIS procedures, and appear to progress more rapidly in MIS versus non-MIS procedures. However, residents fail to achieve meaningful autonomy in nearly 25% of MIS cases in their final year of residency, with higher rates of meaningful autonomy only achieved in a small subset of basic MIS procedures.

Are surgeons behind the scientific eight ball: Delayed acquisition of the NIH K08 mentored career development award.

BACKGROUND: Surgery residents complete their research training early in residency. Non-surgical trainees typically have research incorporated toward the last two years of their fellowship, conferring an advantage to apply for grants with recent research experience and preliminary data. METHODS: The NIH RePORTER database was queried for K08 awardees trained in medicine, pediatrics, and surgery from 2013 to 2017. 406 K08 recipients were identified and time from completion of clinical training to achieving a K08 award was measured. Data were compared using ANOVA and expressed as mean. P < 0.05 was considered significant. RESULTS: Surgeons took longer to obtain a K08 than those trained in internal medicine (surgery = 3.7 years, internal medicine = 2.58 years p < 0.0001)). All K08 recipients without a PhD took longer to obtain a K08 than recipients with a PhD (MD = 3.50 years and MD/PhD = 2.42 years (p=<0.0001). CONCLUSIONS: Surgeons take longer to achieve a K08 award than clinicians trained in internal medicine, possibly due to an inherent disadvantage in training structure.

The effect of gender on operative autonomy in general surgery residents.

BACKGROUND: Despite an increasing number of women in the field of surgery, bias regarding cognitive or technical ability may continue to affect the experience of female trainees differently than their male counterparts. This study examines the differences in the degree of operative autonomy given to female compared with male general surgery trainees. METHODS: A smartphone app was used to collect evaluations of operative autonomy measured using the 4-point Zwisch scale, which describes defined steps in the progression from novice ("show and tell") to autonomous surgeon ("supervision only"). Differences in autonomy between male and female residents were compared using hierarchical logistic regression analysis. RESULTS: A total of 412 residents and 524 faculty from 14 general surgery training programs evaluated 8,900 cases over a 9-month period. Female residents received less autonomy from faculty than did male residents overall (P < .001). Resident level of training and case complexity were the strongest predictors of autonomy. Even after controlling for potential confounding factors, including level of training, intrinsic procedural difficulty, patient-related case complexity, faculty sex, and training program environment, female residents still received less operative autonomy than their male counterparts. The greatest discrepancy was in the fourth year of training. CONCLUSION: There is a sex-based difference in the autonomy granted to general surgery trainees. This gender gap may affect female residents' experience in training and possibly their preparation for practice. Strategies need to be developed to help faculty and residents work together to overcome this gender gap.

"Show Me the Data": A Recipe for Quality Improvement Success in an Academic Surgical Department.

BACKGROUND: Surgeons in academic medical centers have traditionally taken a siloed approach to reducing postoperative complications. We initiated a project focusing on transparency and sharing of data to engage surgeons in collaborative quality improvement. Its key features were the development of a comprehensive department quality dashboard and the creation of the Clinical Operations Council that oversaw quality. The purpose of this study was to assess the impact of those efforts. STUDY DESIGN: We compared inpatient outcomes before and after our intervention, allowing one quarter as the diffusion period. The outcomes analyzed were: risk-adjusted length of stay, mortality, direct cost and unadjusted incidence of complications, and 30-day all-cause readmissions, as determined by the Vizient Clinical Database. We examined the outcomes of three groups: group 1 (surgery); group 2, all other surgical departments (other surgery); and group 3, all other patients (non-surgery). Two-tailed Student's t-test was used for analysis and p < 0.05 was considered statistically significant. RESULTS: Group 1 demonstrated statistically significant improvements in mortality (p = 0.01), length of stay (p = 0.002), cost (p = 0.0001), and complications (p = 0.02), and the all-cause readmission rate was unchanged, resulting in mean decrease of 0.55 length of stay days and direct cost savings of $2,300 per surgical admission. The comparison groups had only modest decreases in some of the analyzed outcomes and an increase in complication rates. CONCLUSIONS: These data suggest that a collaborative, data-driven, and transparent approach to assessing the quality of surgical care can yield significant improvements in patient outcomes.

How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?

OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

The business of educating the next generation of surgeons.

Surgical education community needs to be informed about how education is funded and how it is threatened. In order to explore these issues the Association of Surgical Education convened a panel with significant experience in managing surgery departments to discuss the business of surgical education. They specifically addressed methods to recognize and reward faculty, educate residents on safety, quality and cost, and increase departmental revenue. This information is important in the current educational environment where there is an increased need for institutions to find alternate revenue streams to sustain graduate medical education. It is also important to find additional revenue streams to fund new residency slots to accommodate the greater number medical students who have been admitted to medical schools in response to meet the projected shortage of physicians.

Is the operative autonomy granted to a resident consistent with operative performance quality.

BACKGROUND: We investigated attending surgeon decisions regarding resident operative autonomy, including situations where operative autonomy was discordant with performance quality. METHODS: Attending surgeons assessed operative performance and documented operative autonomy granted to residents from 14 general surgery residency programs. Concordance between performance and autonomy was defined as "practice ready performance/meaningfully autonomous" or "not practice ready/not meaningfully autonomous." Discordant circumstances were practice ready/not meaningfully autonomous or not practice ready/meaningfully autonomous. Resident training level, patient-related case complexity, procedure complexity, and procedure commonality were investigated to determine impact on autonomy. RESULTS: A total of 8,798 assessments were collected from 429 unique surgeons assessing 496 unique residents. Practice-ready and exceptional performances were 20 times more likely to be performed under meaningfully autonomous conditions than were other performances. Meaningful autonomy occurred most often with high-volume, easy and common cases, and less complex procedures. Eighty percent of assessments were concordant (38% practice ready/meaningfully autonomous and 42% not practice ready/not meaningfully autonomous). Most discordant assessments (13.8%) were not practice ready/meaningfully autonomous. For fifth-year residents, practice ready/not meaningfully autonomous ratings (9.7%) were more frequent than not practice ready/meaningfully autonomous ratings (7.5%). Ten surgeons (2.3%) failed to afford residents meaningful autonomy on any occasion. CONCLUSION: Resident operative performance quality is the most important determinant in attending surgeon decisions regarding resident autonomy.

High-speed Intraoperative Assessment of Breast Tumor Margins by Multimodal Ultrasound and Photoacoustic Tomography.

Conventional methods for breast tumor margins assessment need a long turnaround time, which may lead to re-operation for patients undergoing lumpectomy surgeries. Photoacoustic tomography (PAT) has been shown to visualize adipose tissue in small animals and human breast. Here, we demonstrate a customized multimodal ultrasound and PAT system for intraoperative breast tumor margins assessment using fresh lumpectomy specimens from 66 patients. The system provides the margin status of the entire excised tissue within 10 minutes. By subjective reading of three researchers, the results show 85.7% [95% confidence interval (CI), 42.0% - 99.2%] sensitivity and 84.6% (95% CI, 53.7% - 97.3%) specificity, 71.4% (95% CI, 30.3% - 94.9%) sensitivity and 92.3% (95% CI, 62.1% - 99.6%) specificity, and 100% (95% CI, 56.1% - 100%) sensitivity and 53.9% (95% CI, 26.1% - 79.6%) specificity respectively when cross-correlated with post-operational histology. Furthermore, a machine learning-based algorithm is deployed for margin assessment in the challenging ductal carcinoma in situ tissues, and achieved 85.5% (95% CI, 75.2% - 92.2%) sensitivity and 90% (95% CI, 79.9% - 95.5%) specificity. Such results present the potential of using mutlimodal ultrasound and PAT as a high-speed and accurate method for intraoperative breast tumor margins evaluation.

Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.

BACKGROUND: Concerns exist regarding the competency of general surgery graduates with performing core general surgery procedures. Current competence assessment incorporates minimal procedural numbers requirements. METHODS: Based on the Zwisch scale we evaluated the level of autonomy achieved by categorical PGY1-5 general surgery residents at 14 U.S. general surgery resident training programs between September 1, 2015 and December 31, 2016. With 5 of the most commonly performed core general surgery procedures, we correlated the level of autonomy achieved by each resident with the number of procedures they had performed before the evaluation period, with the intent of identifying specific target numbers that would correlate with the achievement of meaningful autonomy for each procedure with most residents. RESULTS: Whereas a definitive target number was identified for laparoscopic appendectomy (i.e. 25), for the other 4 procedures studied (i.e. laparoscopic cholecystectomy, 52; open inguinal hernia repair, 42; ventral hernia repair, 35; and partial colectomy, 60), target numbers identified were less definitive and/or were higher than many residents will experience during their surgical residency training. CONCLUSIONS: We conclude that procedural target numbers are generally not effective in predicting procedural competence and should not be used as the basis for determining residents' readiness for independent practice.

What factors influence attending surgeon decisions about resident autonomy in the operating room?

BACKGROUND: Educating residents in the operating room requires balancing patient safety, operating room efficiency demands, and resident learning needs. This study explores 4 factors that influence the amount of autonomy supervising surgeons afford to residents. METHODS: We evaluated 7,297 operations performed by 487 general surgery residents and evaluated by 424 supervising surgeons from 14 training programs. The primary outcome measure was supervising surgeon autonomy granted to the resident during the operative procedure. Predictor variables included resident performance on that case, supervising surgeon history with granting autonomy, resident training level, and case difficulty. RESULTS: Resident performance was the strongest predictor of autonomy granted. Typical autonomy by supervising surgeon was the second most important predictor. Each additional factor led to a smaller but still significant improvement in ability to predict the supervising surgeon's autonomy decision. The 4 factors together accounted for 54% of decision variance (r = 0.74). CONCLUSION: Residents' operative performance in each case was the strongest predictor of how much autonomy was allowed in that case. Typical autonomy granted by the supervising surgeon, the second most important predictor, is unrelated to resident proficiency and warrants efforts to ensure that residents perform each procedure with many different supervisors.

Readiness of US General Surgery Residents for Independent Practice.

OBJECTIVE: This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy. BACKGROUND: The American Board of Surgery has designated 132 procedures as being "Core" to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate. There is growing concern that not all residents achieve that standard. Lack of operative autonomy may play a role. METHODS: Attendings in 14 General Surgery programs were trained to use a) the 5-level System for Improving and Measuring Procedural Learning (SIMPL) Performance scale to assess resident readiness for independent practice and b) the 4-level Zwisch scale to assess the level of guidance (ie, autonomy) they provided to residents during specific procedures. Ratings were collected immediately after cases that involved a categorical GS resident. Data were analyzed using descriptive statistics and supplemented with Bayesian ordinal model-based estimation. RESULTS: A total of 444 attending surgeons rated 536 categorical residents after 10,130 procedures. Performance: from the first to the last year of training, the proportion of Performance ratings for Core procedures (n = 6931) at "Practice Ready" or above increased from 12.3% to 77.1%. The predicted probability that a typical trainee would be rated as Competent after performing an average Core procedure on an average complexity patient during the last week of residency training is 90.5% (95% CI: 85.7%-94%). This falls to 84.6% for more complex patients and to less than 80% for more difficult Core procedures. Autonomy: for all procedures, the proportion of Zwisch ratings indicating meaningful autonomy ("Passive Help" or "Supervision Only") increased from 15.1% to 65.7% from the first to the last year of training. For the Core procedures performed by residents in their final 6 months of training (cholecystectomy, inguinal/femoral hernia repair, appendectomy, ventral hernia repair, and partial colectomy), the proportion of Zwisch ratings (n = 357) indicating near-independence ("Supervision Only") was 33.3%. CONCLUSIONS: US General Surgery residents are not universally ready to independently perform Core procedures by the time they complete residency training. Progressive resident autonomy is also limited. It is unknown if the amount of autonomy residents do achieve is sufficient to ensure readiness for the entire spectrum of independent practice.

The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial.

PURPOSE: Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. METHODS: Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. RESULTS: A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. CONCLUSIONS: SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

A problem-oriented approach to resident performance ratings.

BACKGROUND: Global, end-of-rotation evaluations are often difficult to interpret due to their high level of abstraction (eg, excellent, good, poor) and the bias toward high ratings. This study documents the utility of and measurement characteristics of serious problem items, an alternative item format. METHODS: This report is based on 4,234 faculty performance ratings for 105 general surgery residents. Faculty members reported whether each resident had a serious problem for each of 8 areas of clinical performance and 6 areas of professional behavior. RESULTS: A total of 263 serious problems were reported. The performance category with the most total serious problems noted was knowledge and that with the least problems noted was relations with patients and family members. Seven residents accounted for 86.9% of all serious problem reports. Each resident had serious problems in multiple performance areas. Problems were reported most frequently in knowledge, management, technical/procedural skills, ability to assume responsibility within level of competence, and problem identification. Citations of these serious problems were most common in year 3. Traditional ratings of global performance were not an adequate means for identifying residents with serious performance problems. CONCLUSION: Serious problem ratings can communicate faculty concerns about residents more directly and can be used as a complement to conventional global rating scales without substantially increasing faculty workload.

Practice Guidelines for Operative Performance Assessments.

OBJECTIVE: To provide recommended practice guidelines for assessing single operative performances and for combining results of operative performance assessments into estimates of overall operative performance ability. SUMMARY BACKGROUND DATA: Operative performance is one defining characteristic of surgeons. Assessment of operative performance is needed to provide feedback with learning benefits to surgical residents in training and to assist in making progress decisions for residents. Operative performance assessment has been a focus of investigation over the past 20 years. This review is designed to integrate findings of this research into a set of recommended operative performance practices. METHODS: Literature from surgery and from other pertinent research areas (psychology, education, business) was reviewed looking for evidence to inform practice guideline development. Guidelines were created along with a conceptual and scientific foundation for each guideline. RESULTS: Ten guidelines are provided for assessing individual operative performances and 10 are provided for combing data from individual operative performances into overall judgments of operative performance ability. CONCLUSIONS: The practice guidelines organize available information to be immediately useful to program directors, to support surgical training, and to provide a conceptual framework upon which to build as the base of pertinent knowledge expands through future research and development efforts.