Trainee Autonomy in Minimally Invasive General Surgery in the United States: Establishing a National Benchmark.

OBJECTIVE: Minimally invasive surgery (MIS) is an integral component of General Surgery training and practice. Yet, little is known about how much autonomy General Surgery residents achieve in MIS procedures, and whether that amount is sufficient. This study aims to establish a contemporary benchmark for trainee autonomy in MIS procedures. We hypothesize that trainees achieve progressive autonomy, but fail to achieve meaningful autonomy in a substantial percentage of MIS procedures prior to graduation. SETTING/PARTICIPANTS: Fifty General Surgery residency programs in the United States, from September 1, 2015 to March 19, 2020. All Categorical General Surgery Residents and Attending Surgeons within these programs were eligible. DESIGN: Data were collected prospectively from attending surgeons and categorical General Surgery residents. Trainee autonomy was assessed using the 4-level Zwisch scale (Show and Tell, Active Help, Passive Help, and Supervision Only) on a smartphone application (SIMPL). MIS procedures included all laparoscopic, thoracoscopic, endoscopic, and endovascular/percutaneous procedures performed by residents during the study. Primary outcomes of interest were "meaningful autonomy" rates (i.e., scores in the top 2 categories of the Zwisch scale) by postgraduate year (PGY), and "progressive autonomy" (i.e., differences in autonomy between PGYs) in MIS procedures, as rated by attending surgeons. Primary outcomes were determined with descriptive statistics, one-way analysis of variance (ANOVA) and Z-tests. Secondary analyses compared (i) progressive autonomy between common MIS procedures, and (ii) progressive autonomy in MIS vs. non-MIS procedures. RESULTS: A total of 106,054 evaluations were performed across 50 General Surgery residency programs, of which 38,985 (37%) were for MIS procedures. Attendings performed 44,842 (42%) of all evaluations, including 16,840 (43%) of MIS evaluations, while residents performed the rest. Overall, meaningful autonomy in MIS procedures increased from 14.1% (PGY1s) to 75.9% (PGY5s), with significant (p < 0.001) increases between each PGY level. Meaningful autonomy rates were higher in the MIS vs. non-MIS group [57.2% vs. 48.0%, p < 0.001], and progressed more rapidly in MIS vs. non-MIS, (p < 0.05). The 7 most common MIS procedures accounted for 83.5% (n = 14,058) of all MIS evaluations. Among PGY5s performing these procedures, meaningful autonomy rates (%) were: laparoscopic appendectomy (95%); laparoscopic cholecystectomy (93%); diagnostic laparoscopy (87%); upper/lower endoscopy (85%); laparoscopic hernia repair (72%); laparoscopic partial colectomy (58%); and laparoscopic sleeve gastrectomy (45%). CONCLUSIONS: US General Surgery residents receive progressive autonomy in MIS procedures, and appear to progress more rapidly in MIS versus non-MIS procedures. However, residents fail to achieve meaningful autonomy in nearly 25% of MIS cases in their final year of residency, with higher rates of meaningful autonomy only achieved in a small subset of basic MIS procedures.

Beyond the Surgical Time-Out: A National Needs Assessment of Preoperative Communication in US General Surgery Residency Programs.

OBJECTIVE: Perioperative communication is critical for procedural learning. In order to develop a periprocedural faculty development tool, we aimed to characterize the current status of preoperative communication in US General Surgery residency programs. DESIGN: After Association of Program Directors in Surgery approval, a survey was distributed to general surgery programs. Participants were asked about perioperative communication, including the frequency of preoperative briefings, defined as dedicated educational discussions prior to a procedure. Data were analyzed using descriptive statistics. SETTING: An anonymous electronic survey was distributed to interested programs in early 2019. PARTICIPANTS: US General Surgery trainees and attending surgeons. RESULTS: A total of 348 responses were recorded from 27 programs: 199 (57%) attending surgeons and 149 (43%) surgical trainees. Most respondents (83%) were from a university-affiliated program. Attending surgeons indicated a higher frequency of performing preoperative briefings compared to trainees (p < 0.001). Both trainees and attending surgeons were more likely to select their own group when asked who initiates a preoperative briefing. The majority of respondents (58%) agreed that discussing autonomy preoperatively improves resident autonomy for the case. In regards to the timing of preoperative briefings, most took place in/adjacent to the operating room, with only 60 participants (17%) participating in preoperative briefings the day/night prior to the operation. The most frequent topic discussed during preoperative briefings was "procedural content." Most participants selected "time constraints" as the greatest barrier to preoperative briefings and indicated that attending surgeon engagement was necessary to facilitate their use. Trainees were less likely to report engaging in immediate postoperative feedback, but more likely to report postoperative self-reflection. CONCLUSIONS: Preoperative briefings are not necessarily routine and attendings and trainees differ on their perceptions related to their content and frequency. Efforts to address timing and scheduling and encourage dual-party engagement in perioperative communication are key to the development of tools to enhance this important aspect of procedural learning.

A Preoperative Educational Time-Out is Associated with Improved Resident Goal Setting and Strengthens Educational Experiences.

OBJECTIVE: The purpose of this study was to assess the impact of a preoperative Educational Time-Out (ETO) with structured postoperative feedback on resident preoperative goal-setting and the educational experience of a clinical rotation. DESIGN: A preoperative ETO was developed during which trainees and faculty jointly identified an operative goal and discussed the trainee's operative autonomy. Postoperative feedback with a smartphone application was encouraged. From November 2016 to October 2017, the intervention was piloted with 1 surgical service. Outcomes included ETO completion rate, goal setting rate, and subjects' perception of the impact of the ETO on identification of performance deficits, trainee autonomy, and receipt of feedback. Data were analyzed using descriptive statistics. SETTING: This study was performed in an institutional hospital setting. PARTICIPANTS: Third-year general surgery residents and surgical faculty in the Department of Hepatobiliary Surgery and Liver Transplantation at Vanderbilt University Medical Center took part in the intervention. RESULTS: Seven residents and 7 attending surgeons participated in this study. Residents performed a median of 15 procurements during an average of 6.5 weeks each on service. The ETO completion rate was 83%. Resident-reported preoperative goal setting increased after the intervention (from 36% to 78%, p = 0.015). Subjects reported a positive impact of the intervention, with high resident agreement that the ETO helped identify deficits (82% median agreement), increased autonomy (82% median agreement), and increased receipt of feedback (84% median agreement). Residents and attendings agreed that the educational experience was stronger due to the ETO (median 81% and 77%, respectively). CONCLUSIONS: The ETO intervention improved rates of resident preoperative goal setting and strengthened perceived educational experiences. Resident participants also reported improvements in autonomy and rates of postoperative feedback. Broader implementation of this brief preoperative pause is an easy way to emphasize procedural education in the operating room.

Using smartphones for trainee performance assessment: A SIMPL case study.

Workplace-based assessments are used by raters to evaluate observed performance of trainees in actual clinical practice. These types of assessments are of growing interest, especially because observed performance is prioritized within the larger competency-based medical educational movement. Implementation of workplace-based assessments has, however, been challenging. This article describes the motivations and implications for workplace-based assessments that leverage smartphone technology. It does so in reference to an app called SIMPL (System for Improving and Measuring Procedural Learning) in order to highlight some of the challenges and benefits one might encounter during implementation of similar systems.

How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?

OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

Is the operative autonomy granted to a resident consistent with operative performance quality.

BACKGROUND: We investigated attending surgeon decisions regarding resident operative autonomy, including situations where operative autonomy was discordant with performance quality. METHODS: Attending surgeons assessed operative performance and documented operative autonomy granted to residents from 14 general surgery residency programs. Concordance between performance and autonomy was defined as "practice ready performance/meaningfully autonomous" or "not practice ready/not meaningfully autonomous." Discordant circumstances were practice ready/not meaningfully autonomous or not practice ready/meaningfully autonomous. Resident training level, patient-related case complexity, procedure complexity, and procedure commonality were investigated to determine impact on autonomy. RESULTS: A total of 8,798 assessments were collected from 429 unique surgeons assessing 496 unique residents. Practice-ready and exceptional performances were 20 times more likely to be performed under meaningfully autonomous conditions than were other performances. Meaningful autonomy occurred most often with high-volume, easy and common cases, and less complex procedures. Eighty percent of assessments were concordant (38% practice ready/meaningfully autonomous and 42% not practice ready/not meaningfully autonomous). Most discordant assessments (13.8%) were not practice ready/meaningfully autonomous. For fifth-year residents, practice ready/not meaningfully autonomous ratings (9.7%) were more frequent than not practice ready/meaningfully autonomous ratings (7.5%). Ten surgeons (2.3%) failed to afford residents meaningful autonomy on any occasion. CONCLUSION: Resident operative performance quality is the most important determinant in attending surgeon decisions regarding resident autonomy.

Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment.

OBJECTIVE: We have previously demonstrated the feasibility and validity of a smartphone-based system called Procedural Autonomy and Supervision System (PASS), which uses the Zwisch autonomy scale to facilitate assessment of the operative performances of surgical residents and promote progressive autonomy. To determine whether the use of PASS in a general surgery residency program is associated with any negative consequences, we tested the null hypothesis that PASS implementation at our institution would not negatively affect resident or faculty satisfaction in the operating room (OR) nor increase mean OR times for cases performed together by residents and faculty. METHODS: Mean OR times were obtained from the electronic medical record at Northwestern Memorial Hospital for the 20 procedures most commonly performed by faculty members with residents before and after PASS implementation. OR times were compared via two-sample t-test. The OR Educational Environment Measure tool was used to assess OR satisfaction with all clinically active general surgery residents (n = 31) and full-time general surgery faculty members (n = 27) before and after PASS implementation. Results were compared using the Mann-Whitney rank sum test. RESULTS: A significant prolongation in mean OR time between control and study period was found for only 1 of the 20 operative procedures performed at least 20 times by participating faculty members with residents. Based on the overall survey score, no significant differences were found between resident and faculty responses to the OR Educational Environment Measure survey before and after PASS implementation. When individual survey items were compared, while no differences were found with resident responses, differences were noted with faculty responses for 7 of the 35 items addressed although after Bonferroni correction none of these differences remained significant. CONCLUSIONS: Our data suggest that PASS does not increase mean OR times for the most commonly performed procedures. Resident OR satisfaction did not significantly change during PASS implementation, whereas some changes in faculty satisfaction were noted suggesting that PASS implementation may have had some negative effect with them. Although the effect on faculty satisfaction clearly requires further investigation, our findings support that use of an autonomy-based OR performance assessment system such as PASS does not appear to have a major negative influence on OR times nor OR satisfaction.

Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.

BACKGROUND: Concerns exist regarding the competency of general surgery graduates with performing core general surgery procedures. Current competence assessment incorporates minimal procedural numbers requirements. METHODS: Based on the Zwisch scale we evaluated the level of autonomy achieved by categorical PGY1-5 general surgery residents at 14 U.S. general surgery resident training programs between September 1, 2015 and December 31, 2016. With 5 of the most commonly performed core general surgery procedures, we correlated the level of autonomy achieved by each resident with the number of procedures they had performed before the evaluation period, with the intent of identifying specific target numbers that would correlate with the achievement of meaningful autonomy for each procedure with most residents. RESULTS: Whereas a definitive target number was identified for laparoscopic appendectomy (i.e. 25), for the other 4 procedures studied (i.e. laparoscopic cholecystectomy, 52; open inguinal hernia repair, 42; ventral hernia repair, 35; and partial colectomy, 60), target numbers identified were less definitive and/or were higher than many residents will experience during their surgical residency training. CONCLUSIONS: We conclude that procedural target numbers are generally not effective in predicting procedural competence and should not be used as the basis for determining residents' readiness for independent practice.

What factors influence attending surgeon decisions about resident autonomy in the operating room?

BACKGROUND: Educating residents in the operating room requires balancing patient safety, operating room efficiency demands, and resident learning needs. This study explores 4 factors that influence the amount of autonomy supervising surgeons afford to residents. METHODS: We evaluated 7,297 operations performed by 487 general surgery residents and evaluated by 424 supervising surgeons from 14 training programs. The primary outcome measure was supervising surgeon autonomy granted to the resident during the operative procedure. Predictor variables included resident performance on that case, supervising surgeon history with granting autonomy, resident training level, and case difficulty. RESULTS: Resident performance was the strongest predictor of autonomy granted. Typical autonomy by supervising surgeon was the second most important predictor. Each additional factor led to a smaller but still significant improvement in ability to predict the supervising surgeon's autonomy decision. The 4 factors together accounted for 54% of decision variance (r = 0.74). CONCLUSION: Residents' operative performance in each case was the strongest predictor of how much autonomy was allowed in that case. Typical autonomy granted by the supervising surgeon, the second most important predictor, is unrelated to resident proficiency and warrants efforts to ensure that residents perform each procedure with many different supervisors.

Readiness of US General Surgery Residents for Independent Practice.

OBJECTIVE: This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy. BACKGROUND: The American Board of Surgery has designated 132 procedures as being "Core" to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate. There is growing concern that not all residents achieve that standard. Lack of operative autonomy may play a role. METHODS: Attendings in 14 General Surgery programs were trained to use a) the 5-level System for Improving and Measuring Procedural Learning (SIMPL) Performance scale to assess resident readiness for independent practice and b) the 4-level Zwisch scale to assess the level of guidance (ie, autonomy) they provided to residents during specific procedures. Ratings were collected immediately after cases that involved a categorical GS resident. Data were analyzed using descriptive statistics and supplemented with Bayesian ordinal model-based estimation. RESULTS: A total of 444 attending surgeons rated 536 categorical residents after 10,130 procedures. Performance: from the first to the last year of training, the proportion of Performance ratings for Core procedures (n = 6931) at "Practice Ready" or above increased from 12.3% to 77.1%. The predicted probability that a typical trainee would be rated as Competent after performing an average Core procedure on an average complexity patient during the last week of residency training is 90.5% (95% CI: 85.7%-94%). This falls to 84.6% for more complex patients and to less than 80% for more difficult Core procedures. Autonomy: for all procedures, the proportion of Zwisch ratings indicating meaningful autonomy ("Passive Help" or "Supervision Only") increased from 15.1% to 65.7% from the first to the last year of training. For the Core procedures performed by residents in their final 6 months of training (cholecystectomy, inguinal/femoral hernia repair, appendectomy, ventral hernia repair, and partial colectomy), the proportion of Zwisch ratings (n = 357) indicating near-independence ("Supervision Only") was 33.3%. CONCLUSIONS: US General Surgery residents are not universally ready to independently perform Core procedures by the time they complete residency training. Progressive resident autonomy is also limited. It is unknown if the amount of autonomy residents do achieve is sufficient to ensure readiness for the entire spectrum of independent practice.

The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial.

PURPOSE: Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. METHODS: Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. RESULTS: A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. CONCLUSIONS: SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

Using just-in-time teaching and peer instruction in a residency program's core curriculum: enhancing satisfaction, engagement, and retention.

PURPOSE: To assess use of the combined just-in-time teaching (JiTT) and peer instruction (PI) instructional strategy in a residency program's core curriculum. METHOD: In 2010-2011, JiTT/PI was piloted in 31 core curriculum sessions taught by 22 faculty in the Northwestern University Feinberg School of Medicine's general surgery residency program. JiTT/PI required preliminary and categorical residents (n=31) to complete Web-based study questions before weekly specialty topic sessions. Responses were examined by faculty members "just in time" to tailor session content to residents' learning needs. In the sessions, residents answered multiple-choice questions (MCQs) using clickers and engaged in PI. Participants completed surveys assessing their perceptions of JiTT/PI. Videos were coded to assess resident engagement time in JiTT/PI sessions versus prior lecture-based sessions. Responses to topic session MCQs repeated in review sessions were evaluated to study retention. RESULTS: More than 70% of resident survey respondents indicated that JiTT/PI aided in the learning of key points. At least 90% of faculty survey respondents reported positive perceptions of aspects of the JiTT/PI strategy. Resident engagement time for JiTT/PI sessions was significantly greater than for prior lecture-based sessions (z=-2.4, P=.016). Significantly more review session MCQ responses were correct for residents who had attended corresponding JiTT/PI sessions than for residents who had not (chi-square=13.7; df=1; P<.001). CONCLUSIONS: JiTT/PI increased learner participation, learner retention, and the amount of learner-centered time. JiTT/PI represents an effective approach for meaningful and active learning in core curriculum sessions.

Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance.

PURPOSE: The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy. METHODS: Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated. RESULTS: Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high. CONCLUSIONS: The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

Defining the autonomy gap: when expectations do not meet reality in the operating room.

OBJECTIVE: To develop operative independence with essential procedures by the end of their training, residents need graded autonomy as they progress through training. This study compares autonomy expectations, as defined by faculty and residents, with autonomy measured in the operating room. METHODS: Operative procedures performed by general surgery residents between November 2012 and June 2013 were each assigned an autonomy score by the operating attending physician using a previously described rating scale (Zwisch). Scores range from minimum autonomy, "show and tell," to maximum autonomy, "supervision only." Autonomy expectations were defined by a survey asking faculty and residents what autonomy-level residents should achieve during each year of training for each of the 10 most commonly performed procedures. Faculty expectations, resident expectations, and actual operating room autonomy data were compared using analysis of variance with post hoc analysis by Tukey honestly significant difference test. RESULTS: A total of 1467 operative cases were scored using the Zwisch scale over the period of the study. The 10 most common procedures accounted for 56.3% (827) of the cases. Resident and faculty expectations of resident operative autonomy were similar. For only laparoscopic cholecystectomy, residents expected significantly more autonomy than the faculty did during the junior years but they agreed with the faculty for the chief year. When expectations were compared with actual performance, the resident autonomy level achieved was significantly less than that expected by residents or faculty or both for all 10 procedures in at least one postgraduate level. For every procedure performed more than 5 times during the study period by postgraduate years 3 to 5 residents, autonomy was significantly less than expected. CONCLUSIONS: Surgical faculty and residents had similar expectations for resident operative autonomy, yet actual resident performance failed to achieve those shared expectations for even the most common procedures. This autonomy gap provides more evidence for concerns about the preparedness of graduating residents for independent practice.