The APGO Academic Scholars and Leaders Program: Supporting obstetrics and gynecology faculty development for 20 years.

PURPOSE: We evaluate the impacts of the Academic Scholars and Leaders (ASL) Program in achieving 3 key objectives: treatment of education as a scholarly pursuit, improved education leadership, and career advancement. MATERIALS AND METHODS: We report on the twenty-year experience of the ASL Program-a national, longitudinal faculty development program of the Association of Professors of Obstetrics and Gynecology (APGO) covering instruction, curriculum development/program evaluation, assessment/feedback, leadership/professional development, and educational scholarship. We conducted a cross-sectional, online survey of ASL participants who graduated in 1999-2017. We sought evidence of impact using Kirkpatrick's 4-level framework. Descriptive quantitative data were analyzed, and open-ended comments were organized using content analysis. RESULTS: 64% (260) of graduates responded. The vast majority (96%) felt the program was extremely worthwhile (Kirkpatrick level 1). Graduates cited learned skills they had applied to their work, most commonly curricular development (48%) and direct teaching (38%) (Kirkpatrick 2&3 A). Since participation, 82% of graduates have held institutional, education-focused leadership roles (Kirkpatrick 3B). Nineteen percent had published the ASL project as a manuscript and 46% additional education papers (Kirkpatrick 3B). CONCLUSIONS: The APGO ASL program has been associated with successful outcomes in treatment of education as a scholarly pursuit, education leadership, and career advancement. Going forward, APGO is considering ways to diversify the ASL community and to support educational research training.

Readiness of US General Surgery Residents for Independent Practice.

OBJECTIVE: This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy. BACKGROUND: The American Board of Surgery has designated 132 procedures as being "Core" to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate. There is growing concern that not all residents achieve that standard. Lack of operative autonomy may play a role. METHODS: Attendings in 14 General Surgery programs were trained to use a) the 5-level System for Improving and Measuring Procedural Learning (SIMPL) Performance scale to assess resident readiness for independent practice and b) the 4-level Zwisch scale to assess the level of guidance (ie, autonomy) they provided to residents during specific procedures. Ratings were collected immediately after cases that involved a categorical GS resident. Data were analyzed using descriptive statistics and supplemented with Bayesian ordinal model-based estimation. RESULTS: A total of 444 attending surgeons rated 536 categorical residents after 10,130 procedures. Performance: from the first to the last year of training, the proportion of Performance ratings for Core procedures (n = 6931) at "Practice Ready" or above increased from 12.3% to 77.1%. The predicted probability that a typical trainee would be rated as Competent after performing an average Core procedure on an average complexity patient during the last week of residency training is 90.5% (95% CI: 85.7%-94%). This falls to 84.6% for more complex patients and to less than 80% for more difficult Core procedures. Autonomy: for all procedures, the proportion of Zwisch ratings indicating meaningful autonomy ("Passive Help" or "Supervision Only") increased from 15.1% to 65.7% from the first to the last year of training. For the Core procedures performed by residents in their final 6 months of training (cholecystectomy, inguinal/femoral hernia repair, appendectomy, ventral hernia repair, and partial colectomy), the proportion of Zwisch ratings (n = 357) indicating near-independence ("Supervision Only") was 33.3%. CONCLUSIONS: US General Surgery residents are not universally ready to independently perform Core procedures by the time they complete residency training. Progressive resident autonomy is also limited. It is unknown if the amount of autonomy residents do achieve is sufficient to ensure readiness for the entire spectrum of independent practice.

Teaching for understanding in medical classrooms using multimedia design principles.

OBJECTIVES: In line with a recent report entitled Effective Use of Educational Technology in Medical Education from the Association of American Medical Colleges Institute for Improving Medical Education (AAMC-IME), this study examined whether revising a medical lecture based on evidence-based principles of multimedia design would lead to improved long-term transfer and retention in Year 3 medical students. A previous study yielded positive effects on an immediate retention test, but did not investigate long-term effects. METHODS: In a pre-test/post-test control design, a cohort of 37 Year 3 medical students at a private, midwestern medical school received a bullet point-based PowerPoint™ lecture on shock developed by the instructor as part of their core curriculum (the traditional condition group). Another cohort of 43 similar medical students received a lecture covering identical content using slides redesigned according to Mayer's evidence-based principles of multimedia design (the modified condition group). RESULTS: Findings showed that the modified condition group significantly outscored the traditional condition group on delayed tests of transfer given 1 week (d = 0.83) and 4 weeks (d = 1.17) after instruction, and on delayed tests of retention given 1 week (d = 0.83) and 4 weeks (d = 0.79) after instruction. The modified condition group also significantly outperformed the traditional condition group on immediate tests of retention (d = 1.49) and transfer (d = 0.76). CONCLUSIONS: This study provides the first evidence that applying multimedia design principles to an actual medical lecture has significant effects on measures of learner understanding (i.e. long-term transfer and long-term retention). This work reinforces the need to apply the science of learning and instruction in medical education.

Overcoming obstacles to resident-patient continuity of care.

OBJECTIVE: Because continuity of care (CC) is a necessary component of resident education, this analysis was done to understand what keeps CC between residents and patients low and how it can be most effectively improved. BACKGROUND: Many authors lament low CC between residents and patients, especially in the era of duty hour regulations. Some have tried lengthening rotations, some have tried increasing clinic attendance, and some have argued for various training models. Little detailed analysis has been done to identify root causes of low CC or ways to improve it. METHODS: Two months of charts were reviewed to estimate baseline CC on a vascular surgery rotation. Probability theory and engineering simulations were used to determine whether CC can be enhanced by (a) lengthening rotations, (b) altering observed logistical patterns, (c) using a "resident return" model where residents are able to see patients postoperatively even if moved to a different rotation, or (d) employing an apprenticeship model. RESULTS: Baseline analysis showed residents had 0% CC given 131 opportunities to do so. Probability analysis and the simulation outcomes suggest that rotation length plays a minor role in achieving CC. Logistical changes showed some improvement in CC, but not as much as using an apprenticeship rotation model. CONCLUSIONS: The limitations placed on CC by rotation duration are real, but lengthening the rotation does not meaningfully resolve the gap between acceptable CC levels and actual levels. Although CC can be enhanced with longer rotations if coupled with the use of the resident return model, the greater barrier to CC is the logistical patterns such as where residents spend time, how cases are assigned, and the lack of an alert system to inform residents about returning postoperative patients. The apprenticeship model enables residents to achieve CC closer to that of the faculty.

Intra-operative decision making: more than meets the eye.

Operating room teams consist of team members with diverse training backgrounds. In addition to differences in training, each team member has unique and complex decision making paths. As such, team members may function in the same environment largely unaware of their team members' perspectives. The goal of our work was to use a theory-based approach to better understand the complexity of knowledge-based intra-operative decision making. Cognitive task analysis methods were used to extract the knowledge, thought processes, goal structures and critical decisions that provide the foundation for surgical task performance. A triangulated and iterative approach is presented.

Applying multimedia design principles enhances learning in medical education.

CONTEXT: The Association of American Medical Colleges' Institute for Improving Medical Education's report entitled 'Effective Use of Educational Technology' called on researchers to study the effectiveness of multimedia design principles. These principles were empirically shown to result in superior learning when used with college students in laboratory studies, but have not been studied with undergraduate medical students as participants. METHODS: A pre-test/post-test control group design was used, in which the traditional-learning group received a lecture on shock using traditionally designed slides and the modified-design group received the same lecture using slides modified in accord with Mayer's principles of multimedia design. Participants included Year 3 medical students at a private, midwestern medical school progressing through their surgery clerkship during the academic year 2009-2010. The medical school divides students into four groups; each group attends the surgery clerkship during one of the four quarters of the academic year. Students in the second and third quarters served as the modified-design group (n=91) and students in the fourth-quarter clerkship served as the traditional-design group (n=39). RESULTS: Both student cohorts had similar levels of pre-lecture knowledge. Both groups showed significant improvements in retention (p<0.0001), transfer (p<0.05) and total scores (p<0.0001) between the pre- and post-tests. Repeated-measures anova analysis showed statistically significant greater improvements in retention (F=10.2, p=0.0016) and total scores (F=7.13, p=0.0081) for those students instructed using principles of multimedia design compared with those instructed using the traditional design. CONCLUSIONS: Multimedia design principles are easy to implement and result in improved short-term retention among medical students, but empirical research is still needed to determine how these principles affect transfer of learning. Further research on applying the principles of multimedia design to medical education is needed to verify the impact it has on the long-term learning of medical students, as well as its impact on other forms of multimedia instructional programmes used in the education of medical students.

Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment.

OBJECTIVE: We have previously demonstrated the feasibility and validity of a smartphone-based system called Procedural Autonomy and Supervision System (PASS), which uses the Zwisch autonomy scale to facilitate assessment of the operative performances of surgical residents and promote progressive autonomy. To determine whether the use of PASS in a general surgery residency program is associated with any negative consequences, we tested the null hypothesis that PASS implementation at our institution would not negatively affect resident or faculty satisfaction in the operating room (OR) nor increase mean OR times for cases performed together by residents and faculty. METHODS: Mean OR times were obtained from the electronic medical record at Northwestern Memorial Hospital for the 20 procedures most commonly performed by faculty members with residents before and after PASS implementation. OR times were compared via two-sample t-test. The OR Educational Environment Measure tool was used to assess OR satisfaction with all clinically active general surgery residents (n = 31) and full-time general surgery faculty members (n = 27) before and after PASS implementation. Results were compared using the Mann-Whitney rank sum test. RESULTS: A significant prolongation in mean OR time between control and study period was found for only 1 of the 20 operative procedures performed at least 20 times by participating faculty members with residents. Based on the overall survey score, no significant differences were found between resident and faculty responses to the OR Educational Environment Measure survey before and after PASS implementation. When individual survey items were compared, while no differences were found with resident responses, differences were noted with faculty responses for 7 of the 35 items addressed although after Bonferroni correction none of these differences remained significant. CONCLUSIONS: Our data suggest that PASS does not increase mean OR times for the most commonly performed procedures. Resident OR satisfaction did not significantly change during PASS implementation, whereas some changes in faculty satisfaction were noted suggesting that PASS implementation may have had some negative effect with them. Although the effect on faculty satisfaction clearly requires further investigation, our findings support that use of an autonomy-based OR performance assessment system such as PASS does not appear to have a major negative influence on OR times nor OR satisfaction.

Implementation and evaluation of a new surgical residency model.

BACKGROUND: The Accreditation Council for Graduate Medical Education (ACGME) duty-hour requirements prompted program directors to rethink the organizational structure of their residency programs. Many surgical educators have expressed concerns that duty-hour restrictions would negatively affect quality of resident education. This article summarizes evaluation research results collected to study the impact of our reengineered residency program designed to preserve important educational activities while meeting duty-hour accreditation requirements. STUDY DESIGN: The traditional residency structure was redesigned to include a mixture of apprenticeship, small team, and night-float models. Impact evaluation data were collected using operative case logs, standardized test scores, quality assurance data, resident perception surveys, a faculty survey, and process evaluation measures. RESULTS: PGY1s and PGY2s enjoyed a substantial increase in operative cases. Operative cases increased overall and no resident has failed to meet ACGME volume or distribution requirements. American Board of Surgery In-Training Examination performance improved for PGY1s and PGY2s. Patient outcomes measures, including monthly mortality and number of and charges for admissions, showed no changes. Anonymously completed rotation evaluation forms showed stable or improved resident perceptions of case load, continuity, operating room teaching, appropriate level of faculty involvement and supervision, encouragement to attend conferences, and general assessment of the learning environment. A quality-of-life survey completed by residents before and after implementation of the new program structure showed substantial improvements. Faculty surveys showed perceived increases in work hours and job dissatisfaction. New physician assistant and nurse positions directly attributed to duty-hour restrictions amounted to about 0.2 full-time equivalent per resident. CONCLUSIONS: Duty-hour restrictions produce new challenges and might require additional resources but need not cause a deterioration of surgical residents' educational experience.

The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial.

PURPOSE: Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. METHODS: Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. RESULTS: A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. CONCLUSIONS: SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

Improving resident performance assessment data: numeric precision and narrative specificity.

PURPOSE: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. METHOD: Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. RESULTS: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). CONCLUSIONS: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

Using just-in-time teaching and peer instruction in a residency program's core curriculum: enhancing satisfaction, engagement, and retention.

PURPOSE: To assess use of the combined just-in-time teaching (JiTT) and peer instruction (PI) instructional strategy in a residency program's core curriculum. METHOD: In 2010-2011, JiTT/PI was piloted in 31 core curriculum sessions taught by 22 faculty in the Northwestern University Feinberg School of Medicine's general surgery residency program. JiTT/PI required preliminary and categorical residents (n=31) to complete Web-based study questions before weekly specialty topic sessions. Responses were examined by faculty members "just in time" to tailor session content to residents' learning needs. In the sessions, residents answered multiple-choice questions (MCQs) using clickers and engaged in PI. Participants completed surveys assessing their perceptions of JiTT/PI. Videos were coded to assess resident engagement time in JiTT/PI sessions versus prior lecture-based sessions. Responses to topic session MCQs repeated in review sessions were evaluated to study retention. RESULTS: More than 70% of resident survey respondents indicated that JiTT/PI aided in the learning of key points. At least 90% of faculty survey respondents reported positive perceptions of aspects of the JiTT/PI strategy. Resident engagement time for JiTT/PI sessions was significantly greater than for prior lecture-based sessions (z=-2.4, P=.016). Significantly more review session MCQ responses were correct for residents who had attended corresponding JiTT/PI sessions than for residents who had not (chi-square=13.7; df=1; P<.001). CONCLUSIONS: JiTT/PI increased learner participation, learner retention, and the amount of learner-centered time. JiTT/PI represents an effective approach for meaningful and active learning in core curriculum sessions.

Questioning skills: the effect of wait time on accuracy of medical student responses to oral and written questions.

BACKGROUND: Pauses (wait time) after asking questions in pre-college classes result in improved discussion and answer accuracy. The authors hypothesized that this would extend to medical students. METHOD: Third-year surgery clerks were randomized to three-second or six-second wait times after questions asked of them during a scripted lecture. Students were randomized within each session to answer 21 scripted questions. Students also completed a post-lecture written examination. RESULTS: Correct responses ranged from 17% to 100% for oral and 22% to 100% for written questions. Answer accuracy could not be distinguished between three- and six-second wait times for oral or written questions. CONCLUSIONS: The benefit of increasing wait times from three to six seconds appears not to extend to medical students. This may represent evolution of learning or different learning modes in medical students. Alternatively, maximum benefit may be achieved in medical students with shorter wait times.

A learning prescription permits feedback on feedback.

BACKGROUND: Students consistently identified inadequate feedback as a deficiency in our third-year clerkship. METHODS: We asked students to solicit one faculty and one resident every 2 weeks for written feedback on a "feedback prescription pad." Each prescription requested four comments: two things the student did well and two things the student needs to improve. Students rated feedback using a five-point scale. A three-point categorization scheme was employed to assess the quality of feedback. RESULTS: Students' rating of feedback improved significantly compared with a previous time period (3.5 +/- 1.2 versus 2.6 +/- 1.2, P <0.01). Interrater reliability of our categorization scheme was high (kappa > or =0.75, P <0.01) and demonstrated that only 10% of comments were specific enough to qualify as effective feedback. CONCLUSIONS: Feedback prescription pads were a simple method to facilitate feedback. Although students appreciated feedback, most feedback was inadequate. Faculty development programs to enhance student feedback should be a priority of clinical medical education.

A critical evaluation of the morbidity and mortality conference.

BACKGROUND: This study was designed to evaluate the impact of changes made to our morbidity and mortality (M&M) conference. METHODS: A 23-item survey using corresponding Likert-type scales was created. Faculty and residents were asked to anonymously complete the surveys in June 1999. Based on this information, specific modifications were made to the conference. The same survey was administered to faculty and residents in the Fall of 2000. Analysis was performed using Student t tests. RESULTS: Postsurvey findings showed residents felt eight components improved significantly (P <0.05). Faculty noted nonsignificant improvement in nine survey items and decline in nine items (five unchanged). CONCLUSIONS: Changes in content and structure made to enhance our M&M conference's educational value resulted in significant improvements as perceived by the surgical residents. Interestingly, these changes had only minimal impact on faculty perceptions.

Facilitating academic institutional change: redefining scholarship.

BACKGROUND AND OBJECTIVES: Academic institutions are typically resistant to change. Redefining scholarship is an important issue for academic health care institutions. This study examines the change process at institutions that have attempted to change the definition of scholarship. METHODS: Five medical schools were identified that had recently redesigned their promotion and tenure systems based on expanded definitions of scholarship. Interviews were conducted with a key leader in this effort. The interviews were designed to identify the forces and barriers involved in change, activities designed to secure faculty "buy-in, "factors needed to sustain change, and advice that would help others who might be considering such an effort at their academic health centers. We organized the results of the interviews within a change leadership and management model. RESULTS: The responses to the survey questions fit well into the change model. Many of the responses were felt to be applicable to multiple stages of the change model. CONCLUSIONS: The leaders of change from the study institutions, either by intention or intuition, identified key factors of their change process that fit well with the study model. Change leaders should include plans that follow an established model for institutional change in their strategy to change the definition of scholarship at their institution.

Residents seeking informed consent: Are they adequately knowledgeable?

PURPOSE: The purpose of this study was to determine the extent to which first-year surgical residents are prepared to obtain informed consent from patients. The study was designed to answer the following research questions: 1) Are first-year residents who are asked to obtain informed consent sufficiently knowledgeable about the risks, benefits, and alternatives of the procedures? 2) Can first-year residents accurately answer the questions patients may pose about these procedures? METHODS: First-year residents (n = 18) were asked to list the risks, benefits, and alternatives for open inguinal hernia repair, laparoscopic cholecystectomy, total thyroidectomy, esophagogastrectomy, and abdominal aortic aneurysm repair, assuming the procedures were elective on otherwise healthy individuals. Residents were also asked to answer questions that patients may pose about each of the procedures. The basic minimum risks, benefits, and alternatives to be listed and answers to the questions were validated by asking faculty representing general (n = 6) and vascular (n = 3) surgery to complete the questionnaires. RESULTS: Few residents were able to correctly list all risks, benefits, and alternatives of any of the procedures. Less than one-half of the questions that patients may ask about the procedures were correctly answered. CONCLUSIONS: Even though first-year residents are commonly obtaining consent for surgical procedures, many are unable to provide patients with the correct descriptions of the risks, benefits, and alternatives. Nor were they able to correctly answer common questions. Surgical faculty must take more time to educate first-year residents on the appropriate issues in informed consent for the procedures being performed.

Evaluation of a blended learning surgical skills course for rural surgeons.

BACKGROUND: Rural surgeons have unique learning needs not easily met by traditional continuing medical education courses. METHODS: A multidisciplinary team developed and implemented a skills curriculum focused on leadership and communication, advanced endoscopy, emergency urology, emergency gynecology, facial plastic surgery, ultrasound, and management of fingertip amputations. RESULTS: Twenty-five of 30 (89%) rural surgeons who completed a follow-up course evaluation reported that the knowledge acquired during the course had improved their practice and/or the quality of patient care, particularly by refining commonly used skills and expanding the care options they could offer to their patients. The surgeons reported incorporating changes in their communication and interaction with colleagues. CONCLUSIONS: This course was successful, from participants' perspectives, in providing hands-on mentored training for a variety of skills that reflect the broad scope of practice of surgeons in rural areas. Attendees felt that their participation resulted in important behavior and practice changes.

Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance.

PURPOSE: The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy. METHODS: Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated. RESULTS: Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high. CONCLUSIONS: The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

Defining the autonomy gap: when expectations do not meet reality in the operating room.

OBJECTIVE: To develop operative independence with essential procedures by the end of their training, residents need graded autonomy as they progress through training. This study compares autonomy expectations, as defined by faculty and residents, with autonomy measured in the operating room. METHODS: Operative procedures performed by general surgery residents between November 2012 and June 2013 were each assigned an autonomy score by the operating attending physician using a previously described rating scale (Zwisch). Scores range from minimum autonomy, "show and tell," to maximum autonomy, "supervision only." Autonomy expectations were defined by a survey asking faculty and residents what autonomy-level residents should achieve during each year of training for each of the 10 most commonly performed procedures. Faculty expectations, resident expectations, and actual operating room autonomy data were compared using analysis of variance with post hoc analysis by Tukey honestly significant difference test. RESULTS: A total of 1467 operative cases were scored using the Zwisch scale over the period of the study. The 10 most common procedures accounted for 56.3% (827) of the cases. Resident and faculty expectations of resident operative autonomy were similar. For only laparoscopic cholecystectomy, residents expected significantly more autonomy than the faculty did during the junior years but they agreed with the faculty for the chief year. When expectations were compared with actual performance, the resident autonomy level achieved was significantly less than that expected by residents or faculty or both for all 10 procedures in at least one postgraduate level. For every procedure performed more than 5 times during the study period by postgraduate years 3 to 5 residents, autonomy was significantly less than expected. CONCLUSIONS: Surgical faculty and residents had similar expectations for resident operative autonomy, yet actual resident performance failed to achieve those shared expectations for even the most common procedures. This autonomy gap provides more evidence for concerns about the preparedness of graduating residents for independent practice.