Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment.

OBJECTIVE: We have previously demonstrated the feasibility and validity of a smartphone-based system called Procedural Autonomy and Supervision System (PASS), which uses the Zwisch autonomy scale to facilitate assessment of the operative performances of surgical residents and promote progressive autonomy. To determine whether the use of PASS in a general surgery residency program is associated with any negative consequences, we tested the null hypothesis that PASS implementation at our institution would not negatively affect resident or faculty satisfaction in the operating room (OR) nor increase mean OR times for cases performed together by residents and faculty. METHODS: Mean OR times were obtained from the electronic medical record at Northwestern Memorial Hospital for the 20 procedures most commonly performed by faculty members with residents before and after PASS implementation. OR times were compared via two-sample t-test. The OR Educational Environment Measure tool was used to assess OR satisfaction with all clinically active general surgery residents (n = 31) and full-time general surgery faculty members (n = 27) before and after PASS implementation. Results were compared using the Mann-Whitney rank sum test. RESULTS: A significant prolongation in mean OR time between control and study period was found for only 1 of the 20 operative procedures performed at least 20 times by participating faculty members with residents. Based on the overall survey score, no significant differences were found between resident and faculty responses to the OR Educational Environment Measure survey before and after PASS implementation. When individual survey items were compared, while no differences were found with resident responses, differences were noted with faculty responses for 7 of the 35 items addressed although after Bonferroni correction none of these differences remained significant. CONCLUSIONS: Our data suggest that PASS does not increase mean OR times for the most commonly performed procedures. Resident OR satisfaction did not significantly change during PASS implementation, whereas some changes in faculty satisfaction were noted suggesting that PASS implementation may have had some negative effect with them. Although the effect on faculty satisfaction clearly requires further investigation, our findings support that use of an autonomy-based OR performance assessment system such as PASS does not appear to have a major negative influence on OR times nor OR satisfaction.

The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial.

PURPOSE: Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. METHODS: Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. RESULTS: A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. CONCLUSIONS: SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance.

PURPOSE: The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy. METHODS: Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated. RESULTS: Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high. CONCLUSIONS: The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

Development and verification of a taxonomy of assessment metrics for surgical technical skills.

PURPOSE: To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. METHOD: In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. RESULTS: The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. CONCLUSIONS: Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

Development and participant assessment of a practical quality improvement educational initiative for surgical residents.

BACKGROUND: As patient-safety and quality efforts spread throughout health care, the need for physician involvement is critical, yet structured training programs during surgical residency are still uncommon. Our objective was to develop an extended quality-improvement curriculum for surgical residents that included formal didactics and structured practical experience. METHODS: Surgical trainees completed an 8-hour didactic program in quality-improvement methodology at the start of PGY3. Small teams developed practical quality-improvement projects based on needs identified during clinical experience. With the assistance of the hospital's process-improvement team and surgical faculty, residents worked through their selected projects during the following year. Residents were anonymously surveyed after their participation to assess the experience. RESULTS: During the first 3 years of the program, 17 residents participated, with 100% survey completion. Seven quality-improvement projects were developed, with 57% completing all DMAIC (Define, Measure, Analyze, Improve, Control) phases. Initial projects involved issues of clinical efficiency and later projects increasingly focused on clinical care questions. Residents found the experience educationally important (65%) and believed they were well equipped to lead similar initiatives in the future (70%). Based on feedback, the timeline was expanded from 12 to 24 months and changed to start in PGY2. CONCLUSIONS: Developing an extended curriculum using both didactic sessions and applied projects to teach residents the theory and implementation of quality improvement is possible and effective. It addresses the ACGME competencies of practice-based improvement and learning and systems-based practice. Our iterative experience during the past 3 years can serve as a guide for other programs.

Career development resource: educational leadership in a department of surgery: vice chairs for education.

The growing appreciation of the need to adopt an evidence-based approach to teaching and assessment has led to a demand for faculty who are well versed in best practices in education. Surgeons with interest and expertise in instruction, curriculum development, educational research, and evaluation can have an important impact on the educational mission of a department of surgery. The increased fervor for accountability in education together with the challenges imposed by accreditation agencies and hospitals has made educational leadership responsibilities more time consuming and complex. In response to this, an increasing number of department chairs created Vice Chair for Education positions to support clerkship and program directors and ensure the department's education mission statement is fulfilled.

Needs assessment for an errors-based curriculum on thoracoscopic lobectomy.

BACKGROUND: Research suggests a benefit from a skills curriculum emphasizing error prevention, identification, and management. Our purpose was to identify common errors committed by trainees during simulated thoracoscopic lobectomy for use in developing an error-based curriculum. METHODS: Twenty-one residents (postgraduate years 1 to 8) performed a thoracoscopic left upper lobectomy on a previously validated simulator. Videos of the procedure were reviewed in a blinded fashion using a checklist listing 66 possible cognitive and technical errors. RESULTS: Of the 21 residents, 15 (71%) self-reported completing the anatomic lobectomy; however, only 7 (33%) had actually divided all of the necessary structures correctly. While dissecting the superior pulmonary vein, 16 residents (76%) made at least one error. The most common (n=13, 62%) was dissecting individual branches rather than the entire vein. On the bronchus, 14 (67%) made at least one error. Again, the most common (n=9, 43%) was dissecting branches. During these tasks, cognitive errors were more common than technical errors. While dissecting arterial branches, 18 residents (86%) made at least one error. Technical and cognitive errors occurred with equal frequency during arterial dissection. The most common arterial error was excess tension on the vessel (n=10, 48%). CONCLUSIONS: Curriculum developers should identify skill-specific technical and judgment errors to verify the scope of errors typically committed. For a thoracoscopic lobectomy curriculum, emphasis should be placed on correct identification of anatomic landmarks during dissection of the vein and airway and on proper tissue handling technique during arterial dissection.

Intra-operative decision making: more than meets the eye.

Operating room teams consist of team members with diverse training backgrounds. In addition to differences in training, each team member has unique and complex decision making paths. As such, team members may function in the same environment largely unaware of their team members' perspectives. The goal of our work was to use a theory-based approach to better understand the complexity of knowledge-based intra-operative decision making. Cognitive task analysis methods were used to extract the knowledge, thought processes, goal structures and critical decisions that provide the foundation for surgical task performance. A triangulated and iterative approach is presented.

Administrative considerations when implementing ACS/APDS Skills Curriculum.

BACKGROUND: With time and cost constraints, implementing an effective, yet efficient, skills curriculum poses significant challenges. Our purpose is to describe a successful curriculum administrative structure that promoted faculty buy-in and accountability, learner responsibility, and acceptable resource usage. METHODS: A total of 14 American College of Surgery (ACS) modules were included in the postgraduate year 1 curriculum. Before arrival, 2 modules were sent to newly matched residents. Remaining modules were administered over a 4-month period, with integrated, independent practice opportunities, as well as 4 mentored and 1 peer practice sessions. A total of 2 verifications of proficiency (VOP) progress exams and 1 final comprehensive VOP were administered. To promote faculty ownership, 1 faculty member was asked to lead each module. Module leaders attended an orientation and development session, and created an instructional management plan. Each module was taught by the leader and 2 additional faculty coinstructors, and evaluated by residents. Equipment, resource costs, and man-hours were tracked. RESULTS: Faculty buy-in was demonstrated by enthusiastic participation, with only 2 absences. Residents gave high ratings to all the modules (range, 4.22-4.89/5). Curriculum costs were approximately $21,500, reduced from potential costs of $187,000 if all simulators would have been purchased new. The estimated budget for year 2 is $17,000. CONCLUSION: It is critical for new curricula to have resident and faculty buy-in, accountability for quality teaching and learning, and reasonable resource use. We provide suggestions for structuring a curriculum to ensure accomplishment of these important drivers.

Development of an online morbidity, mortality, and near-miss reporting system to identify patterns of adverse events in surgical patients.

OBJECTIVES: To design a Web-based system to track adverse and near-miss events, to establish an automated method to identify patterns of events, and to assess the adverse event reporting behavior of physicians. DESIGN: A Web-based system was designed to collect physician-reported adverse events including weekly Morbidity and Mortality (M&M) entries and anonymous adverse/near-miss events. An automated system was set up to help identify event patterns. Adverse event frequency was compared with hospital databases to assess reporting completeness. SETTING: A metropolitan tertiary care center. MAIN OUTCOME MEASURES: Identification of adverse event patterns and completeness of reporting. RESULTS: From September 2005 to August 2007, 15,524 surgical patients were reported including 957 (6.2%) adverse events and 34 (0.2%) anonymous reports. The automated pattern recognition system helped identify 4 event patterns from M&M reports and 3 patterns from anonymous/near-miss reporting. After multidisciplinary meetings and expert reviews, the patterns were addressed with educational initiatives, correction of systems issues, and/or intensive quality monitoring. Only 25% of complications and 42% of inpatient deaths were reported. A total of 75.2% of adverse events resulting in permanent disability or death were attributed to the nature of the disease. Interventions to improve reporting were largely unsuccessful. CONCLUSIONS: We have developed a user-friendly Web-based system to track complications and identify patterns of adverse events. Underreporting of adverse events and attributing the complication to the nature of the disease represent a problem in reporting culture among surgeons at our institution. Similar systems should be used by surgery departments, particularly those affiliated with teaching hospitals, to identify quality improvement opportunities.

Patient assessment and management examination: lack of correlation between faculty assessment and resident self-assessment.

BACKGROUND: The Patient Assessment and Management Examination (PAME) is a standardized patient examination designed to assess management skills of senior residents. This study explored the relationship between faculty and resident self-evaluation by using PAME. METHODS: Nine postgraduate year (PGY) 4 and PGY5 residents were examined with a 5 case PAME. Faculty rated interactions between residents and standardized patients and residents rated themselves based on review of audio-video recordings of their interactions. We examined correlations between faculty and resident self-assessments. RESULTS: Faculty and resident ratings of physical examination skills was the only competency that correlated significantly. Correlations were not significant for the other 15 competencies (Pearson r, -.197 to .262). Correlation was no better when examined within each case. CONCLUSIONS: Although PAME may be a useful tool, this study suggests that even senior residents do not assess their performance as clinicians similarly to faculty. Further research is needed to better understand the source of these disagreements.

Current assessment and future directions of surgical skills laboratories.

BACKGROUND: Educational, medicolegal, and financial constraints have pushed surgical residency programs to find alternative methods to operating room teaching for surgical skills training. Several studies have demonstrated that the use of skills laboratories is effective and enhances performance; however, little is known about the facilities available to residents. STUDY DESIGN: A survey was distributed to 40 general surgery program directors who, in an earlier questionnaire, indicated that they had skills laboratory facilities at their institutions. The survey included the following sections: demographics, facilities, administrative infrastructure, curriculum, learners, and opinions/thoughts of program directors. RESULTS: Of the 34 program directors that completed the survey, 76% are from a university program. The average facility is 1400 square feet, and most skills laboratories are located in the hospital. Nearly all skills facilities have dry laboratories (90%), and the most common equipment is box trainers (90%). Average start-up costs were $450,000. Sixty-two percent of programs have a skills curriculum for residents. Responders agreed that skills laboratories have a high value and should be part of residency curricula. CONCLUSIONS: The results of this survey provide a preliminary view of skills laboratories. There is variation in the size, location, and availability of simulators in skills laboratory facilities. Variations also exist in types of curricula formats, subspecialties who make use of the laboratory, and some administrative approaches. There is strong agreement among respondents that skills laboratories are a necessary and valuable component of residency education. Results also indicated concerns for recruiting faculty to teach in the skills laboratory, securing ongoing funding, and implementing a skills laboratory curriculum.

Improving resident performance assessment data: numeric precision and narrative specificity.

PURPOSE: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. METHOD: Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. RESULTS: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). CONCLUSIONS: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

How do residents manage personal finances?

BACKGROUND: We examined three research questions: How do residents' debts and savings compare to the general public? How do surgical residents' financial choices compare to other residents? How may institutions help residents' personal financial decisions? METHODS: The Survey of Consumer Finances was modified and piloted tested to elicit financial information. The instrument was completed by 612 residents at 8 programs. RESULTS: Only 60% of residents budgeted expenses, and 25% and 10% maintained cash balances <611 dollars and unpaid credit card balances >10,000 dollars, respectively. Compared with controls, residents held greater median ratios of debt to household income (2.46 vs. 1.06, P <0.0001), fewer assets to income (0.64 vs. 2.28, P <0.0001), less net wealth to income -1.43 vs. 0.90, P <0.0001), and lower retirement savings balance to household income (0.01 vs. 0.12, P <0.0001). Surgery residents were the least financially conservative group. Mean annual resident contributions to retirement accounts were $1532 higher at institutions with versus without retirement plans (P <0.01). CONCLUSIONS: Resident debts are higher and savings lower than the general public. This behavior is most common among surgery residents. Residents save more for retirement when they are eligible for tax-deferred retirement plans. Graduate medical programs should instruct residents on financial management.

Developing an ethics curriculum for an internal medicine residency program: use of a needs assessment.

BACKGROUND: Residency programs are required to teach and evaluate trainees in the area of professionalism and medical ethics. Prior to developing a curriculum in this area, residents and fellows were surveyed to assess learning needs. DESCRIPTION: A case-based survey was developed based on published curricula. Residents and fellows were asked to describe their comfort level in 11 clinical scenarios on a Likert-type scale ranging from 1 (not at all comfortable) to 10 (extremely comfortable). EVALUATION: 151 surveys were returned for an overall response rate of 73%. Comfort levels ranged from a low of 3.1 to a high of 8.5 on the 10-point scale. Despite additional years of clinical training, fellows only reported an increased comfort level in 1 case. CONCLUSION: Learning needs exist in residents and fellows in the area of medical ethics. Use of a needs assessment was instrumental in planning and designing an ethics curriculum.