Improving resident performance assessment data: numeric precision and narrative specificity.

PURPOSE: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. METHOD: Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. RESULTS: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). CONCLUSIONS: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

Does a general surgery clerkship influence student perceptions of surgeons and surgical careers?

BACKGROUND: Student perceptions of surgeons and surgical careers may influence their decision to pursue a surgical career. We evaluated the impact of a general surgery clerkship on medical student perceptions of surgeons and surgical careers and of the clerkship on student interest in surgical careers. METHODS: We conducted voluntary pre- and postclerkship surveys of third-year medical students who were enrolled in their required general surgery clerkship and used 5-point Likert scale rankings to capture agreement with declarative statements (1 = strongly agree; 5 = strongly disagree). T-tests were used to evaluate both paired and individual items. RESULTS: Students indicated that their surgical clerkship improved their opinion of surgeons (2.47; P <.001). Several perceptions changed after the clerkship. First, students agreed more strongly that surgeons were compassionate physicians (ranking, 2.87 vs 2.53; P =.003) and that patients respected surgeons (ranking, 1.84 vs 1.62; P =.026). Acknowledgment of career satisfaction by students increased (ranking, 2.57 vs 2.22; P =.008). Students more strongly disagreed that "surgeons [were] respectful of other physicians" (ranking, 3.29 vs 3.62; P =.009). Interest in surgical careers did not change significantly during the clerkship (ranking, 2.83 vs 2.68; P =.218). CONCLUSIONS: Medical student perceptions of surgeons and surgical careers generally improve during the surgical clerkship. However, student impressions of surgeons' collegial behavior and commitment to teaching deteriorate significantly during the surgical clerkship.

Do peer chart audits improve residents' performance in providing preventive care?

PURPOSE: One recommended method to evaluate residents' competence in practice-based learning and improvement is chart audit. This study determined whether residents improved in providing preventive care after a peer chart audit program was initiated. METHOD: Residents audited 1,005 charts and scored their peers on 12 clinical preventive services. The mean total chart audit scores were compared across five time blocks of the 45-month study. RESULTS: Residents' performance in providing preventive care initially improved significantly but declined in the last ten months. However, their performance remained significantly higher than at the beginning. CONCLUSIONS: By auditing their peers' charts, residents improved their own performance in providing preventive care. The diffusion of innovations theory may explain the prolonged implementation phase and problems in maintaining a chart audit program.

An encounter card system for increasing feedback to students.

BACKGROUND: Student satisfaction with the feedback process is essential for motivating students to improve their performance. METHODS: Third-year medical students participated in a system to increase feedback from faculty, fellows, and residents during a 12-week surgery clerkship. Each student received 40 encounter cards to solicit verbal and written feedback on specific domains of clinical performance. The clerkship evaluations from the intervention group (Fall 2000-2001) were compared with a historical control group (Fall 1999-2000) on seven feedback-related items. Effect sizes were calculated and two-tailed t tests performed to determine statistical significance. RESULTS: All seven items had effect sizes greater than 0.30 (range, 0.38 to 0.85). Differences between the two groups' scores were statistically significant for six items. CONCLUSIONS: Implementation of an encounter card system improved student satisfaction with the feedback process. Intervention group students perceived that faculty and residents had improved their understanding of history-taking, physical-examination, and decision-making skills.

How medical students define surgical mentors.

BACKGROUND: Current literature does not define ideal qualities of surgical mentors. We sought to define mentoring qualities desired by medical students in attending and resident surgeons. METHODS: We conducted a survey asking third-year medical students to identify the best attending surgeon and resident surgeon mentors during their surgical clerkship and to explain their selections. Comments were systematically evaluated using content analysis. RESULTS: The survey response rate was 94.8%. Attending surgeon mentors were described by 84 students; 70 students described resident surgeon mentors. Students most frequently described the "teacher" role for attending surgeons. For resident surgeon mentors, students most often ranked the "person" role. Students prioritized characteristics or attributes within roles differently for attending and resident surgeon mentors. CONCLUSIONS: Medical students' expectations of attending and resident surgeon mentors differ. Emphasis on behaviors consonant with identified characteristics may improve mentoring of medical students by surgeons and increase interest in surgical careers.

Surgical intern survival skills curriculum as an intern: does it help?

BACKGROUND: The transition from medical student to surgical intern is fraught with anxiety. We implemented a surgical intern survival skills curriculum to alleviate this through a series of lectures and interactive sessions. The purpose of this pilot study was to evaluate its effectiveness. METHODS: This was a prospective observational pilot study of our surgical intern survival skills curriculum, the components of which included professionalism, medical documentation, pharmacy highlights, radiographic interpretations, nutrition, and mock clinical pages. The participants completed pre-course and post-course surveys to assess their confidence levels in the elements addressed using a 5-point Likert scale (1 = unsatisfactory, 5 = excellent). A P value of less than .05 was considered significant. RESULTS: In 2009, 8 interns participated in the surgical intern survival skills curriculum. Fifty percent were female and their mean age was 27.5 ± 1.5 years. Of 33 elements assessed, interns rated themselves as more confident in 27 upon completion of the course. CONCLUSIONS: The implementation of a surgical intern survival skills curriculum significantly improved the confidence levels of general surgery interns and seemed to ease the transition from medical student to surgical intern.

How accurate is the Accreditation Council for Graduate Medical Education (ACGME) Resident survey? Comparison between ACGME and in-house GME survey.

PURPOSE: Increasing importance is being assigned to the annual Accreditation Council for Graduate Medical Education (ACGME) Resident/Fellow survey. In certain circumstances, the survey has prompted site visits for programs with significant areas of noncompliance. However, the dichotomous "yes/no" responses available for most questions on the ACGME survey limit the range of resident responses. Our Graduate Medical Education (GME) department administers an annual survey similar in content to the ACGME survey but with answers using a 5-point Likert scale. The purpose of the current study was to compare the responses obtained on the ACGME survey with our in-house GME survey. METHODS: Responses to the 2009 ACGME and GME surveys were compared among 26 general surgery residents from a single institution. Fifteen questions assessed similar information on both surveys; these questions related to faculty support, rotation/program evaluation, work and educational environment, and duty-hour compliance. RESULTS: Of the 15 questions compared on the 2 surveys, significant differences were found in the following areas: faculty time spent teaching, interference of other trainees, and rotation emphasis of education over other concerns. In each instance, resident responses on the ACGME survey were less favorable compared with the responses elicited on the GME survey. CONCLUSIONS: The results of the current study suggest that responses obtained on the ACGME survey may inaccurately reflect the magnitude of noncompliance found in certain areas. We propose that this discrepancy may be a result of the limited range of responses available on the ACGME survey.

How accurate are self-reported resident duty hours?

PURPOSE: The Methodist Hospital monitors resident duty hours using self-report for external rotations and an electronic time clock system for internal rotations where residents use their identification badges to "swipe in and out." This study was conducted to compare the accuracy of self-reports against the electronic system. METHODS: This was a prospective observational study. For 1 month, surgical residents in an academic training program self-reported their duty hours and used the electronic system. The primary outcome measure was the accuracy of self-reported duty hours. RESULTS: Twenty two surgical residents accounted for 450 individual duty-hour periods. Sixty-four percent of the residents were men, and the distribution by postgraduate year (PGY) was PGY1 27%, PGY2 27%, PGY3 14%, PGY4 14%, and PGY5 18%. The number of missing duty-hour period reports was significantly greater for self-reports (44/450) than for the electronic system (18/450), p < 0.001. There were no statistically significant differences between either reporting method in regard to total number of duty-hour violations and individual duty-hour violations. CONCLUSION: Self-report was as accurate as the electronic system in determining the occurrence of duty-hour violations. Because residents may be able to manipulate reporting in both systems, the possibility of inaccuracies exists.

Reliability and usefulness of clinical encounter cards for a third-year surgical clerkship.

BACKGROUND: Third-year medical students (MS3) were given clinical encounter cards (CECs) to stimulate feedback during their surgery clerkship. This study analyzes the feedback given on their clinical performance using CECs. METHODS: Two hundred one students enrolled in the 12-week surgery clerkship were given CECs. Each card contained the chief focus of the encounter, which was rated on a six-point Likert scale. The CECs were analyzed to determine if they provided reliable formative information, identified marginal performances, and identified differences between raters, settings, rotations, and aspects of clinical performance evaluated. RESULTS: Seven thousand three hundred eight CECs were submitted from 201 students. The CECs were completed most often (65%) in the inpatient setting. Technical skills were evaluated on 49% of CECs, history/physical examinations on 40%, and case presentations on 30%. There were comments written on half of the CECs and 89% of these were strictly positive. Women (52.8%) and faculty (63.3%) were more likely to provide written comments. The students were most likely to seek feedback from the interns and faculty who gave significantly higher ratings. The Cronbach-alpha reliability coefficient was 0.69, based on 12 raters per student. There was a significant positive correlation (P < 0.05) between the CEC composite rating and the clinical performance rating (r = 0.356), NBME score (r = 0.258), and the final grade (r = 0.250). CONCLUSIONS: The implementation of CECs in a surgery clerkship provided a large quantity of positive feedback. The quality was limited because there were minimal recommendations for improvement and they were a poor predictor of overall clinical performance.

Factors influencing outcome of the American Board of Surgery Certifying examination: an observational study.

BACKGROUND: Because of the content of the American Board of Surgery (ABS) certifying (oral) examination, there is a perception that those in some subspecialty surgical training programs at the time of the examination may have a lower pass rate. In addition, the format of the oral examination has prompted the use of specialized preparation such as "mock orals" and commercial courses. The purpose of this study was to correlate the pass rates on the ABS certifying examination with the practice status and methods of specialized preparation. MATERIALS AND METHODS: A survey covering demographic information, type of surgical practice at the time of the examination, methods of preparation, and results of the examination was distributed to 1997 and 1998 graduates via a request to surgical residency program directors. RESULTS: One hundred one of 268 program directors supplied 717 names. There were 465 responses. Surveys distributed by the other 167 program directors resulted in an additional 81 responses. Four hundred ten (75%) of the respondents had taken the certifying examination. The total pass rate was 91%. There were no significant differences in the pass rate between those in private practice general surgery; those in academic general surgery; Thoracic, Vascular, or Plastic Surgery Fellows; those in other surgical fellowships; and those in the military or research. No significant differences in the pass rates were noted between those who prepared with formal mock orals, with informal mock orals, with a commercial course, with combinations of the three, and with no specialized preparation. CONCLUSION: Performance on the ABS certifying examination was not influenced by the candidate's practice status at the time of the examination. A substantial percentage of examinees either are exposed to or perceive the need to pursue specialized preparation for the examination, a behavior that in general produces good results.