Using an objective structured clinical examination (OSCE) to assess multiple physician competencies in postgraduate training.

BACKGROUND: Competency-based models of medical education require reliable and valid assessment of multiple physician roles. AIMS: To develop and evaluate an objective structured clinical examination (OSCE) designed to assess 7 physician competencies (CanMEDS Roles). METHODS: Twenty four candidates from 4 neonatal-perinatal medicine training programs participated in a 10-station OSCE. Ten 5-point rating scales were developed and used to assess the CanMEDS Roles of Medical Expert, Communicator, Collaborator, Manager, Health Advocate, Scholar and Professional. Three descriptors of performance anchored the ratings. For each station, examiners completed appropriate CanMEDS ratings, a station-specific binary checklist and an overall process-related global rating. Trained standardized patients (SP) and standardized health professionals (SHP) completed rating scales that assessed verbal and non-verbal expression, empathy and coherence as well as the overall global rating. RESULTS: Each station incorporated 3-5 physician Roles. Interstation alpha was 0.80 for checklist scores and 0.88 for examiners' overall global rating. Median interstation alpha for individual CanMEDS ratings was 0.72 (range 0.08-0.91). There were significant correlations between examiner Medical Expert scores and SP/SHP overall global scores and between examiner Communicator scores and 4 SP/SHP assessments of communication skills. Second year trainees' CanMEDS scores for each competency were significantly higher than those of first year trainees (p < 0.05). CONCLUSIONS: The OSCE may be useful as a reliable and valid method of simultaneously assessing multiple physician competencies.

The Professionalism Mini-evaluation Exercise: a preliminary investigation.

BACKGROUND: As the evaluation of professional behaviors has been identified as an area for development, the Professionalism Mini-Evaluation Exercise (P-MEX) was developed using the mini-Clinical Examination Exercise (mini-CEX) format. METHOD: From a set of 142 observable behaviors reflective of professionalism identified at a McGill workshop, 24 were converted into an evaluation instrument modeled on the mini-CEX. This instrument, designed for use in multiple settings, was tested on clinical clerks in medicine, surgery, obstetrics and gynecology, psychiatry, and pediatrics. In all, 211 forms were completed on 74 students by 47 evaluators. RESULTS: Results indicate content and construct validity. Exploratory factor analysis yielded 4 factors: doctor-patient relationship skills, reflective skills, time management, and interprofessional relationship skills. A decision study showed confidence intervals sufficiently narrow for many measurement purposes with as few as 8 observations. Four items frequently marked below expectations may be identifiers for "problem" students. CONCLUSION: This preliminary study suggests that the P-MEX is a feasible format for evaluating professionalism in clinical training.

A measure of quality of life after abdominal surgery.

OBJECTIVE: To develop a reliable and valid measure of short-term quality of life after abdominal surgery. SUMMARY BACKGROUND DATA: A major limitation of clinical trials evaluating laparoscopic surgical procedures has been the lack of a measure of short-term quality of life after abdominal surgery. METHODS: We used existing health status measures, focus groups, and semi-structured patient interviews to generate a prototype questionnaire of 51 items, which was administered to patients within 2 weeks after an abdominal surgical procedure. We used structural equations modeling to reduce the number of items, retaining the three items with the highest factor loadings on each of the factors that accounted for one or more eigenvalue. RESULTS: We administered the prototype questionnaire to 500 patients (mean age [SD] 53.4 [16.0], 51.4% male, 73.0% inpatient) at a mean 4.1 days after an abdominal surgical procedure. Item reduction yielded an 18-item measure with 6 sub-scales. The final instrument demonstrated good model fit in relation to our hypothesized factors (root mean square error of approximation 0.085, goodness-of-fit index 0.89). CONCLUSIONS: We developed a reliable and valid 18-item, 6-subscale measure of health-related quality of life after abdominal surgery, for use as an outcome measure in studies comparing laparoscopic and conventional abdominal surgery.

Using a comprehensive examination to assess multiple competencies in surgical residents: does the oral examination still have a role?

BACKGROUND: While specialty-level evaluations evolve from traditional examinations to objective structured clinical examination-like assessments, a broader range of competencies are tested; consequently, examiners are forced to integrate results when making a determination of competency. The aim of this study was to describe how experts weigh relative performances on specific components of a comprehensive examination to make decisions of overall competency. STUDY DESIGN: The Patient Assessment and Management Examination is a standardized patient-based assessment of general surgery in which each 25-minute station encompasses four components: history and physical examination, investigation interpretation, diagnosis and treatment discussion with the patient, and a structured oral examination (SOE). A six-station Patient Assessment and Management Examination was administered to 21 senior surgery residents. Surgeons marked each station with global rating scales and, in addition, provided an end-of-station overall global assessment of performance. A "gold-standard" examination pass-or-fail decision was determined through videotape review of each candidate's performance across six stations by two blinded surgeons. Multiple linear regression analysis was used to determine which components were associated with the end-of-station overall global assessments. Multivariable logistic regression was used to determine which components were associated with the final "gold-standard" pass-or-fail assessment. RESULTS: The only component notably (p < 0.005) associated with end-of-station global assessment for all six stations was the SOE. Mean SOE score was the only notable independent variable associated with the gold-standard pass-or-fail decision (R(2) = 0.63, p < 0.001). CONCLUSIONS: Performance on the SOE section of a multicompetency examination is markedly associated with the final determination of competency. These results have implications for the design and implementation of comprehensive specialty-level assessments.

Analytic global OSCE ratings are sensitive to level of training.

PURPOSE: There are several reasons for using global ratings in addition to checklists for scoring objective structured clinical examination (OSCE) stations. However, there has been little evidence collected regarding the validity of these scales. This study assessed the construct validity of an analytic global rating with 4 component subscales: empathy, coherence, verbal and non-verbal expression. METHODS: A total of 19 Year 3 and 38 Year 4 clinical clerks were scored on content checklists and these global ratings during a 10-station OSCE. T-tests were used to assess differences between groups for overall checklist and global scores, and for each of the 4 subscales. RESULTS: The mean global rating was significantly higher for senior clerks (75.5% versus 71.3%, t55 = 2.12, P < 0.05) and there were significant differences by level of training for the coherence (t55 = 3.33, P < 0.01) and verbal communication (t55 = 2.33, P < 0.05) subscales. Interstation reliability was 0.70 for the global rating and ranged from 0.58 to 0.65 for the subscales. Checklist reliability was 0.54. CONCLUSION: In this study, a summated analytic global rating demonstrated construct validity, as did 2 of the 4 scales measuring specific traits. In addition, the analytic global rating showed substantially higher internal consistency than did the checklists, a finding consistent with that seen in previous studies cited in the literature. Global ratings are an important element of OSCE measurement and can have good psychometric properties. However, OSCE researchers should clearly describe the type of global ratings they use. Further research is needed to define the most effective global rating scales.

Differential weighting of errors on a test of clinical reasoning skills.

PROBLEM STATEMENT AND BACKGROUND: Examinees can make three types of errors on the short-menu questions in the Clinical Reasoning Skills component of the Medical Council of Canada's Qualifying Examination Part I: (1) failing to select any correct responses, (2) selecting too many responses, or (3) selecting a response that is inappropriate or harmful to the patient. This study compared the information provided by equal and differential weighting of these errors. METHOD: The item response theory nominal model was applied to fit examinees' response patterns on the 1998 test. RESULTS: Differential error weighting resulted in improved model fit and increased test information for examinees in the lower half of the achievement continuum. CONCLUSION: Differential error weighting appears promising. The pass score is near the lower end of the achievement continuum; therefore, this approach may improve the accuracy of pass-fail decisions.

Evaluating the effectiveness of a 2-year curriculum in a surgical skills center.

BACKGROUND: This study was a formative evaluation of a 2-year Surgical Skills Center Curriculum (SSCC) using objective measures of surgical performance and self-reported process-oriented evaluations. METHODS: Fifty postgraduate third-year (PGY-3) residents participated in an Objective Structured Assessment of Technical Skills (OSATS) examination. Nineteen residents underwent the SSCC and 31 residents did not. During the SSCC, self-reported student and faculty evaluations were completed after each session. RESULTS: For the OSATS examination, scores were not significantly different between treatment and control groups, on either the checklist (66.4 +/- 6.1 versus 64.1 +/- 10.8) or global rating scale scores (66.9 +/- 6.9 versus 68.0 +/- 9.6). Further comparisons between groups on individual OSATS stations revealed no significant differences between groups. The majority of student and faculty evaluation remarks were highly positive. CONCLUSIONS: The OSATS results failed to support our hypothesis that training on a core procedure in a single session during a SSCC would have an appreciable and sustained effect after 2 years. Self-reported process-oriented evaluations support the utility of our SSCC.

The effect of candidates' perceptions of the evaluation method on reliability of checklist and global rating scores in an objective structured clinical examination.

PURPOSE: Process-oriented global ratings, which assess "overall performance" on one or a number of domains, have been purported to capture nuances of expert performance better than checklists. Pilot data indicate that students change behaviors depending on their perceptions of how they are being scored, while experts do not. This study examines the impact of the students' orientation to the rating system on OSCE scores and the interstation reliability of the checklist and global scores. METHOD: A total of 57 third- and fourth-year medical students at one school were randomly assigned to two groups and performed a ten-station OSCE. Group 1 was told that scores were based on checklists. Group 2 was informed that performance would be rated using global ratings geared toward assessing overall competence. All candidates were scored by physician-examiners who were unaware of the students' orientations to the rating system and who used both checklists and global rating forms. RESULTS: A mixed two-factor ANOVA identified a significant interaction of rating form by group (F(1,55) = 5.5, p <.05), with Group 1 (checklist-oriented) having higher checklist scores but lower global scores than did Group 2 (oriented to global ratings). In addition, Group 1 had higher interstation alpha coefficients than did Group 2 for both global scores (0.74 versus 0.63) and checklist scores (0.63 versus 0.40). CONCLUSIONS: The interaction effect on total exam scores suggests that students adapt their behaviors to the system of evaluation. However, the lower reliability coefficients for both forms found in the process-oriented global-rating group suggest that an individual's capacity to adapt to the system of global rating forms is relatively station-specific, possibly depending on his or her expertise in the domain represented in each station.