Transatlantic comparison of the competence of surgeons at the start of their professional career.

BACKGROUND: Although the objective in European Union and North American surgical residency programmes is similar-to train competent surgeons-residents' working hours are different. It was hypothesized that practice-ready surgeons with more working hours would perform significantly better than those being educated within shorter working week curricula. METHODS: At each test site, 21 practice-ready candidate surgeons were recruited. Twenty qualified Canadian and 19 qualified Dutch surgeons served as examiners. At both sites, three validated outcome instruments assessing multiple aspects of surgical competency were used. RESULTS: No significant differences were found in performance on the integrative and cognitive examination (Comprehensive Integrative Puzzle) or the technical skills test (Objective Structured Assessment of Technical Skill; OSATS). A significant difference in outcome was observed only on the Patient Assessment and Management Examination, which focuses on skills needed to manage patients with complex problems (P < 0.001). A significant interaction was observed between examiner and candidate origins for both task-specific OSATS checklist (P = 0.001) and OSATS global rating scale (P < 0.001) scores. CONCLUSION: Canadian residents, serving many more working hours, perform equivalently to Dutch residents when assessed on technical skills and cognitive knowledge, but outperformed Dutch residents in skills for patient management. Secondary analyses suggested that cultural differences influence the assessment process significantly.

Methodological problems in the retrospective computation of responsiveness to change: the lesson of Cronbach.

OBJECTIVE: To examine the relation between responsiveness coefficients derived directly from a calculation of average change resulting from a treatment intervention (Responsiveness-Treatment or RT) and those derived from retrospective analysis of changed and unchanged groups (Responsiveness Retrospective or RR) based on a global measure of change. METHOD: Two approaches were used. First, we used simulation methods to examine the analytical relationship between the RT and RR coefficients. We then located eight studies where it was possible to compute both RT and RR coefficients. As anticipated from theoretical arguments, the RR coefficients were larger than the RT coefficients (1.50 versus 0.41, p < .0001). Within study there was no predictable relationship between the two indices. Across studies, the magnitude of the RR coefficient was strongly related to the correlation with the retrospective global scale, and unrelated to the magnitude of the RT coefficient. The simulated curves fit well with the observed data, and substantiated the observation that the relation between RT and RR coefficients is complex and only weakly related to the size of the treatment effect. CONCLUSION: Retrospective methods of computing responsiveness yield little information about the ability of an instrument to detect treatment effects, and should not be used as a basis for choice of an instrument for applications to clinical trials.

Perceptual manifestations of an analytic structure: the priority of holistic individuation.

Properties that make items perceptually distinctive may not be represented in the dimensional structure used by subjects for analysis. In a classification task, a single dimensional structure, actually used by subjects when analyzing, occurred in several perceptual forms. Two types of perceptual variation were compared: (a) feature individuation, whether a feature occurs in a unique form in different items and (b) holistic individuation, the extent to which an item's features cohere into an individuated whole. These types of individuation had separate effects on exemplar-based classification. However, holistic individuation had priority in that the presence of individuated features did not produce exemplar-based transfer if the item's holistic properties were altered. This priority of holistic individuation occurred whether the subjects had been given the classification rule or had attempted to discover it.

A new assessment tool: the patient assessment and management examination.

BACKGROUND: The major goal of certification is to assure the public that the candidate is competent in all facets required of the position. The patient assessment and management examination (PAME) was developed to enable a more comprehensive assessment of competence in the practice of surgery. METHODS: A six-station, 3-hour, standardized-patient-based evaluation was developed. Each station was scored using a set of five-point global rating scales. PAME results were compared to the last two in training evaluation reports (ITER), the clinical knowledge component of the ITER (ITER-CK), an in-house oral examination (OE), and the Canadian Association of General Surgeons' multiple-choice examination (CAGS). RESULTS: Eighteen senior general surgery residents were evaluated. Overall reliability was 0.70 (Cronbach's alpha). Fifth-year residents scored significantly better than fourth-year residents (t = 3.062; p = 0.0074), with 1 year of training accounting for 37% of the variance in scores. Correlations between the PAME and each of the other measures were ITER, 0.24; ITER-CK, 0.38; OE, -0.13; and CAGS, 0.061, with the PAME demonstrating better reliability and stronger evidence of validity than any other. CONCLUSIONS: The PAME had better psychometric properties than other measures and assessed areas often not evaluated. This type of evaluation may be useful for feedback, remediation, or certification decisions.

Communication failures in the operating room: an observational classification of recurrent types and effects.

BACKGROUND: Ineffective team communication is frequently at the root of medical error. The objective of this study was to describe the characteristics of communication failures in the operating room (OR) and to classify their effects. This study was part of a larger project to develop a team checklist to improve communication in the OR. METHODS: Trained observers recorded 90 hours of observation during 48 surgical procedures. Ninety four team members participated from anesthesia (16 staff, 6 fellows, 3 residents), surgery (14 staff, 8 fellows, 13 residents, 3 clerks), and nursing (31 staff). Field notes recording procedurally relevant communication events were analysed using a framework which considered the content, audience, purpose, and occasion of a communication exchange. A communication failure was defined as an event that was flawed in one or more of these dimensions. RESULTS: 421 communication events were noted, of which 129 were categorized as communication failures. Failure types included "occasion" (45.7% of instances) where timing was poor; "content" (35.7%) where information was missing or inaccurate, "purpose" (24.0%) where issues were not resolved, and "audience" (20.9%) where key individuals were excluded. 36.4% of failures resulted in visible effects on system processes including inefficiency, team tension, resource waste, workaround, delay, patient inconvenience and procedural error. CONCLUSION: Communication failures in the OR exhibited a common set of problems. They occurred in approximately 30% of team exchanges and a third of these resulted in effects which jeopardized patient safety by increasing cognitive load, interrupting routine, and increasing tension in the OR.

Issues in cognitive psychology: implications for professional education.

Education and cognitive psychology have tended to pursue parallel rather than overlapping paths. Yet there is, or should be, considerable common ground, since both have major interests in learning and memory. This paper presents a number of topics in cognitive psychology, summarizes the findings in the field, and explores the implications for teaching and learning. THE ORGANIZATION OF LONG-TERM MEMORY: The acquisition of expertise in an area can be characterized by the development of idiosyncratic memory structures called semantic networks, which are meaningful sets of connections among abstract concepts and/or specific experiences. Information (such as the assumptions and hypotheses that are necessary to diagnose and manage cases) is retrieved through the activation of these networks. Thus, when teaching, new information must be embedded meaningfully in relevant, previously existing knowledge to ensure that it will be retrievable when necessary. INFLUENCES ON STORAGE AND RETRIEVAL FROM MEMORY: A wide variety of variables affect the capacity to store and retrieve information from memory, including meaning, the context and manner in which information is learned, and relevant practice in retrieval. Educational strategies must, therefore, be directed at three goals--to enhance meaning, to reduce dependence on context, and to provide repeated relevant practice in retrieving information. PROBLEM SOLVING AND TRANSFER: Much of the development of expertise involves the transition from using general problem-solving routines to using specialized knowledge that reduces the need for classic "problem solving." Two manifestations of this specialized knowledge are the use of analogy and the specialization of general routines in specific domains. To develop these specialized forms of knowledge, the learner must have extensive practice in using relevant problem-solving routines and in identifying the situations in which a particular routine is likely to be useful. CONCEPT FORMATION: Experts possess both abstract proto-typical information about categories and an extensive set of separate, specific examples of categories, which have been obtained through individual experience. Both these sources of information are used in categorization and diagnostic classifications. Thus, it is important for educators to be aware that experience with sample cases is not just an opportunity to apply and practice the rules "at the end of the chapter." Instead, experience with cases provides an alternative method of reasoning that is independent of, but equally useful to, analytical rules. DECISION MAKING: Experts clearly do not use classic formal decision theory, but rather make use of heuristics, or shortcuts, when making decisions. Nonetheless, experts generally make appropriate decisions. This suggests that the shortcuts are useful more often than not. Rather than teaching learners to avoid heuristics, then, it might be more reasonable to help them recognize those relatively infrequent situations where their heuristics are likely to fail.

An objective structured clinical examination for evaluating psychiatric clinical clerks.

PURPOSE: To assess the feasibility, reliability, and validity of an objective structured clinical examination (OSCE) for psychiatric clinical clerks. METHOD: In 1995 two parallel forms of a ten-station OSCE (eight clinical stations, two writing stations) were developed at the University of Toronto Faculty of Medicine Each 12-minute performance-based clinical station was assessed by a faculty psychiatrist using both a checklist for each student's performance content and a global-rating scale of the performance process. The students' clinical-station scores were calculated as the average of their content and process scores (expressed as percentages). Examiners also recorded an overall judgment of each students' performance (pass, borderline, or fail) and wrote [in collaboration with the standardized patient (SP) at that station] comments on each student's performance. There were two criteria for a passing grade: a total mark of 60% or higher across all ten stations and a "pass" or "borderline" mark in at least five of the eight clinical stations. Each OSCE form was administered three times. RESULTS: The first form was used to examine 94 clerks, the second form to examine 98 clerks. The students' mean scores for the two forms were 70.47% (SD, 6.33%) and 67.66% (SD, 7.05%), respectively. In addition to the standard evaluation information collected on the students, several critical incidents occurred (e.g., a student's loss of control of emotions) that may identify potential problems in professional conduct. The direct cost for one administration of the examination was approximately Can$3,300: the largest portion of this was for the SPs' time spent in training and performing their roles. CONCLUSION: Preliminary evidence suggests that a psychiatry OSCE is feasible for assessing complex psychiatric skills. However, careful attention must be paid to SP training, examination monitoring, detection of critical incidents, and provision of feedback to students, faculty, and SPs. The university's previous system of oral examinations required approximately 600 faculty hours per year. The OSCE requires approximately 450 faculty hours, and the 150 hours saved almost cover the Can$20,000 that the examination costs each year. In all, the OSCE is an evaluation system that has demonstrable reliability and is more enjoyable for both the faculty and the students.

Validation of an objective structured assessment of technical skill for surgical residents.

PURPOSE: This study examined the concurrent validity of the Objective Structured Assessment of Technical Skill (OSATS), a new test of technical skill for general surgery residents. METHOD: Twelve residents (six in their senior, or fifth, year and six in their junior, or third, year) at the University of Toronto in 1994-95 were ranked within level of training according to their OSATS marks and by surgical faculty. Correspondence between OSATS and faculty rankings was assessed using Spearman rank-order correlation coefficients. RESULTS: The correlations between test scores and faculty rankings were generally high for the senior residents but low for the junior residents. CONCLUSION: Scores on the OSATS accurately reflect the independent opinions of faculty regarding the technical skills of senior residents, suggesting that it is a valid measure of technical skill for these individuals. The scores did not, however, reproduce faculty rankings of the junior residents. Whether this was a failing of the OSATS or the faculty rankings requires further study.

Using videotaped benchmarks to improve the self-assessment ability of family practice residents.

PURPOSE: To address methodologic and statistical problems of previous studies of self-assessment by exposing participants to relevant standards, anchoring rating scales, and providing practice in the use of the assessment tool. METHOD: Fifty first- and second-year family practice residents performed a ten-minute patient interview with a difficult communication problem. Following each interview, the resident and two experts independently evaluated the resident's communication skills. The resident was then shown a videotape of four performances (ranging in quality from poor to good) of the same scenario. The resident evaluated the communication skills displayed in each performance and then reevaluated his or her own performance. RESULTS: The correlation between experts' evaluations and residents' self-evaluations was moderate immediately after the interview (r = 0.38) but increased significantly after the residents viewed the videotape (r = 0.52). This effect was more pronounced for first-year residents (0.22 to 0.45) than for second-year residents (0.53 to 0.65), although the difference was not significant. Post-hoc analysis revealed that neither initial nor post-benchmark self-assessment ability was related to the ability to accurately evaluate the benchmarks in a manner consistent with the experts. CONCLUSIONS: The ability to self-assess does not seem strongly tied to the ability to assess the performances of others on the same task. Nonetheless, providing a set of benchmarks against which trainees can compare their own performances improves their ability to self-evaluate even if the qualities of the benchmarks are not explicitly identified.

Validation of an objective structured clinical examination in psychiatry.

PURPOSE: To examine the validity of a psychiatry clerkship's objective structured clinical examination (OSCE). METHOD: In 1996, 33 clinical clerks and 17 psychiatry residents at the University of Toronto participated in an eight-station OSCE evaluated by psychiatrist-examiners using binary checklists and global ratings. Prior to the OSCE, communication course instructors were asked to rank the clerks on interviewing ability, and faculty supervisors were asked to identify the OSCE stations on which the clerks were likely to do well or poorly. RESULTS: Mean OSCE scores were significantly higher for the residents than for the clerks on global ratings but not on checklists. The communication instructors accurately predicted the clerks' rankings on the global scores but not their scores on the checklists. The faculty supervisors predicted with moderate accuracy the clerks' success on the OSCE stations as measured by the checklists but not by the global ratings. The residents rated the OSCE scenarios as highly realistic. CONCLUSIONS: The evidence of construct and concurrent validity together with high ratings of realism suggest that a psychiatry OSCE can be a valid assessment of clerks' clinical competence.

Assessing the generalizability of OSCE measures across content domains.

PURPOSE: To assess the degree to which OSCE measures generalize across multiple administrations to the same students. METHODS: Students' scores from three OSCEs at one institution were correlated to determine the generalizability of the scoring systems across course domains. RESULTS: Analysis revealed that while checklist scores showed quite low correlations across examinations from different domains (ranging from 0.14 to 0.25), global process scores showed quite reasonable correlations (ranging from 0.30 to 0.44), with the correlations for global scores being significantly higher than those for checklist scores in all three comparisons. CONCLUSION: These data would seem to confirm the intuitions about each of these measures: the checklist scores are highly content-specific, while the global scores are evaluating a more broadly based set of skills. Implications for the use of these scales are discussed.

Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination.

PURPOSE: To compare the psychometric properties of checklists, global rating scales preceded by a checklist, and global rating scales alone in assessing surgery residents' performances on an OSCE-like technical skills examination. METHOD: In 1996, 53 general surgery residents with one to six years of postgraduate training participated in a performance-based examination of technical skills consisting of eight 15-minute stations (bench-model simulations of operative procedures in general surgery). Two qualified surgeons marked at each station, one using a task-specific checklist (C) and a subsequent global rating scale (Gc), the other using a global rating scale only (G). RESULTS: Interstation reliabilities measured by Cronbach's alpha were .79 for C, .89 for Gc, and .85 for G. A series of multiple regressions predicting level of training from test scores revealed an R2 of .584 for C alone, which increased to .711 when Gc was entered after (p < .001), and increased to .704 when G was entered after C (p < .001). However, R2 for Gc alone was .711, and for G alone was .704, neither of which changed when C was entered into the prediction (p > .10). The R2 for Gc and G predicting level of training (.725) was not significantly greater than that of either Gc or G alone. A very similar pattern of results was seen when C, Gc, and G were used to predict independent evaluations of the operative outcomes. CONCLUSIONS: Global rating scales scored by experts showed higher inter-station reliability, better construct validity, and better concurrent validity than did checklists. Further, the presence of the checklists did not improve the reliability or validity of the global rating scale over that of the global rating scale alone. These results suggest that global rating scales administered by experts are a more appropriate summative measure when assessing candidates on performance-based examinations.

OSCE checklists do not capture increasing levels of expertise.

PURPOSE: To evaluate the effectiveness of binary content checklists in measuring increasing levels of clinical competence. METHOD: Fourteen clinical clerks, 14 family practice residents, and 14 family physicians participated in two 15-minute standardized patient interviews. An examiner rated each participant's performance using a binary content checklist and a global process rating. The participants provided a diagnosis two minutes into and at the end of the interview. RESULTS: On global scales, the experienced clinicians scored significantly better than did the residents and clerks, but on checklists, the experienced clinicians scored significantly worse than did the residents and clerks. Diagnostic accuracy increased for all groups between the two-minute and 15-minute marks without significant differences between the groups. CONCLUSION: These findings are consistent with the hypothesis that binary checklists may not be valid measures of increasing clinical competence.

Who should rate candidates in an objective structured clinical examination?

PURPOSE: To determine who is the better rater of history taking in an objective structured clinical examination (OSCE): a physician or a standardized patient (SP). METHOD: During the 1991 pilot administration of an OSCE for the Medical Council of Canada's qualifying examination, five history-taking stations were videotaped. Candidates at these stations were scored by three raters: a physician (MD), an SP observer (SPO), and an SP rating from recall (SPR). To determine the validity of each rater's scores, these scores were compared with a "gold standard", which was the average of videotape ratings by three physicians, each scoring independently. Analysis included both correlations with the standard and a repeated-measures analysis of variance (ANOVA) comparing raters' mean scores on each station with mean scores of the gold standard. RESULTS: Ninety-one videotapes were scored by the "gold-standard" physicians. Correlations with the standard showed no clear preference for MD, SPO, or SPR raters. ANOVAs revealed significant differences from the standard on three stations for the SPR, two stations for the SPO, and one stations for the MD. CONCLUSIONS: An MD rater is less likely to differ from a standard established by a consensus of MD ratings than are SP raters rating from recall. If an MD cannot be used, an SP observer is preferable to an SP rating from recall.

Structured assessment of minor surgical skills (SAMSS) for family medicine residents.

PURPOSE: To develop a valid and reliable examination to assess the technical proficiency of family medicine residents' performance of minor surgical office procedures. METHOD: A multi-station OSCE-style examination using bench-model simulations of minor surgical procedures was developed. Participants were a randomly selected group of 33 family medicine residents (PGY-1 = 16, PGY-2 = 17) and 14 senior surgical residents who functioned as a validation group. Examiners were qualified surgeons and family physicians who used both checklists and global rating scales to score the participants' performances. RESULTS: When family medicine residents were evaluated by family physicians, interstation reliabilities were .29 for checklists and .42 for global ratings. When family medicine residents were evaluated by surgeons, the reliabilities were .53 for checklists and .75 for global ratings. Interrater reliability, measured as a correlation for total examination scores, was .97. Mean scores on the examination were 60%, 64%, and 87% for PGY-1 family medicine, PGY-2 family medicine, and surgery residents, respectively. The difference in scores between family medicine and surgery residents was significant (p < .001), providing evidence of construct validity. CONCLUSION: A new examination developed for assessing family medicine residents' skills with minor surgical office procedures is reliable and has evidence for construct validity. The examination has low reliability when family physicians serve as examiners, but moderate reliability when surgeons are the evaluators.

Faculty and resident opinions regarding the role of morbidity and mortality conference.

BACKGROUND: The role of the surgical morbidity and mortality (M&M) conference as a forum for examination of surgical failure may remain unclear. This paper reports the results of a national survey of surgical faculty and trainees regarding the role and effectiveness of the M&M conference. METHODS: Based on focus groups and pilot studies from multiple institutions in one geographic area, a questionnaire addressing critical issues in attitudes toward the M&M conference was sent to 1,100 randomly chosen subjects nationwide: 500 residents and 600 staff. The survey includes individual and institutional demographic information, 15 statements answered using a Likert scale, and 2 open-ended questions. RESULTS: Faculty response rate was 501 of 600 (83%) and resident response rate was 166 of 500 (33%). Responses were generally positive in both groups, with staff showing small but significantly more positive attitudes than residents. A higher proportion of residents characterize the M&M conference as too defensive. CONCLUSION: The M&M conference is fulfilling its potential as a teaching tool but there may be room for improvement as residents view the experience slightly less positively than faculty. This questionnaire provides a perspective of expectations for the M&M conference, allowing educators to optimize its effectiveness.

Using operative outcome to assess technical skill.

BACKGROUND: This study examined whether an operative product and time to completion could serve as measures of technical skill. METHODS: Nine final-year (PGY5) and 11 penultimate-year (PGY4) general surgery residents participated in a 6-station bench model examination. Time to completion was recorded. Twelve faculty surgeons (2 per station) evaluated the quality of the final product using a 5-point scale. RESULTS: The mean interrater reliability was 0. 59 for product quality. Interstation reliability was 0.59 for analysis of the final product and 0.72 for time to completion. There was 63% and 78% agreement between attendings' ratings and product quality and time scores respectively. PGY5s' mean product quality score was 4.14 +/- 0.26, compared with 3.82 +/- 0.33 for PGY4s (P < 0.05). PGY5s' mean time was 110 +/- 19 minutes compared with PGY4s' 132 +/- 15 (P < 0.05). CONCLUSIONS: Analysis of the operative end product and time to completion offer efficient alternatives to on-line examiner scoring for bench model examinations of technical competence.

Evaluating surgical resident selection procedures.

PURPOSE: The purposes of this study were to develop and assess a rating form for selection of surgical residents, determine the criteria most important in selection, determine the reliability of the assessment form and process both within and across sites, and document differences in procedure and structure of resident selection processes across Canada. METHODS: Twelve of 13 English-speaking orthopedic surgery training programs in Canada participated during the 1999 selection year. The critical incident technique was utilized to determine the criteria most important in selection. From these criteria a 10-item rating form was developed with each item on a 5-point scale. Sixty-six candidates were invited for interviews across the country. Each interviewer completed one assessment form for each candidate, and independently ranked all candidates at the conclusion of all interviews. Consensus final rank orders were then created for each residency program. Across all programs, pairwise program-by-program correlations for each assessment parameter were made. RESULTS: The internal consistency of assessment form ratings for each interviewer was moderately high (mean Cronbach's alpha = 0.71). A correlation between each item and the final rank order for each program revealed that the items work ethic, interpersonal qualities, orthopedic experience, and enthusiasm correlated most highly with final candidate rank orders (r = 0.5, 0.48, 0.48, 0.45, respectively). The interrater reliabilities (within panels) and interpanel reliabilities (within programs) for the rank orders were 0.67 and 0.63, respectively. Using the Spearman-Brown prophecy formula, it was found that two panels with two interviewers on each panel are required to obtain a stable measure of a given candidate (reliabilities of 0.80). The average pairwise program-by-program correlations were low for the final candidate rank orders (0.14). CONCLUSIONS: A method was introduced to develop a standard, reliable candidate assessment form to evaluate residency selection procedures. The assessment form ratings were found to be consistent within interviewers. Candidate assessments within programs (both between interviewers and between panels) were moderately reliable suggesting agreement within programs regarding the relative quality of candidates, but there was very little agreement across programs.

Testing technical skill via an innovative "bench station" examination.

BACKGROUND: A new approach to testing operative technical skills, the Objective Structured Assessment of Technical Skill (OSATS), formally assesses discrete segments of surgical tasks using bench model simulations. This study examines the interstation reliability and construct validity of a large-scale administration of the OSATS. METHODS: A 2-hour, eight-station OSATS was administered to 48 general surgery residents. Residents were assessed at each station by one of 48 surgeons who evaluated the resident using two methods of scoring: task-specific checklists and global rating scales. RESULTS: Interstation reliability was 0.78 for the checklist score, and 0.85 for the global score. Analysis of variance revealed a significant effect of training for both the checklist score, F(3,44) = 20.08, P <0.001, and the global score, F(3,44) = 24.63, P <0.001. CONCLUSIONS: The OSATS demonstrates high reliability and construct validity, suggesting that we can effectively measure residents' technical ability outside the operating room using bench model simulations.