Transatlantic comparison of the competence of surgeons at the start of their professional career.

BACKGROUND: Although the objective in European Union and North American surgical residency programmes is similar-to train competent surgeons-residents' working hours are different. It was hypothesized that practice-ready surgeons with more working hours would perform significantly better than those being educated within shorter working week curricula. METHODS: At each test site, 21 practice-ready candidate surgeons were recruited. Twenty qualified Canadian and 19 qualified Dutch surgeons served as examiners. At both sites, three validated outcome instruments assessing multiple aspects of surgical competency were used. RESULTS: No significant differences were found in performance on the integrative and cognitive examination (Comprehensive Integrative Puzzle) or the technical skills test (Objective Structured Assessment of Technical Skill; OSATS). A significant difference in outcome was observed only on the Patient Assessment and Management Examination, which focuses on skills needed to manage patients with complex problems (P < 0.001). A significant interaction was observed between examiner and candidate origins for both task-specific OSATS checklist (P = 0.001) and OSATS global rating scale (P < 0.001) scores. CONCLUSION: Canadian residents, serving many more working hours, perform equivalently to Dutch residents when assessed on technical skills and cognitive knowledge, but outperformed Dutch residents in skills for patient management. Secondary analyses suggested that cultural differences influence the assessment process significantly.

The use of virtual reality for training in carotid artery stenting: a construct validation study.

BACKGROUND: Given that carotid artery stenosis (CAS) intervention is procedurally difficult, possesses an extensive learning curve, and involves a grave list of potential complications, construct validation of new non-clinical training devices is of increasing importance. PURPOSE: To evaluate the construct validity of the Procedicus-Virtual Interventional Simulator Trainer (Procedicus-VIST) and its use as a training tool. MATERIAL AND METHODS: Sixteen interventionalists (15 males, one female; mean interventional radiology [IR] experience >11 years) and 16 medical students (15 males, one female; no IR experience) received 1 hour of didactic instruction followed by an hour of familiarization training. Subjects then attempted to complete a carotid artery stenting procedure within 1 hour while their performance metrics were recorded. All participants completed a qualitative exit survey of subjective parameters using a visual analog scale. RESULTS: Procedure and fluoroscopic time was 8.7 and 8.7 min greater in the novice group (P=0.0066 and P=0.0031), respectively. There were no significant differences in performances between the two groups in the remaining metrics of cine loops (number recorded), tool/vessel ratio, coverage percentage, and placement accuracy or residual stenosis. Contrast measurement metrics were found to be too imprecise for statistical analysis. Experienced and novice opinions differed significantly for six of 10 subjective parameters. No statistically significant difference in video-gaming habits was demonstrated. CONCLUSION: With the exception of the metrics of performance time and fluoroscopic use, construct validity of the Procedicus-VIST carotid metrics were not confirmed. Virtual reality simulation as a training method was valued more by novices than by experienced interventionalists.

Resident education in orthopaedic trauma: the future role of competency-based medical education.

As residency training programmes around the globe move towards competency-based medical education (CBME), there is a need to review current teaching and assessment practices as they relate to education in orthopaedic trauma. Assessment is the cornerstone of CBME, as it not only helps to determine when a trainee is fit to practice independently, but it also provides feedback on performance and guides the development of competence. Although a standardised core knowledge base for trauma care has been developed by the leading national accreditation bodies and international agencies that teach and perform research in orthopaedic trauma, educators have not yet established optimal methods for assessing trainees' performance in managing orthopaedic trauma patients. This review describes the existing knowledge from the literature on assessment in orthopaedic trauma and highlights initiatives that have recently been undertaken towards CBME in the United Kingdom, Canada and the United States. In order to support a CBME approach, programmes need to improve the frequency and quality of assessments and improve on current formative and summative feedback techniques in order to enhance resident education in orthopaedic trauma. Cite this article: Bone Joint J 2016;98-B:1320-5.

Small bowel transplantation in the dog using cyclosporine.

The effect of the new immunosuppressant cyclosporine on survival after total small intestinal allotransplantation (TSIA) was studied in a canine model. Successful TSIA was performed in 34 dogs. Eleven dogs were treated with cyclosporine, 25 mg/kg/day i.m., starting the day before the operation and continuing for four weeks. Thereafter the same dose was given orally. Thirteen dogs were given oral cyclosporine only, 25 mg/kg/day from the day after transplantation. Ten dogs served as controls. The dogs treated with intramuscular and oral cyclosporine survived a mean of 103.8 +/- 39.4 days (mean +/- S.E.M.). The longest survivor died after 432 days. Survival in this group was significantly longer than that of the control dogs, which survived 12.5 +/- 4.6 days. The orally treated dogs survived 30.4 +/- 7.6 days. All control dogs, and seven of the orally treated dogs, but only two of the intramuscularly treated dogs, died of acute rejection. It is concluded that cyclosporine is effective in prolonging survival after TSIA in the dog and reduces the incidence of acute rejection.

Improved prediction of success in surgical clerkship through measures of information processing.

A new system for predicting success of surgical student performance has been developed. A test of surgical knowledge, with questions given in the form of analogies, was administered to 16 students in their fourth week of clerkship. While solving test items, students' eye movements and fixations were tracked. By analysis of the recordings, eight scores of information-processing capabilities were derived. The processing scores and conventional predictors of medical school clinical performance were analyzed to determine their power to predict success, defined by ratings given on a 1 to 10 scale by 21 faculty members based on three tests of cognitive knowledge, two performance-based examinations, and faculty reports. The ratings were reliable (generalizability coefficient = 0.72; p less than 0.001). Stepwise regression analysis of all variables selected one MCAT score (science problems) and two information-processing scores to the statistical model that maximally predicted success. Regression coefficient for the science problem subset of the MCAT was 0.42. This was augmented to R2 = 0.77 when information processing variables were included. The increment was significant, F (2, 11) = 9.25; p less than 0.01. A newly developed test, coupled with techniques that made possible the derivation of components of information processing, nearly doubled the power of conventional tests to predict success in surgical clerkship.

A new assessment tool: the patient assessment and management examination.

BACKGROUND: The major goal of certification is to assure the public that the candidate is competent in all facets required of the position. The patient assessment and management examination (PAME) was developed to enable a more comprehensive assessment of competence in the practice of surgery. METHODS: A six-station, 3-hour, standardized-patient-based evaluation was developed. Each station was scored using a set of five-point global rating scales. PAME results were compared to the last two in training evaluation reports (ITER), the clinical knowledge component of the ITER (ITER-CK), an in-house oral examination (OE), and the Canadian Association of General Surgeons' multiple-choice examination (CAGS). RESULTS: Eighteen senior general surgery residents were evaluated. Overall reliability was 0.70 (Cronbach's alpha). Fifth-year residents scored significantly better than fourth-year residents (t = 3.062; p = 0.0074), with 1 year of training accounting for 37% of the variance in scores. Correlations between the PAME and each of the other measures were ITER, 0.24; ITER-CK, 0.38; OE, -0.13; and CAGS, 0.061, with the PAME demonstrating better reliability and stronger evidence of validity than any other. CONCLUSIONS: The PAME had better psychometric properties than other measures and assessed areas often not evaluated. This type of evaluation may be useful for feedback, remediation, or certification decisions.

A rationale for the teaching of statistics to surgical residents.

The two aims of this study were to investigate the use of statistics in the surgical literature and to assess the degree of statistical comprehension possessed by graduating surgical residents. Two hundred journal articles were randomly selected from the 1984 issues of four surgical journals and were reviewed for statistical content. A classification of statistical techniques was created. A reader who has knowledge of descriptive statistics only has access to 44.5% of the articles. The addition of knowledge of t tests, contingency table analysis, other nonparametric techniques, and life table analysis to a reader's repertoire increases the access rate to 80.5%. The data indicate the specific statistical techniques that would best serve the surgeon who is attempting to increase access rate to the surgical literature. Ninety-one surgical residents in their fifth postgraduate year (PGY-5) responded to a questionnaire regarding their knowledge of statistics. While 90% of the respondents thought they would benefit from a course on statistics, 92% reported that they had received less than 5 hours of instruction in statistics during their residency. Both subjective self-ratings and objective testing revealed that the residents surveyed have a suboptimal knowledge of statistics. The results suggest the need for formal instruction in statistics during surgical residency.

Results and hemodynamic changes after interposition mesocaval shunt.

Forty-seven patients have been treated by interposition mesocaval shunting for portal hypertension and variceal bleeding between December 1973 and March 1980. The average age was 55 years. The underlying diseases were alcoholic cirrhosis in 26 patients (56%), macronodular cirrhosis in 11 patients (23%), and other causes in 10 patients (21%). Thirty-five operations (75%) were performed on an emergency basis for patients who continued to bleed after failure of conservative management. In these patients, the early mortality rate was 43%. Overall survival, rebleeding, and postshunt encephalopathy rates are correlated with the preoperative Child's classification. These figures are similar to those reported for end-to-side portocaval shunts. The improvement in postshunt encephalopathy rates as reported by Drapanas is not borne out by our results. Postshunt angiography was performed in 31 patients and shunt patency was confirmed in 28 (90%). In 26 patients, selective studies to determine portal flow patterns were carried out, and in only three patients was there any evidence of hepatopedal flow. In each of these patients, some kinking of the shunt was noted. Mesocaval shunting is a reasonable alternative to end-to-side portocaval shunts and is associated with similar rates of patency, rebleeding, mortality, and late postoperative encephalopathy. A well-constructed, patent mesocaval shunt totally diverts portal flow.

Description and evaluation of the Surgeons as Educators course.

BACKGROUND: Since 1993, the American College of Surgeons has sponsored an annual 6-day course entitled the Surgeons as Educators. The course was designed to provide academic surgeons with the knowledge and skills necessary to enhance the surgical education curriculum, teaching strategies, educational program administration, and performance evaluation. This article describes the development, implementation, and effect of the course on the classes graduating in 1993 and 1994. STUDY DESIGN: The effect of the course was studied by using a longitudinal survey approach. A survey was mailed to participants 3 to 6 months after they completed the course. Graduates were asked to describe any education-related actions taken attributable to attending the Surgeons as Educators course. The quality of course content and presentations were evaluated by using end-of-course evaluation forms and daily feedback forms and by an external reviewer. RESULTS: Within 6 months of returning from the course, more than one half of the graduates initiated actions related to curriculum development, teaching strategies, or educational administration. One third or more of the graduates modified their performance and program evaluation systems. Using a five-point scale, ratings of the course content ranged from 3.78 to 4.64 for "value of topic" and from 3.77 to 4.76 for "quality of presentation." Items evaluated by the graduates on the end-of-course evaluation forms ranged from 7.8 to 8.7 on a nine-point scale. CONCLUSIONS: The Surgeons as Educators course offered an opportunity for participants to interact among themselves and with course faculty about educational issues and to practice teaching skills. The course was highly rated for educational quality and value. The retreat environment and the length of the program helped attendees become immersed during this "protected time" to analyze strengths and weaknesses of their programs and devise achievable plans to improve their abilities as educators and the effectiveness of their programs.

Communication failures in the operating room: an observational classification of recurrent types and effects.

BACKGROUND: Ineffective team communication is frequently at the root of medical error. The objective of this study was to describe the characteristics of communication failures in the operating room (OR) and to classify their effects. This study was part of a larger project to develop a team checklist to improve communication in the OR. METHODS: Trained observers recorded 90 hours of observation during 48 surgical procedures. Ninety four team members participated from anesthesia (16 staff, 6 fellows, 3 residents), surgery (14 staff, 8 fellows, 13 residents, 3 clerks), and nursing (31 staff). Field notes recording procedurally relevant communication events were analysed using a framework which considered the content, audience, purpose, and occasion of a communication exchange. A communication failure was defined as an event that was flawed in one or more of these dimensions. RESULTS: 421 communication events were noted, of which 129 were categorized as communication failures. Failure types included "occasion" (45.7% of instances) where timing was poor; "content" (35.7%) where information was missing or inaccurate, "purpose" (24.0%) where issues were not resolved, and "audience" (20.9%) where key individuals were excluded. 36.4% of failures resulted in visible effects on system processes including inefficiency, team tension, resource waste, workaround, delay, patient inconvenience and procedural error. CONCLUSION: Communication failures in the OR exhibited a common set of problems. They occurred in approximately 30% of team exchanges and a third of these resulted in effects which jeopardized patient safety by increasing cognitive load, interrupting routine, and increasing tension in the OR.

Validation of an objective structured assessment of technical skill for surgical residents.

PURPOSE: This study examined the concurrent validity of the Objective Structured Assessment of Technical Skill (OSATS), a new test of technical skill for general surgery residents. METHOD: Twelve residents (six in their senior, or fifth, year and six in their junior, or third, year) at the University of Toronto in 1994-95 were ranked within level of training according to their OSATS marks and by surgical faculty. Correspondence between OSATS and faculty rankings was assessed using Spearman rank-order correlation coefficients. RESULTS: The correlations between test scores and faculty rankings were generally high for the senior residents but low for the junior residents. CONCLUSION: Scores on the OSATS accurately reflect the independent opinions of faculty regarding the technical skills of senior residents, suggesting that it is a valid measure of technical skill for these individuals. The scores did not, however, reproduce faculty rankings of the junior residents. Whether this was a failing of the OSATS or the faculty rankings requires further study.

Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination.

PURPOSE: To compare the psychometric properties of checklists, global rating scales preceded by a checklist, and global rating scales alone in assessing surgery residents' performances on an OSCE-like technical skills examination. METHOD: In 1996, 53 general surgery residents with one to six years of postgraduate training participated in a performance-based examination of technical skills consisting of eight 15-minute stations (bench-model simulations of operative procedures in general surgery). Two qualified surgeons marked at each station, one using a task-specific checklist (C) and a subsequent global rating scale (Gc), the other using a global rating scale only (G). RESULTS: Interstation reliabilities measured by Cronbach's alpha were .79 for C, .89 for Gc, and .85 for G. A series of multiple regressions predicting level of training from test scores revealed an R2 of .584 for C alone, which increased to .711 when Gc was entered after (p < .001), and increased to .704 when G was entered after C (p < .001). However, R2 for Gc alone was .711, and for G alone was .704, neither of which changed when C was entered into the prediction (p > .10). The R2 for Gc and G predicting level of training (.725) was not significantly greater than that of either Gc or G alone. A very similar pattern of results was seen when C, Gc, and G were used to predict independent evaluations of the operative outcomes. CONCLUSIONS: Global rating scales scored by experts showed higher inter-station reliability, better construct validity, and better concurrent validity than did checklists. Further, the presence of the checklists did not improve the reliability or validity of the global rating scale over that of the global rating scale alone. These results suggest that global rating scales administered by experts are a more appropriate summative measure when assessing candidates on performance-based examinations.

Structured assessment of minor surgical skills (SAMSS) for family medicine residents.

PURPOSE: To develop a valid and reliable examination to assess the technical proficiency of family medicine residents' performance of minor surgical office procedures. METHOD: A multi-station OSCE-style examination using bench-model simulations of minor surgical procedures was developed. Participants were a randomly selected group of 33 family medicine residents (PGY-1 = 16, PGY-2 = 17) and 14 senior surgical residents who functioned as a validation group. Examiners were qualified surgeons and family physicians who used both checklists and global rating scales to score the participants' performances. RESULTS: When family medicine residents were evaluated by family physicians, interstation reliabilities were .29 for checklists and .42 for global ratings. When family medicine residents were evaluated by surgeons, the reliabilities were .53 for checklists and .75 for global ratings. Interrater reliability, measured as a correlation for total examination scores, was .97. Mean scores on the examination were 60%, 64%, and 87% for PGY-1 family medicine, PGY-2 family medicine, and surgery residents, respectively. The difference in scores between family medicine and surgery residents was significant (p < .001), providing evidence of construct validity. CONCLUSION: A new examination developed for assessing family medicine residents' skills with minor surgical office procedures is reliable and has evidence for construct validity. The examination has low reliability when family physicians serve as examiners, but moderate reliability when surgeons are the evaluators.

A comparative analysis of the costs of administration of an OSCE (objective structured clinical examination).

BACKGROUND: The objective structured clinical examination (OSCE) has become an accepted technique for the evaluation of clinical competence in medicine. Although advances have been made in our knowledge of the psychometric aspects of the OSCE, extremely little has been written about feasibility and cost issues. Given the current economic imperative to control costs and the extremely scant literature on the costs of developing and administering an examination in medicine, the authors felt it timely and relevant to explore issues related to the cost of the OSCE. METHOD: In 1991-92 and in 1992-93, costs and time requirements to implement and administer a structured oral (SO) examination and a six-station OSCE for a surgical clerkship at the University of Toronto Faculty of Medicine were gathered by review of invoices, interviews with those involved, and perusal of diaries kept by staff. RESULTS: To develop and administer the six-station OSCE, 327.5 hours of staff and faculty time were required for each rotation of surgical clerks (8.2 hours per student). The SO examination required 110 hours of staff and faculty time (2.75 hours per student). Direct expenses for the OSCE amounted to U.S. $6.90 per student per station, compared with no direct expense for the SO examination. CONCLUSION: The OSCE was more time-consuming and more expensive in human and material costs than the SO examination. However, costs of the OSCE can be substantially reduced from approximately U.S. $35 to U.S. ! per student per station if test developers, standardized patients, support staff, and examiners can donate their time. The authors compare the costs and time requirements of their OSCE with those of other OSCEs reported in the literature, and they provide guidelines to assist educators in deciding whether the costs of an OSCE are justifiable in the educators' individual settings.

Guidelines for estimating the real cost of an objective structured clinical examination.

A major impediment to the use of the objective structured clinical examination (OSCE) is that it is a labor-intensive and costly form of assessment. The cost of an OSCE is highly dependent on the particular model used, the extent to which hidden costs are reported, and the purpose of the examination. The authors detail hypothetical costs of running a four-hour OSCE for 120 medical students at one medical school. Costs are reported for four phases of this process: development, production, administration, and post-examination reporting and analysis. Costs are reported at two ends of the spectrum: the high end, where it is assumed that little is paid for by the institution and that faculty receive honoraria for work put into the examination; and the low end, where it is assumed that the sponsoring institution defrays basic costs and that faculty do not receive honoraria for their participation. The total costs reported for a first-time examination were $104,400 and $59,460 (Canadian dollars) at the high and low ends, respectively. These translate to per-student costs of $870 and $496. The cost of running an OSCE is high. However, the OSCE is uniquely capable of assessing many fundamental clinical skills that are presently not being assessed in a rigorous way in most medical schools.

Who should rate candidates in an objective structured clinical examination?

PURPOSE: To determine who is the better rater of history taking in an objective structured clinical examination (OSCE): a physician or a standardized patient (SP). METHOD: During the 1991 pilot administration of an OSCE for the Medical Council of Canada's qualifying examination, five history-taking stations were videotaped. Candidates at these stations were scored by three raters: a physician (MD), an SP observer (SPO), and an SP rating from recall (SPR). To determine the validity of each rater's scores, these scores were compared with a "gold standard", which was the average of videotape ratings by three physicians, each scoring independently. Analysis included both correlations with the standard and a repeated-measures analysis of variance (ANOVA) comparing raters' mean scores on each station with mean scores of the gold standard. RESULTS: Ninety-one videotapes were scored by the "gold-standard" physicians. Correlations with the standard showed no clear preference for MD, SPO, or SPR raters. ANOVAs revealed significant differences from the standard on three stations for the SPR, two stations for the SPO, and one stations for the MD. CONCLUSIONS: An MD rater is less likely to differ from a standard established by a consensus of MD ratings than are SP raters rating from recall. If an MD cannot be used, an SP observer is preferable to an SP rating from recall.

An objective structured clinical examination for the licentiate: report of the pilot project of the Medical Council of Canada.

The Medical Council of Canada (MCC) administers a qualifying examination for the issuance of a license to practice medicine. To date, this examination does not test the clinical skills of history taking, physical examination, and communication. The MCC is implementing an objective structured clinical examination (OSCE) to test these skills in October 1992. A pilot examination was developed to test the feasibility, reliability, and validity of running a multisite, two-form, four-hour, 20-station OSCE for national licensure. In February 1991, 240 volunteer first- and second-year residents were tested at four sites. The candidates were randomly assigned to one of two forms of the test and one of two sites for two of the four sites. Generalizability analysis revealed that the variance due to form was 0.0 and that due to site was .16 compared with a total variance of 280.86. The reliabilities (inter-station) were .56 and .60 for the two forms. Station total-test score correlations, used to measure station validity, were significant for 38 of the 40 stations used (range .14-.60). The results of the OSCE correlated moderately with the MCC qualifying examination; these correlations were .32 and .35 for the two test forms. Content validity was assessed by postexamination questionnaires given to the physician examiners using a scale of 0 (low) to 10 (high). The physicians' mean ratings were: importance of the stations, 8.1 (SD, 1.8); success of the examination in testing core skills, 8.1 (SD, 1.6); and degree of challenge, 7.8 (SD, 2.1). The results indicate that a full-scale national administration of an OSCE for licensure is feasible using the model developed. Aspects of validity have been established and strategies to augment reliability have been developed.

Teaching and testing technical skills.

Teaching technical skills is one of the most important tasks of a surgeon. This article discusses current issues in teaching and testing technical skills. For the most part, the level of technical skills cannot be predicted before a surgical resident starts a program. Different methods of teaching technical skills are reviewed (in and out of the operating room). For optimal effectiveness in teaching residents, we must apply principles of adult learning to the surgical domain. A methodologic framework for skill acquisition, adapted from the educational psychology literature, is discussed. Five methods of assessing technical skills are presented. Structuring the assessment process has resulted in higher levels of reliability and improved validity.

Effect of sleep deprivation on the performance of surgical residents.

Surgical residents were tested under sleep-deprived and nonsleep-deprived conditions. Three performance domains were tested: factual recall, the ability to concentrate, and manual dexterity. Sleep deprivation was defined as less than 3 hours of sleep in a 24 hour period. A randomized, repeated measures design was used. Split-plot analysis of variances was used to analyze the mean scores, and it revealed that there were no significant differences in performance in the sleep-deprived condition compared with performance in the nonsleep-deprived condition in any of the three domains tested. Moderate sleep deprivation appeared to be well tolerated by the surgical residents studied in this investigation.

The structured oral examination as a method for assessing surgical residents.

The purpose of this study was to determine the overall reliability, inter-rater reliability, and criterion validity of the structured oral examination (SOE) for assessing surgical residents. An SOE consisting of four predetermined clinically oriented scenarios was administered to 23 second postgraduate year surgical residents. Each scenario had five to six questions, each with a specific marking scheme. Candidates were assessed by two examiners and scores were derived independently. Overall reliability (Cronbach's alpha) was 0.75. Inter-rater reliability was significant for each pair of examiners and each question (r = 0.78 to 0.91: p less than 0.0001). Criterion validity was measured by correlating SOE scores with multiple-choice examination (MCQ) and objective structured clinical examination (OSCE) scores. Correlations between the SOE and MCQ and OSCE were significant and fell into the moderate range (0.48 to 0.51). The results of this study show that the SOE is useful in the assessment of clinical knowledge and problem-solving abilities of the surgical resident. Overall and inter-rater reliabilities achieved exceed those of traditional oral examination formats.