ABSTRACT
As residency training programmes around the globe move towards competency-based medical education (CBME), there is a need to review current teaching and assessment practices as they relate to education in orthopaedic trauma. Assessment is the cornerstone of CBME, as it not only helps to determine when a trainee is fit to practice independently, but it also provides feedback on performance and guides the development of competence. Although a standardised core knowledge base for trauma care has been developed by the leading national accreditation bodies and international agencies that teach and perform research in orthopaedic trauma, educators have not yet established optimal methods for assessing trainees' performance in managing orthopaedic trauma patients. This review describes the existing knowledge from the literature on assessment in orthopaedic trauma and highlights initiatives that have recently been undertaken towards CBME in the United Kingdom, Canada and the United States. In order to support a CBME approach, programmes need to improve the frequency and quality of assessments and improve on current formative and summative feedback techniques in order to enhance resident education in orthopaedic trauma. Cite this article: Bone Joint J 2016;98-B:1320-5.
Subject(s)
Clinical Competence/standards , Competency-Based Education/methods , Education, Medical, Graduate/methods , Internship and Residency , Orthopedics/education , Physicians/standards , Wounds and Injuries , Canada , Humans , United Kingdom , United StatesABSTRACT
Valid and reliable techniques for assessing performance are essential to surgical education, especially with the emergence of competency-based frameworks. Despite this, there is a paucity of adequate tools for the evaluation of skills required during joint replacement surgery. In this scoping review, we examine current methods for assessing surgeons' competency in joint replacement procedures in both simulated and clinical environments. The ability of many of the tools currently in use to make valid, reliable and comprehensive assessments of performance is unclear. Furthermore, many simulation-based assessments have been criticised for a lack of transferability to the clinical setting. It is imperative that more effective methods of assessment are developed and implemented in order to improve our ability to evaluate the performance of skills relating to total joint replacement. This will enable educators to provide formative feedback to learners throughout the training process to ensure that they have attained core competencies upon completion of their training. This should help ensure positive patient outcomes as the surgical trainees enter independent practice.
Subject(s)
Arthroplasty, Replacement/education , Clinical Competence , Education, Medical, Graduate/methods , Physicians/standards , Humans , Reproducibility of ResultsABSTRACT
BACKGROUND: Although the objective in European Union and North American surgical residency programmes is similar-to train competent surgeons-residents' working hours are different. It was hypothesized that practice-ready surgeons with more working hours would perform significantly better than those being educated within shorter working week curricula. METHODS: At each test site, 21 practice-ready candidate surgeons were recruited. Twenty qualified Canadian and 19 qualified Dutch surgeons served as examiners. At both sites, three validated outcome instruments assessing multiple aspects of surgical competency were used. RESULTS: No significant differences were found in performance on the integrative and cognitive examination (Comprehensive Integrative Puzzle) or the technical skills test (Objective Structured Assessment of Technical Skill; OSATS). A significant difference in outcome was observed only on the Patient Assessment and Management Examination, which focuses on skills needed to manage patients with complex problems (P < 0.001). A significant interaction was observed between examiner and candidate origins for both task-specific OSATS checklist (P = 0.001) and OSATS global rating scale (P < 0.001) scores. CONCLUSION: Canadian residents, serving many more working hours, perform equivalently to Dutch residents when assessed on technical skills and cognitive knowledge, but outperformed Dutch residents in skills for patient management. Secondary analyses suggested that cultural differences influence the assessment process significantly.
Subject(s)
Clinical Competence/standards , General Surgery/standards , Internship and Residency/standards , Canada , Culture , Humans , Netherlands , Personnel Staffing and SchedulingABSTRACT
BACKGROUND: Given that carotid artery stenosis (CAS) intervention is procedurally difficult, possesses an extensive learning curve, and involves a grave list of potential complications, construct validation of new non-clinical training devices is of increasing importance. PURPOSE: To evaluate the construct validity of the Procedicus-Virtual Interventional Simulator Trainer (Procedicus-VIST) and its use as a training tool. MATERIAL AND METHODS: Sixteen interventionalists (15 males, one female; mean interventional radiology [IR] experience >11 years) and 16 medical students (15 males, one female; no IR experience) received 1 hour of didactic instruction followed by an hour of familiarization training. Subjects then attempted to complete a carotid artery stenting procedure within 1 hour while their performance metrics were recorded. All participants completed a qualitative exit survey of subjective parameters using a visual analog scale. RESULTS: Procedure and fluoroscopic time was 8.7 and 8.7 min greater in the novice group (P=0.0066 and P=0.0031), respectively. There were no significant differences in performances between the two groups in the remaining metrics of cine loops (number recorded), tool/vessel ratio, coverage percentage, and placement accuracy or residual stenosis. Contrast measurement metrics were found to be too imprecise for statistical analysis. Experienced and novice opinions differed significantly for six of 10 subjective parameters. No statistically significant difference in video-gaming habits was demonstrated. CONCLUSION: With the exception of the metrics of performance time and fluoroscopic use, construct validity of the Procedicus-VIST carotid metrics were not confirmed. Virtual reality simulation as a training method was valued more by novices than by experienced interventionalists.
Subject(s)
Carotid Stenosis/therapy , Education, Medical/methods , Radiology, Interventional/education , Stents , User-Computer Interface , Carotid Stenosis/diagnostic imaging , Female , Fluoroscopy , Humans , MaleABSTRACT
BACKGROUND: Pilot studies of complex interventions such as a team checklist are an essential precursor to evaluating how these interventions affect quality and safety of care. We conducted a pilot implementation of a preoperative team communication checklist. The objectives of the study were to assess the feasibility of the checklist (that is, team members' willingness and ability to incorporate it into their work processes); to describe how the checklist tool was used by operating room (OR) teams; and to describe perceived functions of the checklist discussions. METHODS: A checklist prototype was developed and OR team members were asked to implement it before 18 surgical procedures. A research assistant was present to prompt the participants, if necessary, to initiate each checklist discussion. Trained observers recorded ethnographic field notes and 11 brief feedback interviews were conducted. Observation and interview data were analyzed for trends. RESULTS: The checklist was implemented by the OR team in all 18 study cases. The rate of team participation was 100% (33 vascular surgery team members). The checklist discussions lasted 1-6 minutes (mean 3.5) and most commonly took place in the OR before the patient's arrival. Perceived functions of the checklist discussions included provision of detailed case related information, confirmation of details, articulation of concerns or ambiguities, team building, education, and decision making. Participants consistently valued the checklist discussions. The most significant barrier to undertaking the team checklist was variability in team members' preoperative workflow patterns, which sometimes presented a challenge to bringing the entire team together. CONCLUSIONS: The preoperative team checklist shows promise as a feasible and efficient tool that promotes information exchange and team cohesion. Further research is needed to determine the sustainability and generalizability of the checklist intervention, to fully integrate the checklist routine into workflow patterns, and to measure its impact on patient safety.
Subject(s)
Communication , Operating Rooms , Patient Care Team , Safety Management , Feasibility Studies , Humans , Interprofessional Relations , Interviews as Topic , Pilot Projects , Time Factors , Vascular Surgical Procedures , WorkforceABSTRACT
BACKGROUND: Ineffective team communication is frequently at the root of medical error. The objective of this study was to describe the characteristics of communication failures in the operating room (OR) and to classify their effects. This study was part of a larger project to develop a team checklist to improve communication in the OR. METHODS: Trained observers recorded 90 hours of observation during 48 surgical procedures. Ninety four team members participated from anesthesia (16 staff, 6 fellows, 3 residents), surgery (14 staff, 8 fellows, 13 residents, 3 clerks), and nursing (31 staff). Field notes recording procedurally relevant communication events were analysed using a framework which considered the content, audience, purpose, and occasion of a communication exchange. A communication failure was defined as an event that was flawed in one or more of these dimensions. RESULTS: 421 communication events were noted, of which 129 were categorized as communication failures. Failure types included "occasion" (45.7% of instances) where timing was poor; "content" (35.7%) where information was missing or inaccurate, "purpose" (24.0%) where issues were not resolved, and "audience" (20.9%) where key individuals were excluded. 36.4% of failures resulted in visible effects on system processes including inefficiency, team tension, resource waste, workaround, delay, patient inconvenience and procedural error. CONCLUSION: Communication failures in the OR exhibited a common set of problems. They occurred in approximately 30% of team exchanges and a third of these resulted in effects which jeopardized patient safety by increasing cognitive load, interrupting routine, and increasing tension in the OR.
Subject(s)
Communication Barriers , Interprofessional Relations , Operating Rooms/standards , Patient Care Team/standards , Surgical Procedures, Operative/standards , Anesthesia Department, Hospital/standards , Humans , Medical Errors/prevention & control , Observation , Problem Solving , Quality Indicators, Health Care , Safety , Sentinel Surveillance , Surgery Department, Hospital/standards , Surgical Procedures, Operative/classification , Systems Analysis , Vascular Surgical Procedures/standardsABSTRACT
PURPOSE: Endovascular therapy has had a major effect on vascular surgery; surgeons perform tasks in three dimensions (3D) while viewing two-dimensional (2D) displays. This fundamental change in how surgeons perform operations has educational implications related to learning curves and patient safety. We studied the effects of experience, training, and visual-spatial ability on 3D perception of 2D angiographic images of abdominal aortic aneurysms (AAA). METHODS: A novel computer-based method was developed to produce 3D depth maps based on subjects' interpretations of 2D images. Seven experts (certified vascular surgeons) and 20 novices (medical or surgical trainees) were presented with a 2D AAA angiographic image. With software specifically designed for this study, a depth map representing each subject's 3D interpretation of the 2D angiogram was produced. The novices were then randomized into a control group and a treatment group, who received a 5-minute AAA anatomy educational session. All subjects repeated the exercise on a second AAA image. Finally, all novices were given tests of visual-spatial ability, including the Surface Development Test and the Mental Rotations Test. Comparisons between experts and novices were made with depth map comparison, a subject's perception of overall object contour. RESULTS: The depth maps were significantly different (depth map comparison, P <.001) between the expert and both novice groups for the first image. After the educational intervention, the control group and the treatment group exhibited significantly different depth maps (depth map comparison, P <.001), with treatment group depth maps more similar to those of the expert group. There were no significant correlations between the visual-spatial tests and the novice depth map comparison with the expert group. CONCLUSIONS: This is the first study to examine perception of endovascular images in an educational context. Perception of overall surface contour of 3D structures from 2D angiographic images is affected by experience and training. With application of methods of vision science to an important problem in surgery, this research represents a first step in understanding the nature of visual perceptual processes involved in execution of an increasingly common clinical task. These results have implications for understanding and studying the endovascular learning curve. CLINICAL RELEVANCE: This research represents a unique collaboration in an effort to understand and solve one of the greatest problems facing surgical educators and surgeons. This research uses applied tools in vision science to understand the perceptual constraints involved in minimally invasive surgery. Specifically, we examined the mental three-dimensional maps experts use when viewing two-dimensional displays. Furthermore, we compared experts with novices in an effort to assist surgical trainees.
Subject(s)
Image Interpretation, Computer-Assisted , Imaging, Three-Dimensional , Aortic Aneurysm, Abdominal/diagnostic imaging , Aortic Aneurysm, Abdominal/surgery , Canada , Clinical Competence , Depth Perception , Education, Medical, Graduate , Health Knowledge, Attitudes, Practice , Humans , Internship and Residency , Photic Stimulation , Radiographic Image Enhancement , Statistics as Topic , Treatment Outcome , Vascular Surgical Procedures/education , Visual PerceptionSubject(s)
Clinical Competence , General Surgery , Accreditation , General Surgery/education , Humans , Internship and Residency , United StatesABSTRACT
BACKGROUND: Decision making on the competency of surgical trainees to perform laparoscopic procedures has been hampered by the lack of reliable methods to evaluate operative performance. The goal of this study was to develop a feasible and reliable method of evaluation. METHODS: Twenty-nine senior surgical residents were videotaped performing a low anterior resection and a Nissen fundoplication in a pig. Ten blinded laparoscopists rated the videos independently on two scales. Rating time was minimized by allowing raters to fast-forward through the tapes at their discretion. Interrater reliability and the time required to rate a procedure were assessed. RESULTS: Rating time per procedure was a median of 15 min (range, 6-40). The mean interrater reliability for the two scales was 0.74. CONCLUSIONS: The use of videotapes of operations enabled multiple raters to assess a performance reliably and shortened assessment times by 80%. This assessment technique shows potential as a means of evaluating the performance of advanced laparoscopic procedures by surgical trainees.
Subject(s)
Clinical Competence , Internship and Residency , Laparoscopy/standards , Video Recording , Feasibility Studies , Reproducibility of ResultsABSTRACT
BACKGROUND: Inter-professional health care teams represent the nucleus of both patient care and the clinical education of novices. Both activities depend upon the'talk' that team members use to interact with one another. This study explored team members' interpretations of tense team communications in the operating room (OR). METHODS: The study was conducted using 52 team members divided into 14 focus groups. Team members comprised 13 surgeons, 19 nurses, nine anaesthetists and 11 trainees. Both uni-disciplinary (n = 11) and multi-disciplinary (n = 3) formats were employed. All groups discussed three communication scenarios, derived from prior ethnographic research. Discussions were audio-recorded and transcribed. Using a grounded theory approach, three researchers individually analysed sample transcripts, after which group discussions were held to resolve discrepancies and confirm a coding structure. Using the confirmed code, the complete data set was coded using the 'NVivo' qualitative data analysis software program. RESULTS: There were substantial differences in surgeons', nurses', anaesthetists', and trainees' interpretations of the communication scenarios. Interpretations were accompanied by subjects' depictions of disciplinary roles on the team. Subjects' constructions of other professions' roles, values and motivations were often dissonant with those professions' constructions of themselves. CONCLUSIONS: Team members, particularly novices, tend to simplify and distort others' roles and motivations as they interpret tense communication. We suggest that such simplifications may be rhetorical, reflecting professional rivalries on the OR team. In addition, we theorise that novices' echoing of role simplification has implications for their professional identity formation.
Subject(s)
Clinical Competence/standards , Interprofessional Relations , Patient Care Team , Verbal Behavior , Education, Medical/standards , Humans , OntarioABSTRACT
OBJECTIVE: Learning curves pose a difficult problem in the teaching of technical skills: how do you teach procedural skills without compromising patients' health? A simulator-based curriculum has been designed to minimize the risks to patients undergoing amniocentesis by shifting the learning curve away from patients and into the laboratory. This study evaluated the effectiveness of a high-fidelity simulator-based curriculum in improving the performance of amniocentesis by obstetric trainees. DESIGN: Thirty trainees received a course on the practice of amniocentesis. The curriculum consisted of a lecture, a syllabus, and a hands-on training session with the simulator. Pre- and post-training performance were evaluated with two rating scales. Training and performance evaluation were completed using the same simulator. The effectiveness of the simulator-based workshop and the effect of year of training were assessed using a two-way analysis of variance. RESULTS: Performance scores improved from a mean score of 55% to 94% using checklist scoring and from 57% to 88% using global ratings. The two-way analysis of variance revealed a significant effect of training (F1,60 = 43.57; P < 0.001) accounting for 45% of the variance in scores, and a significant effect of experience level (F2,60 = 9.16; P < 0.001) accounting for 25% of the variance in scores. CONCLUSIONS: A comprehensive curriculum based on a high-fidelity simulator was effective at improving skills demonstrated on the simulator. The challenge remains to establish that skills acquired on a simulator are transferable to the clinical setting.
Subject(s)
Amniocentesis/methods , Curriculum , Obstetrics/education , Patient Simulation , Teaching/methods , Clinical Competence , Education, Medical, Undergraduate , Female , Humans , Internship and Residency , Male , Models, Anatomic , Ontario , Pregnancy , Sensitivity and Specificity , Ultrasonography, PrenatalABSTRACT
PURPOSE: To develop a valid and reliable examination to assess the technical proficiency of family medicine residents' performance of minor surgical office procedures. METHOD: A multi-station OSCE-style examination using bench-model simulations of minor surgical procedures was developed. Participants were a randomly selected group of 33 family medicine residents (PGY-1 = 16, PGY-2 = 17) and 14 senior surgical residents who functioned as a validation group. Examiners were qualified surgeons and family physicians who used both checklists and global rating scales to score the participants' performances. RESULTS: When family medicine residents were evaluated by family physicians, interstation reliabilities were .29 for checklists and .42 for global ratings. When family medicine residents were evaluated by surgeons, the reliabilities were .53 for checklists and .75 for global ratings. Interrater reliability, measured as a correlation for total examination scores, was .97. Mean scores on the examination were 60%, 64%, and 87% for PGY-1 family medicine, PGY-2 family medicine, and surgery residents, respectively. The difference in scores between family medicine and surgery residents was significant (p < .001), providing evidence of construct validity. CONCLUSION: A new examination developed for assessing family medicine residents' skills with minor surgical office procedures is reliable and has evidence for construct validity. The examination has low reliability when family physicians serve as examiners, but moderate reliability when surgeons are the evaluators.
Subject(s)
Clinical Competence , Educational Measurement , Family Practice/education , Internship and Residency , Minor Surgical Procedures , Ambulatory Surgical Procedures , Analysis of Variance , Humans , Random Allocation , Reproducibility of ResultsABSTRACT
BACKGROUND: The Objective Structured Assessment of Technical Skill (OSATS) is a multistation performance-based examination that assesses the technical skills of surgery residents. This study explores the implementation issues involved in remote administration of the OSATS focusing on feasibility and the psychometric properties of the examination. METHODS: An eight-station OSATS was administered to surgical residents in Los Angeles and Chicago. The University of Toronto and the local institutions shared responsibility for organization and administration of the examination. RESULTS: There was good reliability for both the checklist (alpha = 0.68 for LA, 0.73 for Chicago) and global rating forms (alpha = 0.82 for both sites). Both iterations also showed evidence of construct validity, with a significant effect of training year for the checklist and global rating forms at both sites (analysis of variance: F = 8.66 to 19.93, P <0.01). Despite some challenges, the model of central organization and peripheral delivery was effective for the administration of the examinations. CONCLUSIONS: Two iterations of the OSATS at remote sites demonstrated psychometric properties that are highly consistent with previously reported data suggesting that the examination is portable. Both faculty and residents indicated satisfaction with the examination experience. A model of central administration with peripheral delivery was feasible and effective.
Subject(s)
Educational Measurement/methods , General Surgery/education , Internship and Residency , California , Clinical Competence/standards , Feasibility Studies , Illinois , PsychometricsABSTRACT
BACKGROUND: This study examined whether an operative product and time to completion could serve as measures of technical skill. METHODS: Nine final-year (PGY5) and 11 penultimate-year (PGY4) general surgery residents participated in a 6-station bench model examination. Time to completion was recorded. Twelve faculty surgeons (2 per station) evaluated the quality of the final product using a 5-point scale. RESULTS: The mean interrater reliability was 0. 59 for product quality. Interstation reliability was 0.59 for analysis of the final product and 0.72 for time to completion. There was 63% and 78% agreement between attendings' ratings and product quality and time scores respectively. PGY5s' mean product quality score was 4.14 +/- 0.26, compared with 3.82 +/- 0.33 for PGY4s (P < 0.05). PGY5s' mean time was 110 +/- 19 minutes compared with PGY4s' 132 +/- 15 (P < 0.05). CONCLUSIONS: Analysis of the operative end product and time to completion offer efficient alternatives to on-line examiner scoring for bench model examinations of technical competence.
Subject(s)
Benchmarking/standards , Clinical Competence/standards , General Surgery/education , Internship and Residency/standards , Feasibility Studies , Humans , Ontario , Reproducibility of ResultsABSTRACT
PURPOSE: This was a multicenter, randomized, controlled trial to compare the effectiveness of topical nitroglycerin with internal sphincterotomy in the treatment of chronic anal fissure. METHODS: Patients with symptomatic chronic anal fissures were randomly assigned to 0.25 percent nitroglycerin tid or internal sphincterotomy. Both groups received stool softeners and fiber supplements and were assessed at six weeks and six months. RESULTS: Ninety patients were accrued, but 8 were excluded from the analysis because they refused internal sphincterotomy after randomization (6), the fissure healed before surgery (1), or a fissure was not observed at surgery (1). There were 38 patients in the internal sphincterotomy group (22 males; mean age, 40.3 years) and 44 patients in the nitroglycerin group (15 males; mean age, 38.7 years). At six weeks 34 patients (89.5 percent) in the internal sphincterotomy group compared with 13 patients (29.5 percent) in the nitroglycerin group had complete healing of the fissure (P = 5x10(-8)). Five of the 13 patients in the nitroglycerin group relapsed, whereas none in the internal sphincterotomy group did. At six months fissures in 35 (92.1 percent) patients in the internal sphincterotomy group compared with 12 (27.2 percent) patients in the nitroglycerin group had healed (P = 3x10(-9)). One (2.6 percent) patient in the internal sphincterotomy group required further surgery for a superficial fistula compared with 20 (45.4 percent) patients in the nitroglycerin group who required an internal sphincterotomy (P = 9x10(-6)). Eleven (28.9 percent) patients in the internal sphincterotomy group developed side effects compared with 37 (84 percent) patients in the nitroglycerin group (P<0.0001). Nine (20.5 percent) patients discontinued the nitroglycerin because of headaches (8) or a severe syncopal attack (1). CONCLUSIONS: Internal sphincterotomy is superior to topical nitroglycerin 0.25 percent in the treatment of chronic anal fissure, with a high rate of healing, few side effects, and low risk of early incontinence. Thus, internal sphincterotomy remains the treatment of choice for chronic anal fissure.
Subject(s)
Anal Canal/surgery , Fissure in Ano/drug therapy , Fissure in Ano/surgery , Nitroglycerin/therapeutic use , Vasodilator Agents/therapeutic use , Administration, Topical , Adult , Aged , Female , Fissure in Ano/pathology , Humans , Male , Middle Aged , Nitroglycerin/administration & dosage , Treatment Outcome , Vasodilator Agents/administration & dosageABSTRACT
BACKGROUND: Two complimentary examinations designed to comprehensively assess competence for surgical practice have been developed. The Objective Structured Assessment of Technical Skill (OSATS) evaluates a resident's operative skill, and the Patient Assessment and Management Examination (PAME) evaluates clinical management skills. METHODS: Twenty-four postgraduate year (PGY)-4 and PGY-5 general surgery residents from four training programs were examined. Each examination had eight stations, with a total of 6 hours of testing time. RESULTS: Interstation reliability for the OSATS was 0.64, for the PAME was 0.71, and for the total test was 0. 74. Examination scores discriminated between PGY-4 and PGY-5 residents for the OSATS (t = 4.39, P <.01), the PAME (t = 1.86, P <. 05), and the total examination (t = 3.90, P <.01). Year of training accounted for 40%, of the variance of scores. CONCLUSIONS: This comprehensive examination is a reliable and valid method of assessing critical skills in senior surgical residents and may be useful for the formal assessment of readiness for practice.
Subject(s)
Clinical Competence , Educational Measurement/methods , General Surgery/education , Internship and Residency , Clinical Competence/standards , Educational Measurement/standards , Feasibility Studies , Humans , Internship and Residency/classification , Internship and Residency/standards , Ontario , Reproducibility of Results , Time FactorsABSTRACT
Epistaxis is a common problem faced by primary care physicians. Typically, first-hand experience with this problem is not obtained in medical school. A computer learning module was developed to address practical management of epistaxis. It was evaluated by use of third-year clerks and compared with an interactive seminar based on the same material. Fifty-eight students doing their required otolaryngology rotations were randomly separated into 3 groups: (1) doing a preinstruction test, (2) using the computer module, and (3) participating in a small-group seminar. All participants were tested with a short written test and a practical test that involved performing anterior nasal pack placement in a model patient. Percentage scores for the 2 groups were compared by use of t tests, and there was no significant difference between the written, practical, or combined scores at a level where P = 0.05. This study shows that basic patient management and a simple procedure can be taught as effectively with a computer module as with a small-group interactive seminar.