The Interprofessional Collaborative Competency Attainment Survey (ICCAS): A replication validation study.

This study replicates a validation of the Interprofessional Collaboration Competency Attainment Survey (ICCAS), a 20-item self-report instrument designed to assess behaviours associated with patient-centred, team-based, collaborative care. We appraised the content validity of the ICCAS for a foundation course in interprofessional collaboration, investigated its internal (factor) structure and concurrent validity, and compared results with those obtained previously by ICCAS authors. Self-assessed competency ratings were obtained from a broad spectrum of pre-licensure, health professions students (n = 785) using a retrospective, pre-/post-design. Moderate to large effect sizes emerged for 16 of 20 items. Largest effects (1.01, 0.94) were for competencies emphasized in the course; the smallest effect (0.35) was for an area not directly taught. Positive correlations were seen between all individual item change scores and a separate item assessing overall change, and item-total correlations were moderate to strong. Exploratory factor analysis was used to understand the interrelationship of ICCAS items. Principal component analysis identified a single factor (Cronbach's alpha = 0.96) accounting for 85% of the total variance-slightly higher than the 73% reported previously. Findings suggest strong overlaps in the proposed constructs being assessed; use of a total average score is justifiable for assessment and evaluation.

Learning by (video) example: a randomized study of communication skills training for end-of-life and error disclosure family care conferences.

BACKGROUND: Teaching residents to lead end of life (EOL) and error disclosure (ED) conferences is important. METHODS: We developed and tested an intervention using videotapes of EOL and error disclosure encounters from previous Objective Structured Clinical Exams. Residents (n = 72) from general and orthopedic surgery programs at 2 sites were enrolled. Using a prospective, pre-post, block group design with stratified randomization, we hypothesized the treatment group would outperform the control on EOL and ED cases. We also hypothesized that online course usage would correlate positively with post-test scores. RESULTS: All residents improved (pre-post). At the group level, treatment effects were insignificant, and post-test performance was unrelated to course usage. At the subgroup level for EOL, low performers assigned to treatment scored higher than controls at post-test; and within the treatment group, post graduate year 3 residents outperformed post graduate year ​1 residents. CONCLUSIONS: To be effective, online curricula illustrating communication behaviors need face-to-face interaction, individual role play with feedback and discussion.

Unanticipated teaching demands rise with simulation training: strategies for managing faculty workload.

INTRODUCTION: Using simulation to teach and assess learners represents a powerful approach to training, but one that comes with hidden costs in terms of faculty time, even if programs adopt existing curricula. Some simulators are built to be used independently by learners, but much of the surgical simulation curricula developed for cognitive and psychomotor tasks requires active faculty involvement and low learner-to-faculty teaching ratios to ensure sufficient practice with feedback. The authors hypothesize that the added teaching demands related to simulation have resulted in a significant financial burden to surgery training programs. To date, the effect of simulation-based training on faculty workload has not been estimated objectively and reported in the literature. METHODS: To test their hypothesis, the authors analyzed data from 2 sources: (1) changes over time (2006-2014) in formal teaching hours and estimated faculty costs at the University of Minnesota, General Surgery Department and (2) a 2014 online survey of general surgery program directors on their use of simulation for teaching and assessment and their perceptions of workload effects. RESULTS: At the University of Minnesota, the total number of hours spent by department faculty in resident and student simulation events increased from 81 in annual year 2006 to 365 in annual year 2013. Estimated full-time equivalent faculty costs rose by 350% during the same period. Program directors (n = 48) of Association of Program Directors in Surgery reported either a slight (60%) or a significant (33%) increase in faculty workload with the advent of simulation, and moderate difficulty in finding enough instructors to meet this increase. Calling upon leadership for support, using diverse instructor types, and relying on "the dedicated few" represent the most common strategies. CONCLUSION: To avoid faculty burnout and successfully sustain faculty investment in simulation-based training over time, programs need to be creative in building, sustaining, and managing the instructor workforce.

Replicating an established open skills curriculum: are the same results obtained in a different setting?

INTRODUCTION: The aim of this study was to ascertain whether the University of Texas, Southwestern (UTSW) open skills curriculum (Goova et al.(13)) could be successfully implemented in the University of Minnesota (UMN) multisite training program. We posed 4 questions: (1) Is the curriculum feasible? (2) Did residents' skills improve? (3) Did UMN residents achieve the same level as UTSW residents? (4) What factors were associated with posttest success? METHODS: Postgraduate year-1 residents (n = 22) were enrolled in the curriculum, which included orientation, access to video instruction, 3 months of independent practice using portable kits and practice logbooks, presurveys and postsurveys, and pretesting and posttesting. Evaluation was based on UTSW proficiency measures (time, errors, and total proficiency score). Descriptive statistics, paired sample t tests, analysis of variance, and bivariate correlations were calculated. Results were compared with Goova. RESULTS: Startup costs at UMN were $9804 vs $776 at UTSW. Our curriculum required 51 direct faculty hours vs 376 at UTSW. UMN trainees' skills improved significantly (mean score = 973 [standard deviation = 267] at baseline vs 1325 [standard deviation = 215] at posttest), but they achieved proficiency in only 38.6% of tasks at posttest, compared with 88.7% by UTSW trainees. Best predictors of UMN posttest proficiency score were (1) categorical vs preliminary resident status (p < 0.001), (2) pretest proficiency score (r = 0.510, p = 0.008), and (3) self-assessed baseline proficiency (r = 0.415, p = 0.027). Participation in skills laboratories during clerkship or fourth year medical school, estimated number of cases (surgeon or first assistant), and number of practice repetitions recorded in booklets were not predictive of the posttest score. CONCLUSIONS: The UTSW open skills curriculum is feasible and effective in a new setting. Differences from UTSW-published success rates may be related to their superior onsite monitoring of practice and a policy requiring residents to achieve proficiency for each task before posttesting.

How much guidance is given in the operating room? Factors influencing faculty self-reports, resident perceptions, and faculty/resident agreement.

BACKGROUND: Guidance in the operating room impacts resident confidence and ability to function independently. The purpose of this study was to explore attending surgeon guidance practices in the operating room as reported by faculty members themselves and by junior and senior residents. METHODS: This was an exploratory, cross-sectional survey research study involving 91 categorical residents and 82 clinical faculty members at two academic general surgery training programs. A series of analyses of variance along with descriptive statistics were performed to understand the impact of resident training year, program, and surgeon characteristics (sex and type of surgery performed routinely) on guidance practices. RESULTS: Resident level (junior versus senior) significantly impacted the amount of guidance given as reported by faculty and as perceived by residents. Within each program, junior residents perceived less guidance than faculty reported giving. For senior guidance practices, however, the differences between faculty and resident practices varied by program. In terms of the effects of surgeon practice type (mostly general versus mostly complex cases), residents at both institutions felt they were more supervised closely by the faculty who perform mostly complex cases. CONCLUSION: More autonomy is given to senior than to junior residents. Additionally, faculty report a greater amount of change in their guidance practices over the training period than residents perceive. Faculty and resident agreement about the need for guidance and for autonomy are important for achieving the goals of residency training.

Surgical skills acquisition: performance of students trained in a rural longitudinal integrated clerkship and those from a traditional block clerkship on a standardized examination using simulated patients.

OBJECTIVES: Rural longitudinal integrated clerkship (LIC) programs for third-year medical students provide strong educational curricula and can nurture interest in rural surgical practice. Students learn technical skills in an apprenticeship model. Variability in instruction and patient experiences across sites, coupled with a lack of simulation facilities, raise some concerns about technical skill development. To explore the adequacy of skills acquisition for students in the University of Minnesota Rural Physician Associate Program (RPAP), this study compared RPAP students' performance on a scenario-based Objective Structured Assessment of Technical Skills (OSATS) with that of traditional surgery block clerkship students (Course 7500). DESIGN, SETTING, AND PARTICIPANTS: This is a nonexperimental post-only study. All enrolled students (n = 254) completed the OSATS examination. Students in the Course 7500 (n = 222) completed 15 hours of simulation skills training and supervised practice during their 6-week clerkship. RPAP students (n = 32) completed 3 hours of skills training before their 9-month rural assignment. Both groups had access to comprehensive online materials. Mean OSATS checklist, global rating, and total scores were compared at the end of training using t tests (p < 0.05). Self-reported OR and clinical experiences were explored. RESULTS: Both groups did well on the OSATS. There were no statistical differences in completion time, checklist scores, mean global ratings, or total scores. RPAP students reported significantly more days in the OR, surgery cases, and first assists. Experience with OSATS tasks reported by RPAP students during clinical rotations correlated with their OSATS performance. CONCLUSION: This study supports the viability of the LIC model for fundamental skills acquisition when augmented with introductory simulation skills training and online resources. It also suggests that simulation fills a training gap for students in a traditional surgery block clerkship program. It opens a dialog about the potential partnership of surgery departments with rural LICs to address rural general surgery shortages. Further research in this aspect is needed.

Reliability and validity of a test designed to assess combat medics' readiness to perform life-saving procedures.

OBJECTIVES: Reducing preventable deaths because of uncontrolled hemorrhage, tension pneumothorax, and airway loss is a priority. As part of a research initiative comparing different training models, this study evaluated the reliability and validity of a test that assesses combat medic performance during a polytrauma scenario using live animal models. METHODS: Nine procedural checklists and seven global rating scales were piloted with four cohorts of soldiers (n = 94) at two U.S. training sites. Cohorts represented "novice" to "proficient" trainees. Procedure scores and a mean global score were calculated per subject. The intraclass correlation was calculated per procedure, with 0.70 as the threshold for acceptability. An overall difference among cohorts was hypothesized: Cohort 4 (proficient) > Cohort 3 (competent) > Cohort 2 (beginners) > Cohort 1 (novice) trainees. Data were analyzed using Kruskal-Wallis and analysis of variance. RESULTS: At Site A, intraclass correlation coefficients ranged from 74% to 93% for 6 of 9 procedures. Cohorts differed significantly on hemorrhage control, needle decompression, cricothyrotomy, amputation management, chest tube insertion, and mean global scores. Cohort 4 outperformed the others, and Cohorts 2 and 3 outperformed Cohort 1. CONCLUSION: The test differentiates novices from beginners, competent, and proficient trainees on difficult procedures and overall performance.

Development and verification of a taxonomy of assessment metrics for surgical technical skills.

PURPOSE: To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. METHOD: In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. RESULTS: The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. CONCLUSIONS: Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

Colorectal surgeons teaching general surgery residents: current challenges and opportunities.

Effective teaching for general surgery residents requires that faculty members with colorectal expertise actively engage in the education process and fully understand the current context for residency training. In this article, we review important national developments with respect to graduate medical education that impact resident supervision, curriculum implementation, resident assessment, and program evaluation. We argue that establishing a culture of respect and professionalism in today's teaching environment is one of the most important legacies that surgical educators can leave for the coming generation. Faculty role modeling and the process of socializing residents is highlighted. We review the American College of Surgeons' Code of Professional Conduct, summarize some of the current strategies for teaching and assessing professionalism, and reflect on principles of motivation that apply to resident training both for the trainee and the trainer.

Being an effective surgical educator.

Training in thoracic surgical residencies has evolved in the past several years, with significant advances in simulation technology, heightened pressure regarding work-hour reforms, and initiation of integrated training programs. This article highlights current concepts in surgical education and methods of incorporating teaching opportunities into practice. General strategies on how to be a better teacher and increase student feedback evaluation scores are addressed. Finally, the evolving roles and responsibilities of a mentor in assisting residents and colleagues in developing successful thoracic surgical careers are explored.

A multi-institutional study of the Family Conference Objective Structured Clinical Exam: a reliable assessment of professional communication.

BACKGROUND: To test the value of a simulated Family Conference Objective Structured Clinical Exam (OSCE) for resident assessment purposes, we examined the generalizability and construct validity of its scores in a multi-institutional study. METHODS: Thirty-four first-year (PG1) and 27 third-year (PG3) surgery residents (n = 61) from 6 training programs were tested. The OSCE consisted of 2 cases (End-of-Life [EOL] and Disclosure of Complications [DOC]). At each program, 2 clinicians and 2 standardized family members rated residents using case-specific tools. Performance was measured as the percentage of possible score obtained. We examined the generalizability of scores for each case separately. To assess construct validity, we compared PG1 with PG3 performance using repeated measures multivariate analysis of variance (MANOVA). RESULTS: The relative G-coefficient for EOL was .890. For DOC, the relative G-coefficient was .716. There were no significant performance differences between PG1 and PG3 residents. CONCLUSIONS: This OSCE provides reliable assessments suitable for formative evaluation of residents' interpersonal communication skills and professionalism.

Preimplementation predictors of website use: preliminary findings from the SCORE portal pilot study.

BACKGROUND: in 2008, the Surgical Council on Resident Education selected 33 residency programs to pilot its General Surgery Resident Curriculum Website Portal. The portal aims to reduce program variability in curricula, align teaching and learning with essential content, and improve resident study and performance. METHODS: two online surveys were sent to all program directors and their residents before releasing the portal. Data from 32 programs and 899 residents (84%) were analyzed to determine the extent to which preimplementation characteristics supported the portal's rationale and illuminated barriers to its use and impact on learning. RESULTS: the need for curriculum content and access to online texts varied markedly across programs. Residents had easy onsite access to the Internet and used it heavily for immediate purposes. Fewer residents used the Web for planned activities and proactive study. On average, residents reported studying an hour or less a day. CONCLUSIONS: the portal appears to serve curricular resource needs and may better direct resident study. Programs are advised to consciously integrate the SCORE curriculum and portal into residency training and faculty development.

Synthesis versus imitation: evaluation of a medical student simulation curriculum via Objective Structured Assessment of Technical Skill.

BACKGROUND: Given the investment that programs make to simulation training, it is important to evaluate its effects on student learning. Tasks (e.g., gowning and gloving, suturing) are typically taught in isolation over a series of linked sessions. This study assessed students' ability to integrate such tasks while executing an unrehearsed procedure before and after a new simulation curriculum was introduced. METHODS: An Objective Structured Assessment of Technical Skill (OSATS) was administered to 26 students in the 2007 clerkship who received a 3-hour orientation to the operating room followed by a 3-hour animate laboratory, and to 167 students in the 2008 clerkship who received a 9-hour simulation skills curriculum. The OSATS task involved a live volunteer "patient" with an arm laceration. Students had 40 minutes to explain the procedure, start an intravenous line, administer a local anesthetic, prepare the wound (pig's foot), gown and glove, and suture the wound. The OSATS was scored by trained raters using a tool with 57 checklist and 7 global rating items. Its internal consistency reliability was 0.82. Independent sample t tests were used to analyze differences between "pre" and "post" groups. RESULTS: Mean scores were significantly higher for the post group for the checklist score (83% vs 62%, p < 0.001), the average global item score (3.62 vs 3.07, p = 0.003) and the OSATS total score (79% correct vs 62%, p < 0.001). Students from both groups were weakest in maintaining a sterile field, motion, and flow. Although superior, post group students still struggled with organizing a plan of action when faced with an unrehearsed procedure. CONCLUSIONS: The revised curriculum had a positive impact on students' mastery of basic surgical skills, despite the loss of the animal laboratory. Implications for instruction include greater use of discovery-learning techniques to teach productive versus reproductive skills.

Improving surgery intern confidence through the implementation of expanded orientation sessions.

BACKGROUND: New surgical interns may be unprepared for job-related tasks and harbor anxiety that could interfere with job performance. To address these problems, we extended our intern orientation with the principal aim of demonstrating the need for expanded instruction on execution of daily tasks. Additionally, we sought to show that an enriched orientation curriculum durably augments intern confidence. METHODS: Twenty-one surgical interns participated in an extended orientation program, consisting of interactive didactics, case scenario presentations, and small group discussions. Evaluations collected at completion of orientation and 1-month follow-up assessed self-reported confidence levels on job-related tasks before, immediately afterward, and 1-month after orientation. Statistical analyses were performed using Student t tests (P < .05 significant). RESULTS: Self-reports of confidence on job-related tasks before the orientation sessions were low; however, program participation resulted in immediate confidence increases in all areas. Evaluations at 1-month follow-up showed persistence of these gains. CONCLUSION: Interns reported considerable anxiety in all job-related tasks before orientation. After the sessions, confidence levels were significantly and durably improved in all areas. Our findings suggest the need for specific instruction on job-related tasks of surgical internship and demonstrate the effectiveness of an expanded orientation in improving intern confidence in execution of these tasks.

Structured teaching versus experiential learning of palliative care for surgical residents.

BACKGROUND: Previous end-of-life and palliative care curricula for surgical residents have shown improved learner confidence, but have not measured cognitive knowledge or skill acquisition. METHODS: A nonrandomized trial evaluated a structured palliative care curriculum for 7 postgraduate year 2 surgical residents (intervention group) compared with 6 postgraduate year 5 surgical residents (comparison group). Outcomes were measured using an 18-item knowledge test, a 20-minute objective structured clinical examination simulating an intensive care unit family conference, and a survey measuring self-confidence. RESULTS: The mean knowledge test scores for the intervention group, both before and after undergoing the structured palliative care curriculum, were no different from the comparison group. There was also no difference in objective structured clinical examination scores between the 2 groups. The intervention group felt less comfortable managing pain, breaking bad news, or addressing ethical issues. CONCLUSIONS: Junior surgical residents have similar palliative care knowledge to senior residents without a palliative care curriculum. After participating in a palliative care curriculum, they have simulated skills that are similar to chief residents. However, self-confidence is lower among junior residents despite undergoing a palliative care curriculum.

Establishing a usable electronic portfolio for surgical residents: trying to keep it simple.

Elaborate web-based portfolios may not be needed for resident evaluation and career development. An approach for busy academic medical departments is described.

Rotation as a course: lessons learned from developing a hybrid online/on-ground approach to general surgical resident education.

PURPOSE: To improve the consistency and the quality of resident education on clinical rotations, 5 surgical rotations (thoracic, bariatrics, surgical oncology, pediatrics, and critical care) were restructured "as courses" with learning objectives, educational activities (online and on-ground), pretests, posttests, and oral examinations. SETTING/PARTICIPANTS: University surgical training program in a large metropolitan area, which serves approximately 65 residents per year. METHODS: The online course management system, WebCT/VISTA (Blackboard Inc., Washington, DC), was used to build 5 online course sites. To engage and garner support from faculty, several organizational change tactics and resources were employed, such as Grand Rounds presentations, a faculty retreat, consultation and support from professional staff, and the use of residents as reviewers and codevelopers. To support resident use of the online sites, a designated education coordinator provided individual and group orientation sessions and employed weekly tracking and reminder systems; completion of pretests and posttests was mandated. RESULTS: Between 6 and 8 learning modules were created per rotation, with over 50 reading assignments (collectively) and 45 online presentations. Since July 2006, 53 residents have completed a total of 106 rotations on these services. Preliminary results from a longitudinal study suggest that the hybrid approach is well received and effective when fully executed, but that online course materials are used by residents only if they feel that the faculty members are truly engaged and actively promoting the site. CONCLUSIONS: Changing the culture of learning on rotation to include learning objectives, assessment, and integrated online/on-ground activities takes significant leadership, resident input, professional staff support, faculty engagement, and time.