Establishing an Orthopedic Program-Specific, Comprehensive Competency-Based Education Program.

BACKGROUND: Competency-based education (CBE) seeks to determine resident proficiency in the knowledge, skills, and behaviors required for independent patient care. Multiple assessment instruments evaluate technical skills or direct patient care in the clinic setting, but there are few reports incorporating both within an orthopedic specialty rotation. This study reports a residency program's comprehensive CBE initiative using formative assessments in the clinic and operating room during a sports medicine rotation. MATERIALS AND METHODS: The sports medicine rotation used validated formative assessments to evaluate resident performance during clinic encounters and program-defined surgical entrustable professional activities (EPAs). Junior resident (postgraduate year [PGY] 1-2) EPAs included basic knee/shoulder arthroscopic procedures. Senior resident (PYG 5) EPAs comprised anterior cruciate ligament reconstruction, biceps tenodesis, shoulder stabilization, and rotator cuff repair. Assessment scores were compared between individuals and PGY groups. RESULTS: Sixty-six clinical skills (CS) and 106 surgical skills assessments were conducted for 22 residents in one academic year. Surgical skills assessments demonstrated significant differences between each PGY group (P < 0.01). All PGY2 and PGY5 residents achieved independence on the evaluated EPAs. PGY5s earned higher scores in CS assessments than the other classes (P < 0.01). PGY2 residents scored higher than PGY1s in 7 of 9 CS domains. CS independence was achieved by 21 of 22 residents by the end of the rotation. CONCLUSIONS: The CBE program effectively quantified expected differences in resident performance by PGY for clinic and surgical assessments on a sports medicine rotation. Assessments built an environment where feedback was more structured and standardized, creating a culture to improve resident education.

Chest Tube Insertion Among Surgical and Nonsurgical Trainees: How Skilled Are Our Residents?

BACKGROUND: Competency-based medical education has renewed focus on the attainment and evaluation of resident skill. Proper evaluation is crucial to inform educational interventions and identify residents in need of increased training and supervision. Currently, there is a paucity of studies rigorously evaluating resident chest tube insertion skill. MATERIALS AND METHODS: Residents of all training levels before their intensive care unit rotation or currently rotating through the intensive care unit were invited to participate. Trainees inserted a thoracostomy tube on a high-fidelity simulator. Their performances were recorded and scored by blinded raters using the validated TUBE-iCOMPT rubric. Surgical and nonsurgical residents were compared. RESULTS: Forty-nine residents participated; 30 from nonsurgical and 19 from surgical training programs. Overall, trainees were most deficient in the "preprocedural checks" and "patient positioning and local anesthetic" domains. Surgical trainees demonstrated higher chest tube insertion skill than their nonsurgical peers (median total score 88 [interquartile range, 74-90] versus 75 [interquartile range, 66-85], respectively, P = 0.01), particularly in the "patient positioning" and "blunt dissection" domains (P = 0.01 and P = 0.03, respectively). These differences were no longer significant when controlled for experience and Advanced Trauma Life Support certification. CONCLUSIONS: Overall, surgical residents were more skilled than nonsurgical residents in tube thoracostomy placement. Relative skill deficits within the domains of chest tube insertion have also been identified among residents of different specialties. These areas can be targeted with educational interventions to improve resident performance, and ultimately, patient safety.

The future of psychiatric education.

Mental health disorders cause a large burden on global public health, with many patients living years with the disability. However, many doctors are ill-equipped to treat mental health disorders given inadequate training during their undergraduate years. In some countries, psychiatry is even considered an elective course rather than a core module. There is, therefore, a pressing need to improve the training of medical students in managing mental health disorders. Measures need to be implemented to attract students to choose psychiatry as their career. Given the developments in the fields and the challenges currently faced by trainees and early career psychiatrists, changes may also be made to the training programme in the postgraduate stage to unify the variations across the world in terms of the training duration and format. This paper will describe the ways that undergraduate and postgraduate psychiatry training may be ameliorated to improve the delivery of mental healthcare around the world and to equip doctors to face challenges in the future.

Programmatic assessment: Can we provide evidence for saturation of information?

Purpose: According to the principles of programmatic assessment, a valid high-stakes assessment of the students' performance should amongst others, be based on a multiple data points, supposedly leading to saturation of information. Saturation of information is generated when a data point does not add important information to the assessor. In establishing saturation of information, institutions often set minimum requirements for the number of assessment data points to be included in the portfolio. Methods: In this study, we aimed to provide validity evidence for saturation of information by investigating the relationship between the number of data points exceeding the minimum requirements in a portfolio and the consensus between two independent assessors. Data were analyzed using a multiple logistic regression model. Results: The results showed no relation between the number of data points and the consensus. This suggests that either the consensus is predicted by other factors only, or, more likely, that assessors already reached saturation of information. This study took the first step in investigating saturation of information, further research is necessary to gain in-depth insights of this matter in relation to the complex process of decision-making.

Do they teach what they need to? An analysis of the impact of curriculum mapping on the learning objectives taught in a lecture series in surgery.

Introduction: Curriculum mapping shows concordances and differences between the intended and the taught curriculum. To our knowledge, no previous studies describe the effects that this mapping has on the curriculum. The aim of the present study is to map the content of a lecture series in surgery to the National Catalogue of Learning Objectives in Surgery and analyze the effects this mapping has on the content of the following lecture series. Methods: All lecturers in the lecture series were directly observed by a minimum of two reviewers and learning objectives and the level of competence were documented. After the lecture series, the results were visualized within the catalog of learning objectives and were sent to the lecturers. In the following lecture series, learning objectives were documented correspondingly. Results: In the first lecture series, 47% of the learning objectives were taught. After the mapping, the number of learning objectives that were taught increased to 59% (p < 0.001). The increase was found in all surgical disciplines and in all levels of competences without any changes in the average duration of the lectures. Conclusions: The presented method for mapping a curriculum effectively increased the number of taught learning objectives without requiring longer lecture durations.

Integrating behaviour change counselling into chronic disease management: a square peg in a round hole? A system-level exploration in primary health care.

OBJECTIVES: The objective of this study is to evaluate the uptake of competency-based behaviour change counselling training within a primary healthcare setting. Specific questions concerning provider readiness for training, perceived importance of training in the context of service demands and perceptions of competence after training were addressed. STUDY DESIGN: A process-focused study which adopted a complex systems approach to implementation. Each step was evaluated before the next step was developed. The design was guided by the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework. METHODS: Four specific primary care services were identified and behaviour change counselling training tailored to each service was provided, based on a model of training built around competencies in establishing change-based relationships, assessing and promoting readiness to change, using evidence-based behaviour modification skills when ready and addressing psychosocial determinants of behaviour within scope of practice. Before training, a manager's readiness to facilitate training and identification of peer leaders to support ongoing practice of skills were completed. RESULTS: Two programs negotiated 8 h of formal training, one program received 10 h and one program received 12 h. All programs engaged in peer support activities. Despite willingness to support training, 90% of managers were ambivalent about training activities, relative to one half of healthcare providers (HCPs). Few HCPs and no managers self-identified as ready without ambivalence. Furthermore, HCPs were reluctant to be evaluated by an expert and preferred self-evaluation methods. In contrast, HCPs uniformly endorsed the relevance, value and professional commitment to all component skills of the behaviour change counselling model. At the end of the training, over 75% of staff reported receiving formal training (reach). Almost 80% of staff reported using change-based relationship skills daily, with less frequent use of skills associated with addressing psychosocial issues. The degree of corrective feedback was generally low, however. An index of competency based on formal training, frequent use and receiving corrective feedback indicated that most HCPs did not meet these criteria. CONCLUSION: Training in behaviour change counselling competencies was successfully implemented in this project. The vast majority of HCPs received training, despite ambivalence. Furthermore, HCPs strongly valued these skills and used them frequently. However, they were reluctant to accept corrective feedback. Future research is needed to evaluate innovative strategies to overcome obstacles to receiving corrective feedback in the use of behaviour change counselling skills.

Video feedback and e-Learning enhances laboratory skills and engagement in medical laboratory science students.

BACKGROUND: Traditionally, the training of medical laboratory science students has taken place in the laboratory and has been led by academic and pathology experts in a face-to-face context. In recent years, budgetary pressures, increasing student enrolments and limited access to laboratory equipment have resulted in reduced staff-student contact hours in medical laboratory science education. While this restructure in resources has been challenging, it has encouraged innovation in online blended learning. METHODS: Blended learning histology lessons were implemented in a face-to-face and e-Learning format in a medical laboratory science program to teach tissue morphology and technical procedures outside of the traditional laboratory classroom. Participating students were randomly allocated to either the 'video' group (n = 14) or the 'control' group (n = 14). After all students attempted the e-Learning lessons and viewed expert-led video recordings online, students demonstrated their hands-on practical skills in the laboratory. Technical skills, demonstration of safety awareness, and use of histology equipment was captured by video through first person 'point of view' recordings for the 'video' group only. The 'control' group performed the same activities but were not recorded. Prior to summative assessment, the 'video' group students had a digital resource portfolio that enabled them to review their skills, receive captured feedback and retain a visual copy of their recorded procedure. RESULTS: Results showed that students who participated in the online video format had statistically better practical examination scores and final grades compared to the control group. CONCLUSION: Findings from this study suggest that students are engaged and motivated when being taught in a blended learning format and respond positively to the use of video recordings with expert feedback for the initial learning of hands-on techniques. For the academic, developing a blended learning medical laboratory science program, which includes annotated virtual microscopy, video demonstrations, and online interactive e-Learning activities, provides an effective and economic approach to learning and teaching.

Informing the medical education reform in Tajikistan: evidence on the learning environment at two nursing colleges.

BACKGROUND: The Tajik medical education system is undergoing a complex reform to enhance the transition of the healthcare system from its soviet legacy of emphasizing secondary level care/specialisation to become more family medicine and primary health care oriented. The current study presents the first empirical evaluation of the educational environment for nursing students in Tajikistan using the Dundee Ready Educational Environment Measure (DREEM). The study results contribute to the benchmarking efforts of monitoring and positively steering the educational environment over time. METHOD: The study was based on a cross-sectional survey involving 630 nursing students at two nursing colleges in Tajikistan. Students' perception of the learning environment was measured using the DREEM. Internal consistency was examined using Cronbach's alpha. General scores were calculated and measured against international benchmarks. Data was further interpreted by comparing DREEM scores between students of different sex, at different colleges and different study years using T tests. RESULTS: Cronbach's alpha ranged from 0.30 to 0.75 with an overall alpha of 0.89. General DREEM scores were slightly above average compared to similar studies with nursing students in other countries. In particular, results showed that students' academic self-perception and teachers' technical competences were generally favourably rated. Teachers' pedagogical skills were critically perceived by the study participants and teaching was generally viewed as too teacher-centred with an over-emphasis on factual learning. CONCLUSIONS: Statistical results indicated acceptable levels of reliability of the DREEM tool when applied to the Tajik nursing educational context. Students rated the learning environment as generally satisfactory with average scores similar or slightly higher than comparable scores from similar studies involving nursing students. However, the on-going educational reform could have placed more emphasis on developing faculty pedagogical skills in nursing schools. Teaching approaches would benefit from being more competency based rather than so heavily focused on factual knowledge.

How accurate are we? A comparison of resident and staff physician billing knowledge and exposure to billing education during residency training

Background: Practice management is an overlooked and undertaught subject in medical education. Many physicians feel that their exposure to billing education during residency training was inadequate. The purpose of this study was to compare resident and staff physicians in terms of their billing knowledge and exposure to billing education during residency training. Methods: Senior residents and staff physicians completed a scenario-based clinical billing assessment. Posttest surveys were completed to determine exposure to practice management and billing education during training. Results: A total of 16 resident physicians and 17 staff physicians completed the billing assessment. Overall, the billing accuracy of respondents was poor. Staff physicians had a greater percentage of correct billing codes (55.3% v. 37.5%, p < 0.001) and underbilled codes (6.2% v. 3.4%, p = 0.009), with fewer missed billing codes (38.5% v. 59.1%, p < 0.001), compared with resident physicians. The percentage value of correct billings was significantly higher for staff physicians (71.5% v. 56.8%, p = 0.01). In the posttest survey, 100.0% of residents and 79.0% of staff physicians desired more billing education during training. Conclusion: In general, staff physicians billed more accurately than resident physicians, but even experienced staff physicians missed a substantial amount of potential revenue because of billing errors and omissions. The majority of the residents and staff physicians who participated in our study felt that current billing education is both insufficient and ineffective. Incorporating practice management and billing education into residency training is critical to ensure that the next generation of medical trainees possess the financial competence to required to manage a successful medical practice.

Contexte: La gestion médicale est un sujet souvent oublié et trop peu enseigné durant les études de médecine. Beaucoup de médecins ont l'impression que la formation sur la facturation offerte durant leur résidence était insuffisante. L'objectif de cette étude était de comparer les connaissances sur la facturation et l'exposition, durant la résidence, à la formation sur ce sujet des résidents et des médecins membres du personnel. Méthodes: Les résidents seniors et les médecins membres du personnel ont effectué une évaluation de facturation clinique à partir de mises en situation. Ils ont répondu à un sondage après le test pour déterminer leur exposition à la formation sur la gestion médicale et la facturation durant leurs études. Résultats: Au total, 16 médecins résidents et 17 médecins membres du personnel ont fait l'évaluation de facturation. Dans l'ensemble, l'exactitude de leur facturation était faible. Les médecins membres du personnel avaient un pourcentage plus élevé de codes de facturation corrects (55,3 % contre 37,5 %, p < 0,001) et de codes de facturation insuffisants (6,2 % contre 3,4 %, p = 0,009), et avaient moins de codes manquants (38,5 % contre 59,1 %, p < 0,001), comparativement aux médecins résidents. Le pourcentage de facturations correctes était significativement plus élevé chez les médecins membres du personnel (71,5 % contre 56,8 %, p = 0,01). Dans le sondage post-test, 100,0 % des résidents et 79,0 % des médecins membres du personnel désiraient avoir davantage de formation sur la facturation durant les études. Conclusion: En général, les médecins membres du personnel ont produit des factures plus exactes que les médecins résidents, mais même des médecins membres du personnel expérimentés ont perdu des revenus potentiels considérables en raison d'erreurs de facturation et d'omissions. La majorité des résidents et des médecins membres du personnel qui ont participé à l'étude avaient l'impression que la formation actuelle sur la facturation était à la fois insuffisante et inefficace. Il est essentiel d'intégrer la formation sur la gestion médicale et la facturation dans la résidence pour garantir que la prochaine génération de futurs médecins possède les compétences financières nécessaires pour gérer un cabinet prospère.

Unexpected result of competency-based medical education: 9-year application trends to enhanced skills programs by family medicine residents at a single institution in Canada.

In Canada, family physicians may pursue extended training for added competence in areas such as Palliative Care or Emergency Medicine by applying to Enhanced Skills (ES) programmes. Despite the increasing popularity of ES programmes, there are no studies that examine trends in applications to ES programmes before and after the introduction of family medicine focused competency-based curricula at individual universities. Additionally, there is a scarcity of research examining factors common among applicants to ES programmes. We undertook a retrospective observational study using secondary data analysis of archived resident files from a large Canadian family medicine residency programme. The proportion of applicants to ES programmes decreased since implementation of a competency-based curriculum in the subject programme. Older, male, and Canadian medical graduates (CMGs) applied to ES programmes more often than their respective counterparts. Residents in a family medicine competency-based curriculum may be less inclined to extend their training by applying to ES programmes. This is remarkable considering that the Canadian residency programme is the shortest among high-income countries. Further studies are needed to investigate the role of competency-based medical education programmes in enabling shortening residency training around the world.

Demographics and study practices of psychiatry trainees sitting the RANZCP written examinations.

OBJECTIVES: This study explored trainee preparation for the Royal Australian and New Zealand College of Psychiatrists (RANZCP) written examinations. We examined the relationship between candidates' prior psychiatry experience, preparation techniques and expenses, perceptions of the exam and a pass result. We also examined perceptions of well-being. METHOD: A web-based survey was sent to RANZCP trainees. The association between responses and exam outcome was analysed using SPSS Statistics 23. RESULTS: A total of 23% ( n = 38) of the cohort responded to the survey. Respondents studied for an average of 7.8 months for the essay-style exam and 4.4 months for the multiple-choice questions. The mean months of total psychiatry experience prior to sitting was 36.1 months. Every additional 50 hours of study increased odds of passing by 23%. Respondents who believed that exam preparation would affect outcome had an almost 4-fold higher odds of passing than those who did not. Ratings of well-being before release of results were also positively associated with passing. Conclusions Trainees are sitting the written exams with clinical experience commensurate with RANZCP recommendations. Total study hours and favourable perceptions of exam processes appear to be predictors of success.

Assessment of knowledge concerning urology among medical students in Poland.

BACKGROUND AND OBJECTIVE: Urological diseases represent a significant health issue worldwide. Presented study aimed at assessing current urological knowledge and confidence in performing urological diagnostic and therapeutic procedures among medical students at Jagiellonian University Medical College in Poland and compare it on different stages of the undergraduate medical education. MATERIAL AND METHODS: We designed an anonymous survey distributed among Polish students from 1st to 6th year of medical studies, before and after clinical urology course. Questions concerned general urological knowledge, prostate diseases, erectile dysfunction, and self-reported practical urological skills. RESULTS: Overall, 437 respondents participated in the survey. Mean total test score in our study group was 50.08%, mean general urological knowledge score was 53.44%, mean prostate diseases knowledge score was 55.43%, mean erectile dysfunction score was 36% and mean practical skills score was 45.83%. Mean total test score increased with consecutive years of studies (R = 0.58; p <0.001). The risk of an above average total test score was significantly influenced by the urology course (OR = 7.95, 95%CI = 1.81-34.84, p = 0.006) and the year of medical studies (4th-6th vs. 1st-3rd) (OR = 5.16, 95%CI = 3.41-7.81, p <0.001). Practical skills score above average was significantly more frequent in the group after the urology course (OR = 6.75, 95%CI = 1.54-29.58, p = 0.011). CONCLUSIONS: Results of this study reveal low mean scores obtained by students, even after completing the urology course, which implies that curriculum requires further development. Urological knowledge and self-assessed practical skills increased with years of medical education. The urology course improved the score obtained in our survey, both in terms of total test score and practical skills.

[Association between self-directed learning behaviors, socio-demographic and academic variables among medical students]. / Aprendizaje autodirigido en estudiantes de primer año de medicina de la Universidad de Concepción y su relación con el perfil sociodemográfico y académico.

BACKGROUND: Medical education must encourage autonomous learning behaviors among students. However the great income profile disparity among university students may influence their capacity to acquire such skills. AIM: To assess the association between self-directed learning, socio-demographic and academic variables. MATERIAL AND METHODS: The self-directed learning readiness scale was applied to 202 medical students aged between 17 and 25 years (64% males). Simultaneously information about each surveyed participant was obtained from the databases of the medical school. RESULTS: There is an association between socio-demographic and academic variables with the general scale of self-directed learning and the subscales learning planning and willingness to learn. Participants coming from municipal schools have a greater willingness to learn than their counterparts coming from subsidized and private schools. High school grades are related to self-directed learning and the subscales learning planning and self-assessment. CONCLUSIONS: Among the surveyed medical students, there is a relationship between self-directed learning behaviors, the type of school where they come from and the grades that they obtained during high school.

Disaster curricula in medical education: pilot survey.

INTRODUCTION: An understanding of disaster medicine and the health care system during mass-casualty events is vital to a successful disaster response, and has been recommended as an integral part of the medical curriculum by the Association of American Medical Colleges (AAMC). It has been documented that medical students do not believe that they have received adequate training for responding to disasters. The purpose of this pilot study was to determine the inclusion of disaster medicine in the required course work of medical students at AAMC schools in the United States, and to identify the content areas addressed. METHODS: An electronic on-line survey was developed based upon published core competencies for health care workers, and distributed via e-mail to the education liaison for each medical school in the United States that was accredited by the AAMC. The survey included questions regarding the inclusion of disaster medicine in the medical school curricula, the type of instruction, and the content of instruction. RESULTS: Of the 29 (25.2%) medical schools that completed the survey, 31% incorporated disaster medicine into their medical school curricula. Of those schools that included disaster medicine in their curricula, 20.7% offered disaster material as required course work, and 17.2% offered it as elective course work. Disaster medicine topics provided at the highest frequency included pandemic influenza/severe acute respiratory syndrome (SARS, 27.5%), and principles of triage (10.3%). The disaster health competency included most frequently was the ability to recognize a potential critical event and implement actions at eight (27.5%) of the responding schools. CONCLUSIONS: Only a small percentage of US medical schools currently include disaster medicine in their core curriculum, and even fewer medical schools have incorporated or adopted competency-based training within their disaster medicine lecture topics and curricula.

Identifying Entrustable Professional Activities for Shared Decision Making in Postgraduate Medical Education: A National Delphi Study.

PURPOSE: Although shared decision making (SDM) is considered the preferred approach in medical decision making, it is currently not routinely used in clinical practice. To bridge the transfer gap between SDM training and application, the authors aimed to reach consensus on entrustable professional activities (EPAs) for SDM and associated behavioral indicators as a framework to support self-directed learning during postgraduate medical education. METHOD: Using existing literature on SDM frameworks and competencies; input from an interview study with 17 Dutch experts in SDM, doctor-patient communication, and medical education; and a national SDM expert meeting as a starting point, in 2017, the authors conducted a modified online Delphi study with a multidisciplinary Dutch panel of 32 experts in SDM and medical education. RESULTS: After 3 Delphi rounds, consensus was reached on 4 EPAs-(1) the resident discusses the desirability of SDM with the patient, (2) the resident discusses the options for management with the patient, (3) the resident explores the patient's preferences and deliberations, and (4) the resident takes a well-argued decision together with the patient. Consensus was also reached on 18 associated behavioral indicators. Of the 32 experts, 30 (94%) agreed on this list of SDM EPAs and behavioral indicators. CONCLUSIONS: The authors succeeded in developing EPAs and associated behavioral indicators for SDM for postgraduate medical education to improve the quality of SDM training and the application of SDM in clinical practice. These EPAs are characterized as process EPAs for SDM in contrast with content EPAs related to diverse medical complaints. A next step is the implementation of the SDM EPAs in existing competency-based workplace curricula.

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.

BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Use of competency-based metrics to determine effectiveness of a postgraduate thoracoscopy course.

BACKGROUND: Despite the paradigm shift from process to competency-based education, no study has explored how competency-based metrics might be used to assess short-term effectiveness of thoracoscopy-related postgraduate medical education. OBJECTIVES: To assess the use of a single-group, pre-/post-test model comprised of multiple-choice questions (MCQ) and psychomotor skill measures to ascertain the effectiveness of a postgraduate thoracoscopy program. METHODS: A 37-item MCQ test of cognitive knowledge was administered to 17 chest physicians before and after a 2-day continued medical education-approved program. Pre- and post-course technical skills were assessed using rigid videothoracoscopy simulation stations. Competency-based metrics (mean relative gain, mean absolute gain, and class-average normalized gain ) were calculated. A >30% was used to determine curricular effectiveness. RESULTS: Mean cognitive knowledge score improved significantly from 20.9 to 28.7 (7.8 ± 1.3 points, p < 0.001), representing a relative gain of 37% and an absolute gain of 21%. Mean technical skill score improved significantly from 5.20 to 7.82 (2.62 ± 0.33 points, p < 0.001), representing a relative gain of 50% and an absolute gain of 33%. Non-parametric testing confirmed t test results (p < 0.001). Class-average normalized gains were 48 and 92%, respectively. CONCLUSION: Competency-based metrics, including class-average normalized gain, can be used to assess course effectiveness and to determine if a program meets predesignated objectives of knowledge acquisition and psychomotor technical skill.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation.

PURPOSE: Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change. METHOD: Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences. RESULTS: In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow. CONCLUSIONS: Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Educational impact of the mini-Clinical Evaluation Exercise in resident standardization training: a comparative study between resident and professional degree postgraduate trainees.

OBJECTIVE: We aimed to explore differences in the educational impact of the mini-Clinical Evaluation Exercise (mini-CEX) on resident (RE) and professional degree postgraduate (PDPG) trainees, as well as influencing factors, to provide suggestions for hospital managers, trainers, and trainees. METHODS: We performed a retrospective analysis of all scores among first-year resident standardization training trainees registered during 2017 to 2019 at Xinqiao Hospital of Army Medical University, to identify differences in mini-CEX outcomes between REs and PDPGs. RESULTS: We collected data of 154 registered trainees for retrospective analysis, including 57 PDPG trainees and 97 RE trainees. The mean (standard deviation) overall performance score of PDPGs was 84.18 (4.25), which was higher than that of REs (81.48 (3.35)). In terms of domain analysis, PDPG trainees performed significantly better than REs in history taking, physical examination, clinical diagnosis/treatment regimen, and the knowledge examination; communication skills/humanistic care were comparable between the groups. CONCLUSIONS: PDPGs performed better than REs in overall competency, history taking, physical examination, clinical diagnosis/treatment regimen, and the knowledge examination. A better knowledge base, supervisor-dominated one-to-one teaching mode, higher self-esteem and learning goals, and more sophisticated responses to feedback were potential contributors to a superior educational impact of the mini-CEX.

Association of Gender With Learner Assessment in Graduate Medical Education.

Importance: Gender bias may affect assessment in competency-based medical education. Objective: To evaluate the association of gender with assessment of internal medicine residents. Design, Setting, and Participants: This multisite, retrospective, cross-sectional study included 6 internal medicine residency programs in the United States. Data were collected from July 1, 2016, to June 30, 2017, and analyzed from June 7 to November 6, 2019. Exposures: Faculty assessments of resident performance during general medicine inpatient rotations. Main Outcomes and Measures: Standardized scores were calculated based on rating distributions for the Accreditation Council for Graduate Medical Education's core competencies and internal medicine Milestones at each site. Standardized scores are expressed as SDs from the mean. The interaction of gender and postgraduate year (PGY) with standardized scores was assessed, adjusting for site, time of year, resident In-Training Examination percentile rank, and faculty rank and specialty. Results: Data included 3600 evaluations for 703 residents (387 male [55.0%]) by 605 faculty (318 male [52.6%]). Interaction between resident gender and PGY was significant in 6 core competencies. In PGY2, female residents scored significantly higher than male residents in 4 of 6 competencies, including patient care (mean standardized score [SE], 0.10 [0.04] vs 0.22 [0.05]; P = .04), systems-based practice (mean standardized score [SE], -0.06 [0.05] vs 0.13 [0.05]; P = .003), professionalism (mean standardized score [SE], -0.04 [0.06] vs 0.21 [0.06]; P = .001), and interpersonal and communication skills (mean standardized score [SE], 0.06 [0.05] vs 0.32 [0.06]; P < .001). In PGY3, male residents scored significantly higher than female patients in 5 of 6 competencies, including patient care (mean standardized score [SE], 0.47 [0.05] vs 0.32 [0.05]; P = .03), medical knowledge (mean standardized score [SE], 0.47 [0.05] vs 0.24 [0.06]; P = .003), systems-based practice (mean standardized score [SE], 0.30 [0.05] vs 0.12 [0.06]; P = .02), practice-based learning (mean standardized score [SE], 0.39 [0.05] vs 0.16 [0.06]; P = .004), and professionalism (mean standardized score [SE], 0.35 [0.05] vs 0.18 [0.06]; P = .03). There was a significant increase in male residents' competency scores between PGY2 and PGY3 (range of difference in mean adjusted standardized scores between PGY2 and PGY3, 0.208-0.391; P ≤ .002) that was not seen in female residents' scores (range of difference in mean adjusted standardized scores between PGY2 and PGY3, -0.117 to 0.101; P ≥ .14). There was a significant increase in male residents' scores between PGY2 and PGY3 cohorts in 6 competencies with female faculty and in 4 competencies with male faculty. There was no significant change in female residents' competency scores between PGY2 to PGY3 cohorts with male or female faculty. Interaction between faculty-resident gender dyad and PGY was significant in the patient care competency (ß estimate [SE] for female vs male dyad in PGY1 vs PGY3, 0.184 [0.158]; ß estimate [SE] for female vs male dyad in PGY2 vs PGY3, 0.457 [0.181]; P = .04). Conclusions and Relevance: In this study, resident gender was associated with differences in faculty assessments of resident performance, and differences were linked to PGY. In contrast to male residents' scores, female residents' scores displayed a peak-and-plateau pattern whereby assessment scores peaked in PGY2. Notably, the peak-and-plateau pattern was seen in assessments by male and female faculty. Further study of factors that influence gender-based differences in assessment is needed.