The rapid growth and expansion of Master of health professions education (MHPE) programs: A mixed-methods study of international curricular trends and guidelines for programs.

PURPOSE: This study explores international trends and standards of Master's degree programs through a comprehensive environmental scan and focus group interviews to understand curricular structure, content, program director expectations, educational context, and future directions. METHOD: Authors conducted a two-phase mixed-methods sequential explanatory design to conduct the environmental scan (phase 1), and subsequently conducting focus groups (phase 2) with program directors. A population list of Master's programs was used to generate a sampling frame, considering the geographic region (continent) and institution type (university, organization, public institution). Qualitative data were coded to analyze the breadth and depth of courses. Three one-hour virtual focus group interviews were conducted with ten program directors. RESULTS: The population list of 159 Masters programs worldwide was used to create a sample for analysis in the environmental scan (n = 46 Masters programs), representing programs from North America, Europe, Australia, and South Africa. Most programs (39%) delivered their courses online, with 20% exclusively offering an in-person program. Focus group participants indicated expectations of graduates, context in which they learn, as well as future directions for improving health professions education graduate programs. CONCLUSION: Program directors should consider programmatic aims, localized needs, and quality/standard of the program in designing Masters programs, with individualized growth opportunities for learners.

The Inconspicuous Learner Handover: An Exploratory Study of U.S. Emergency Medicine Program Directors' Perceptions of Learner Handovers from Medical School to Residency.

Phenomenon: Central to competency-based medical education is the need for a seamless developmental continuum of training and practice. Trainees currently experience significant discontinuity in the transition from undergraduate (UME) to graduate medical education (GME). The learner handover is intended to smooth this transition, but little is known about how well this is working from the GME perspective. In an attempt to gather preliminary evidence, this study explores U.S. program directors (PDs) perspective of the learner handover from UME to GME. Approach: Using exploratory qualitative methodology, we conducted semi-structured interviews with 12 Emergency Medicine PDs within the U.S. from October to November, 2020. We asked participants to describe their current perception of the learner handover from UME to GME. Then we performed thematic analysis using an inductive approach. Findings: We identified two main themes: The inconspicuous learner handover and barrier to creating a successful UME to GME learner handover. PDs described the current state of the learner handover as "nonexistent," yet acknowledged that information is transmitted from UME to GME. Participants also highlighted key challenges preventing a successful learner handover from UME to GME. These included: conflicting expectations, issues of trust and transparency, and a dearth of assessment data to actually hand over. Insights: PDs highlight the inconspicuous nature of learner handovers, suggesting that assessment information is not shared in the way it should be in the transition from UME to GME. Challenges with the learner handover demonstrate a lack of trust, transparency, and explicit communication between UME and GME. Our findings can inform how national organizations establish a unified approach to transmitting growth-oriented assessment data and formalize transparent learner handovers from UME to GME.

The Impact of Simulation Based Training on the Fundamentals of Endoscopic Surgery Performance Examination.

OBJECTIVE: To determine if simulation training is required to pass the FES skills test and assess the relationship between simulation training, clinical training, and FES skills test performance. SUMMARY OF BACKGROUND DATA: The ABS began requiring completion of the Flexible Endoscopy Curriculum for all applicants beginning in 2018. The role of simulation-based training in FES skills test performance after this requirement has not been evaluated. METHODS: De-identified data from the initial FES skills tests after the Flexible Endoscopy Curriculum requirement was reviewed, and 731 unique participants with reported simulation experience demographics were identified. Self-reported data included sex, upper (UE) and lower (LE) endoscopy experience, and simulator training hours (SE). Final FES skills exam scores and pass/fail designations for each participant were reported by the FES program staff. RESULTS: There was a statistically discernible difference in mean FES total scores between those reporting no SE and more experienced groups ( P = 0.002), and between less and more experienced UE and LE groups ( P < 0.001). There was no statistically discernible difference in FES skills exam pass rates between SE groups ( P = 0.2), but there was a strong relationship between clinical experience (UE & LE) and pass rate ( P < 0.001). Finally, on logistic regression analysis, LE was a discernible predictor of passing [odds ratio (OR) = 1.4, 95% confidence interval (CI) 1.1-1.8, P = 0.02], while UE [odds ratio (OR) = 1, 95% CI 0.8-1.3, P = 0.9] and SE (OR = 1,95% CI 0.9- 1.3, P = 0.7) were not. CONCLUSIONS: There is no threat to the validity of the FES skills test from a need for simulation training to pass the FES skills test. Similarly, the amount of simulation practice is not predictive of passing, but can improve performance on certain FES tasks.

The Use of Learning Analytics to Enable Detection of Underperforming Trainees: An Analysis of National Vascular Surgery Trainee ACGME Milestones Assessment Data.

OBJECTIVE: This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets. SUMMARY BACKGROUND DATA: ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation. METHODS: National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 subcompetencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings. RESULTS: A total of 122 vascular surgery fellows (VSFs) (95.3%) and 52 integrated vascular surgery residents (IVSRs) (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6% to 25.4% across subcompetencies, which was not significantly different between the 2 groups for any of the subcompetencies ( P = 0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of 2.5, with 1 year remaining before graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% to 77.9% for VSFs and 33.3% to 75.0% for IVSRs. CONCLUSION: The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

Learning transfusion medicine through scoring objective structured clinical examination.

BACKGROUND: Transfusion medicine education at the undergraduate level is typically limited in duration. In view of limitations of traditional teaching methods, we explore effectiveness of scoring (Objective Structured Clinical Examination) OSCE as an educational method. MATERIALS AND METHODS: The study was of a randomized interventional three group pre-test-post-test design. Participants were undergraduate medical students in their two final years. The intervention was watching and scoring 2 videotaped OSCE stations about obtaining consent for blood transfusions and assessing the ability to explain risks, benefits, and alternatives of blood transfusion. Participants were asked to assess the performance of the videotaped actor using checklists. Participants were randomized to watch and evaluate one set of videos at either the highest, intermediate, or lowest compliance with required consent elements. Main measure was performance in a knowledge test containing multiple-choice and true/false questions. This was given before (pre-test), immediately after the intervention (post-test 1), and after 8 weeks (post-test 2). Student perceptions regarding the intervention was assessed immediately after the session. RESULTS: Sixty-nine students were randomized. Post-test 1 results (mean 16.52, SD 1.88) were significantly greater than pre-test results (mean 11.83, SD 2.13) by group and across all groups (p < 0.001). Post-test 2 results for the complete cohort showed maintenance of significant improvement in comparison with the pre-test. The majority of students agreed that learning through scoring OSCE was an effective educational experience. CONCLUSIONS: In the undergraduate medical setting, scoring OSCE stations may enhance learning of content discussed and evaluated in the stations.

Surgical skills training for practicing surgeons founded on established educational theories and frameworks.

Following completion of structured training in residency or fellowship programs, surgeons need to acquire new skills throughout their careers to address changes in their surgical practices, and to perform new procedures and use new technologies as these are developed. The opportunities available to surgeons to acquire the new skills, safely introduce these skills into surgical practice, and then maintain the skills, vary greatly and may not fully support a surgeon's needs. The article shares background information relating to the vexing problems, and describes established educational theories and frameworks as well as specific frameworks relating to surgical skills training and verification that may be used in designing and implementing comprehensive and impactful surgical skills training programs aimed at practicing surgeons. A number of practical strategies based on these theories and frameworks are highlighted. National professional organizations and academic medical centers need to play a critical role in pursuing such efforts, which will remain pivotal in supporting the professional work of surgeons and in providing optimal surgical care well into the future.

Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs.

BACKGROUND: Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions. METHODS: Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses. RESULTS: 687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame. CONCLUSIONS: Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

Quality Assessment of Online Resources for the Most Common Cancers.

The internet is a common source of health information for patients with cancer. Despite research surrounding the quality of online resources for individual types of cancer, these results may not necessarily be easily extrapolated to cancer resources as a whole. Thus, we aim to use a standardized tool to produce generalizable results by analyzing the quality of online resources for the most common cancers. Educational websites pertaining to breast, lung, prostate, and colorectal cancers were searched using multiple search engines. After screening against pre-specified inclusion criteria, the most visible 100 websites for each cancer were extracted for analysis. A validated tool was then used to assess their quality. Pooled results were evaluated using descriptive and inferential statistics. Of the 400 analyzed websites, 43% were commercially affiliated, and these were significantly associated with greater use of biased language. Thirty percent of websites disclosed authorship, 47% cited at least one reliable source, and 43% were updated within the last 2 years. The average Flesch-Kincaid readability was determined to be at a grade 10.9 level, which is significantly more difficult than the recommended grade 6 level. Risk factors, symptoms, and detection were the most accurately covered topics. However, most websites did not cover prognosis. This study comprehensively examines the quality of online cancer resources for the four most common cancers. Our results could help guide the development of future resources, support patient education endeavors, and raise awareness among healthcare providers about the limitations of online cancer resources.

Exploring knowledge, attitudes and perceived myths towards foetal programming and factors affecting it among the public and health professionals in Pakistan: a needs assessment study.

OBJECTIVE: To explore knowledge, attitudes and practices of laypersons and health professionals towards foetal programming, and factors affecting it. METHODS: The mixed methods study was conducted at the Aga Khan University, Karachi, from January 20, 2021 to May 13, 2022, and comprised adults of either gender with access to social media platforms. Data was collected using an online survey questionnaire in English and Urdu developed to capture responses from a diverse pool of participants. The survey tool was circulated through WhatsApp, Facebook and Instagram. Two focus group discussions were conducted; one with laypersons in group A and the other with health and allied professionals in group B. Data was analysed using SPSS 21, while data related to focus group discussions was subjected to thematic analysis. RESULTS: Of the 358 participants, 173(48.3%) were in group A and 185(51.7%) were in group B. There were 34(18.4) subjects in group A and 27(15.6) in group B who had knowledge of foetal programming (p>0.05). Only factors related to father's health and dietary elements on the foetus were significantly different between the groups (p<0.05). Thematic analysis led to the formation of 3 overarching themes: parent's lifestyle, comorbidity and diet on foetal health; myths and cultural beliefs regarding foetal development; and the need for training / awareness for practitioners and community. CONCLUSIONS: Lack of knowledge and misinformation about foetal programming and development was common among health professionals and laypersons.

Expert consensus on the conceptual alignment of Accreditation Council for Graduate Medical Education competencies with patient outcomes after common vascular surgical procedures.

BACKGROUND: The quality and effectiveness of vascular surgery education should be evaluated based on patient care outcomes. To investigate predictive associations between trainee performance and subsequent patient outcomes, a critical first step is to determine the conceptual alignment of educational competencies with clinical outcomes in practice. We sought to generate expert consensus on the conceptual alignment of the Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery subcompetencies with patient care outcomes across different Vascular Quality Initiative (VQI) registries. METHODS: A national panel of vascular surgeons with expertise in both clinical care and education were recruited to participate in a modified Delphi expert consensus building process to map ACGME Vascular Surgery subcompetencies (educational markers of resident performance) to VQI clinical modules (patient outcomes). A master list of items for rating was created, including the 31 ACGME Vascular Surgery subcompetencies and 8 VQI clinical registries (endovascular abdominal aortic aneurysm repair, open abdominal aortic aneurysm, thoracic endovascular aortic repair, carotid endarterectomy, carotid artery stent, infrainguinal, suprainguinal, and peripheral vascular intervention). These items were entered into an iterative Delphi process. Positive consensus was reached when 75% or more of the participants ranked an item as mandatory. Intraclass correlations (ICCs) were used to evaluate consistency between experts for each Delphi round. RESULTS: A total of 13 experts who contributed to the development of the Vascular Surgery Milestones participated; 12 experts (92%) participated in both rounds of the Delphi process. Two rounds of Delphi were conducted, as suggested by excellent expert agreement (round 1, ICC = 0.79 [95% confidence interval, 0.74-0.84]; round 2, ICC = 0.97 [95% confidence interval, 0.960-.98]). Using the predetermined consensus cutoff threshold, the Delphi process reduced the number of subcompetencies mapped to patient care outcomes from 31 to a range of 9 to 15 across the 8 VQI clinical registries. Practice-based learning and improvement, and professionalism subcompetencies were identified as less relevant to patient outcome variables captured by the VQI registries after the final round, and the only the systems-based practice subcompetency that was identified as relevant was radiation safety in two of the endovascular registries. CONCLUSIONS: A national panel of vascular surgeon experts reported a high degree of agreement on the relevance of ACGME subcompetencies to patient care outcomes as captured in the VQI clinical registry. Systems-based practice, practice-based learning and improvement, and professionalism competencies were identified as less relevant to patient outcomes after specific surgical procedures.

Patient involvement in assessment of postgraduate medical learners: A scoping review.

CONTEXT: Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent. METHODS: We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment. RESULTS: We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise. CONCLUSION: Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

Using Learning Analytics to Examine Achievement of Graduation Targets for Systems-Based Practice and Practice-Based Learning and Improvement: A National Cohort of Vascular Surgery Fellows.

BACKGROUND: Surgeons provide patient care in complex health care systems and must be able to participate in improving both personal performance and the performance of the system. The Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery Milestones are utilized to assess vascular surgery fellows' (VSF) achievement of graduation targets in the competencies of Systems Based Practice (SBP) and Practice Based Learning and Improvement (PBLI). We investigate the predictive value of semiannual milestones ratings for final achievement within these competencies at the time of graduation. METHODS: National ACGME milestones data were utilized for analysis. All trainees entering the 2-year vascular surgery fellowship programs in July 2016 were included in the analysis (n = 122). Predictive probability values (PPVs) were obtained for each SBP and PBLI sub-competencies by biannual review periods, to estimate the probability of VSFs not reaching the recommended graduation target based on their previous milestones ratings. RESULTS: The rate of nonachievement of the graduation target level 4.0 on the SBP and PBLI sub-competencies at the time of graduation for VSFs was 13.1-25.4%. At the first time point of assessment, 6 months into the fellowship program, the PPV of the SBP and PBLI milestones for nonachievement of level 4.0 upon graduation ranged from 16.3-60.2%. Six months prior to graduation, the PPVs across the 6 sub-competencies ranged from 14.6-82.9%. CONCLUSIONS: A significant percentage of VSFs do not achieve the ACGME Vascular Surgery Milestone targets for graduation in the competencies of SBP and PBLI, suggesting a need to improve curricula and assessment strategies in these domains across vascular surgery fellowship programs. Reported milestones levels across all time point are predictive of ultimate achievement upon graduation and should be utilized to provide targeted feedback and individualized learning plans to ensure graduates are prepared to engage in personal and health care system improvement once in unsupervised practice.

Constructing Approaches to Entrustable Professional Activity Development that Deliver Valid Descriptions of Professional Practice.

Issue: Entrustable Professional Activities (EPAs) describe the core tasks health professionals must be competent performing prior to promotion and/or moving into unsupervised practice. When used for learner assessment, they serve as gateways to increased responsibility and autonomy. It follows that identifying and describing EPAs is a high-stakes form of work analysis aiming to describe the core work of a profession. However, hasty creation and adoption of EPAs without rigorous attention to content threatens the quality of judgments subsequently made from using EPA-based assessment tools. There is a clear need for approaches to identify validity evidence for EPAs themselves prior to their deployment in workplace-based assessment. Evidence: For EPAs to realize their potential in health professions education, they must first be constructed to reflect accurately the work of that profession or specialty. If the EPAs fail to do so, they cannot predict a graduate's readiness for or future performance in professional practice. Evaluating the methods used for identification, description, and adoption of EPAs through a construct validity lens helps give leaders and stakeholders of EPA development confidence that the EPAs constructed are, in fact, an accurate representation of the profession's work. Implications: Application of a construct validity lens to EPA development impacts all five commonly followed steps in EPA development: selection of experts; identification of candidate EPAs; iterative revisions; evaluation of proposed EPAs; and formal adoption of EPAs into curricula. It allows curricular developers to avoid pitfalls, bias, and common mistakes. Further, construct validity evidence for EPA development provides assurance that the EPAs adopted are appropriate for use in workplace-based assessment and entrustment decision-making.

Entrustment decisions: Implications for curriculum development and assessment.

With increased interest in the use of entrustable professional activities (EPAs) in undergraduate and postgraduate medical education, comes questions about their implications for curriculum development and assessment. This paper addresses some of those questions, discussed at a symposium of the 2017 conference of AMEE, by presenting the components of an EPA, describing their importance and application, identifying their implications for assessment, and pinpointing some of challenges they pose in undergraduate and postgraduate settings. It defines entrustment, describes the three levels of trust, and presents trainee and supervisor factors that influence it as well as perceived benefits, and risks. Two aspects of EPAs have implications for assessment: units of professional practice and decisions based on entrustment, which impact an assessment's blueprint, test methods, scores, and standards. In an undergraduate setting EPAs have great appeal, but work is needed to identify and develop a robust assessment system for core EPAs. At the postgraduate level, there is tension between the granularity of the competencies and the integrated nature of the EPAs. Even though work remains, EPAs offer an important step in the evolution of competency-based education.

Senior-Year Curriculum in U.S. Medical Schools: A Scoping Review.

Phenomenon: Many U.S. medical schools have responded to the adoption of competency-based medical education (CBME) frameworks by renewing their final-year curricula and including internship preparatory courses. The purpose of this scoping review was to map the published literature regarding the final year to discern how medical schools have responded to this paradigm change. Approach: A structured 5-step approach was used to conduct this scoping review. Electronic searches of PubMed, ERIC, Scopus, MedEdPortal, and 8 medical education journals were conducted to identify relevant articles published from 2006 to 2016. Four authors screened articles for inclusion using standardized eligibility criteria; interrater agreement was discussed and calculated. Authors extracted data elements, and a consensus-based approach was used to categorize, sort, and structure information gathered. Findings: Among 6,485 articles retrieved, 817 articles were included in the study. From 2007-2011 to 2012-2016, articles addressing the final year of medical school increased 93%, whereas articles describing internship preparatory courses increased 218%. The majority of articles did not reference a CBME framework (572/817; 70%), the frequency of mentions increased 268% from 2007-2011 to 2012-2016. Nearly three fourths of preparatory course-related papers reference a CBME framework (37/50; 74%). Insights: Our findings may reflect a movement in U.S. medical schools toward using shared assessment metrics to support 4th-year medical students' preparation for their postgraduate medical education. Despite an increased reference to CBME, there is relatively little use of competency-based assessment frameworks to evaluate learning outcomes. This review also found a substantial increase in the frequency of articles referencing senior-year preparatory courses in U.S. medical schools. Articles tended to describe institution-specific research or experiences. Opportunities likely exist for multi-institutional and organizational collaboration in determining the structure of the final year of medical school.

Managing the tension: From innovation to application in health professions education.

Innovations in education are essential for solving problems and introducing new ways of thinking. However, implementation of these innovations must take several factors into consideration, including the context, the environment, the stakeholders, the technology needed, the cost, the pace of implementation, appropriateness, and available resources. When these factors are not balanced and considered, tensions arise. This paper describes tensions in five major educational domains, namely curriculum, instruction, assessment, accreditation/regulation, and faculty development. For each of these domains, the typical problems are described, and solutions are presented to manage the tension.

Medical assessment in the age of digitalisation.

BACKGROUND: Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed. METHODS: In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified. RESULTS: The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions. CONCLUSIONS: It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.

Understanding Assessment Systems for Clinical Competency Committee Decisions: Evidence from a Multisite Study of Psychiatry Residency Training Programs.

OBJECTIVE: This multisite study examines how clinical competency committees in Psychiatry synthesize resident assessments to inform milestones decisions to provide guidelines that support their use. METHODS: The study convened training directors and associate training directors from three psychiatry residency programs to examine decision-making processes of clinical competency committees. Annual resident assessments for one second year and one third year resident were used in a mock clinical competency committee format to assign milestones for two consecutive reporting periods. The committees reflected on the process and rated how the assessment tools impacted the assessment of milestones and evaluated the overall process. The authors compared reliability of assessment between the mock committees and examined both reliability of end of rotation assessments and their composite scores when combined with clinical skills evaluations. RESULTS: End of rotation evaluations were the most informative tool for assigning milestones and clarifying discrepancies in performance. In particular, the patient care and medical knowledge competencies were the easiest to rate, while the systems-based practice and practice-based learning and improvement were the most difficult. Reliability between committees was low although higher number of available evaluations improved reliability in decision-making. CONCLUSIONS: The results indicate that the medical knowledge and patient care competencies are the easiest to rate and informed most by end of rotation evaluations and clinical skills examinations. Other evaluation tools may better capture performance on specific sub-competencies beyond workplace-based assessment, or it may be helpful to reconsider the utility of how individual sub-competencies are evaluated.

Determining defensible cut-off scores for dental courses.

PURPOSE: Many countries and institutions have a fixed passing score of 60%. For example, this is a national policy in all Saudi universities where students at all levels must score 60% or higher to pass courses. The purpose of this study is to broaden the scope of standard-setting applications, using data from five major dental courses (Endodontics, Operative, Removable and Fixed Prosthodontics, and Pedodontics) in the 5th dental academic year for two consecutive years to determine the cut-off scores. METHODOLOGY: Four expert raters conducted Angoff's method on final examinations of five dental courses to determine the cut-off scores for the two academic years (2015, and 2016). Cohort size of 2015 was 120 students, whilst that of 2016 was 142 students. The study was conducted at the faculty of Dentistry, at King Abdulaziz University (KAU) from December 2017 to June 2018. RESULTS: All five dental courses yielded different cut-off scores than the fixed 60%. The cut-off scores for 2015 and 2016 examinations were 59.15%, 63.75% for Removable Prosthodontics, 51.86%, 53.75% for Fixed Prosthodontics, 47.42%, 50.5% for Pedodontics, 74.22%, 74.31% for Endodontics and 58%, 55.55% for Operative Dentistry. CONCLUSION: Angoff method yielded different cut-off scores than the fixed 60% passing score currently used by the dental school, yielding a difference in pass-fail rates by as much as a staggering 41%. This finding provides support to use evidence-based standard-setting methods to determine the cut-off score for the courses in Dentistry programme.

Creating Entrustable Professional Activities to Assess Internal Medicine Residents in Training: A Mixed-Methods Approach.

Background: Competency-based medical education has not advanced residency training as much as many observers expected. Some medical educators now advocate reorienting competency-based approaches to focus on a resident's ability to do authentic clinical work. Objective: To develop descriptions of clinical work for which internal medicine residents must gain proficiency to deliver meaningful patient care (for example, "Admit and manage a medical inpatient with a new acute problem"). Design: A modified Delphi process involving clinical experts followed by a conference of educational experts. Setting: The Royal College of Physicians and Surgeons of Canada. Participants: In phase 1 of the project, members of the Specialty Committee for Internal Medicine participated in a modified Delphi process to identify activities in internal medicine that represent the scope of the specialty. In phase 2 of the project, 5 experts who were scholars and leaders in competency-based medical education reviewed the results. Measurements: Phase 1 identified important activities, revised descriptions to improve accuracy and avoid overlap, and assigned activities to stages of training. Phase 2 compared proposed activity descriptions with published guidelines for their development and application in medical education. Results: The project identified 29 activities that qualify as entrustable professional activities. The project also produced a detailed description of each activity and guidelines for using them to assess residents. Limitation: These activities reflect the practice patterns of the developers and may not fully represent internal medicine practice in Canada. Conclusion: Identification of these activities is expected to facilitate modification of training and assessment programs for medical residents so that programs focus less on isolated skills and more on integrated tasks. Primary Funding Source: Southeastern Ontario Academic Medical Organization Endowed Scholarship and Education Fund and Queen's University Department of Medicine Innovation Fund.