Managing assessment during curriculum change: Ottawa Consensus Statement.

Curriculum change is relatively frequent in health professional education. Formal, planned curriculum review must be conducted periodically to incorporate new knowledge and skills, changing teaching and learning methods or changing roles and expectations of graduates. Unplanned curriculum evolution arguably happens continually, usually taking the form of "minor" changes that in combination over time may produce a substantially different programme. However, reviewing assessment practices is less likely to be a major consideration during curriculum change, overlooking the potential for unintended consequences for learning. This includes potentially undermining or negating the impact of even well-designed and important curriculum changes. Changes to any component of the curriculum "ecosystem "- graduate outcomes, content, delivery or assessment of learning - should trigger an automatic review of the whole ecosystem to maintain constructive alignment. Consideration of potential impact on assessment is essential to support curriculum change. Powerful contextual drivers of a curriculum include national examinations and programme accreditation, so each assessment programme sits within its own external context. Internal drivers are also important, such as adoption of new learning technologies and learning preferences of students and faculty. Achieving optimal and sustainable outcomes from a curriculum review requires strong governance and support, stakeholder engagement, curriculum and assessment expertise and internal quality assurance processes. This consensus paper provides guidance on managing assessment during curriculum change, building on evidence and the contributions of previous consensus papers.

The Associations Between United States Medical Licensing Examination Performance and Outcomes of Patient Care.

PURPOSE: The United States Medical Licensing Examination (USMLE) comprises a series of assessments required for the licensure of U.S. MD-trained graduates as well as those who are trained internationally. Demonstration of a relationship between these examinations and outcomes of care is desirable for a process seeking to provide patients with safe and effective health care. METHOD: This was a retrospective cohort study of 196,881 hospitalizations in Pennsylvania over a 3-year period (January 1, 2017 to December 31, 2019) for 5 primary diagnoses: heart failure, acute myocardial infarction, stroke, pneumonia, or chronic obstructive pulmonary disease. The 1,765 attending physicians for these hospitalizations self-identified as family physicians or general internists. A converted score based on USMLE Step 1, Step 2 Clinical Knowledge, and Step 3 scores was available, and the outcome measures were in-hospital mortality and log length of stay (LOS). The research team controlled for characteristics of patients, hospitals, and physicians. RESULTS: For in-hospital mortality, the adjusted odds ratio was 0.94 (95% confidence interval [CI] = 0.90, 0.99; P < .02). Each standard deviation increase in the converted score was associated with a 5.51% reduction in the odds of in-hospital mortality. For log LOS, the adjusted estimate was 0.99 (95% CI = 0.98, 0.99; P < .001). Each standard deviation increase in the converted score was associated with a 1.34% reduction in log LOS. CONCLUSIONS: Better provider USMLE performance was associated with lower in-hospital mortality and shorter log LOS for patients, although the magnitude of the latter is unlikely to be of practical significance. These findings add to the body of evidence that examines the validity of the USMLE licensure program.

The potential of structured learning diaries for combining the development and assessment of self-regulated learning.

Structured Self-Regulated Learning (SSRL) diaries have the potential for combining the development and assessment of a student's SRL processes over time. The aim of this study was to evaluate the extent to which an SSRL diary can develop SRL and provide a reliable longitudinal assessment of SRL development in academically low-achieving undergraduate medical students. We conducted a quasi-experimental study with low-achieving medical students at Tehran University of Medical Sciences. The intervention was a weekly SSRL diary, with 21 items in two parts (before and after studying) that was integrated with weekly explicit SRL training. A repeated measures ANOVA was performed to assess the participants' SRL development across time. We conducted generalizability theory analysis in two designs; the first was to assess the reliability of the total diary score in assessing low-achieving medical students' SRL and the second was to assess the efficacy of the four-week intervention results in improving the low-achieving medical students' SRL. Each participant (n = 20) completed four SSRL diaries. There were significant positive changes during the intervention in the students' measures of total SRL, time estimation of study, time spent on study, concentration, elaboration, organization, self-monitoring, and self-evaluation. The absolute and relative generalizability coefficients for the first design, which indicates the reliability of the students' SRL scores, were 0.71 and 0.77. The absolute and relative generalizability coefficients for the second design, which presents the reliability of the improvement of students' SRL across time were 0.79 and 0.87. The results of this study suggest that SSRL diaries combined with explicit training can provide an effective and reliable method for combining the development and assessment of SRL in low-achieving medical students' SRL over time.

On Purpose: The Case for Alignment in Assessment.

In this issue, Ryan and colleagues underscore the need for criterion-based assessments in the context of competency-based curricula in undergraduate medical education (UME). They also point out that the same scores are often interpreted from a norm-referenced perspective to support the admissions process for residency training. This problem is not unique to UME because in graduate medical education (GME), the same assessments are often used for both decision making and providing feedback. Unfortunately, an assessment with 2 purposes is neither optimal nor efficient for either purpose and may be accompanied by significant side effects. One approach to addressing these challenges is to develop a system of assessment that addresses both purposes but where each component is focused on a single purpose. This leads to alignment and transparency from purpose to test content and from test content to score interpretation and/or feedback. It ensures that the test material is optimized for the task, that individual assessments are constructed to enhance the validity of their scores, and that undesirable side effects are limited.

Excess Mortality Among US Physicians During the COVID-19 Pandemic.

This cross-sectional study examines the death rates among active and nonactive physicians aged 45 to 84 years.

Synthesis and perspectives from the Ottawa 2022 conference on the assessment of competence.

INTRODUCTION: The Ottawa Conference on the Assessment of Competence in Medicine and the Healthcare Professions was first convened in 1985 in Ottawa. Since then, what has become known as the Ottawa conference has been held in various locations around the world every 2 years. It has become an important conference for the community of assessment - including researchers, educators, administrators and leaders - to share contemporary knowledge and develop international standards for assessment in medical and health professions education. METHODS: The Ottawa 2022 conference was held in Lyon, France, in conjunction with the AMEE 2022 conference. A diverse group of international assessment experts were invited to present a symposium at the AMEE conference to summarise key concepts from the Ottawa conference. This paper was developed from that symposium. RESULTS AND DISCUSSION: This paper summarises key themes and issues that emerged from the Ottawa 2022 conference. It highlights the importance of the consensus statements and discusses challenges for assessment such as issues of equity, diversity, and inclusion, shifts in emphasis to systems of assessment, implications of 'big data' and analytics, and challenges to ensure published research and practice are based on contemporary theories and concepts.

Quality assurance in health professions education: Role of accreditation and licensure.

OBJECTIVE: The aim of this paper is to provide an overview of the major quality assurance strategies, accreditation and licensure, in health professions education. It explores the nature of these regulatory processes using Brazil and the United States as examples because these large systems are at different ends of the developmental continuum. For each, it describes the tensions that arise, offers a critical synthesis of the evidence and maps out future directions. RESULTS: Given wide variability among operating medical schools in curricular design, length of study, resources and facilities for clinical training and supervision, the nature of regulatory bodies varies considerably. Nonetheless, they share tensions related purpose and process including quality assurance versus quality improvement, outcomes versus process and continuous versus episodic evaluations and assessments. Clear evidence of effectiveness, especially for accreditation, is scarce and difficult to obtain, particularly as it relates to health outcomes. CONCLUSIONS: Regulatory processes need to be built around clear definitions of the goals for each stage of professional development, the current movement towards competency-based education and the variable durations of medical education. These changes must motivate revisions in the content and process of programmes for accreditation and licensure, complimentary efforts towards quality of care, and stimulate a significant research effort.

Associations between initial American Board of Internal Medicine certification and maintenance of certification status of attending physicians and in-hospital mortality of patients with acute myocardial infarction or congestive heart failure: a retrospective cohort study of hospitalisations in Pennsylvania, USA.

OBJECTIVE: To determine whether internists' initial specialty certification and the maintenance of that certification (MOC) is associated with lower in-hospital mortality for their patients with acute myocardial infarction (AMI) or congestive heart failure (CHF). DESIGN: Retrospective cohort study of hospitalisations in Pennsylvania, USA, from 2012 to 2017. SETTING: All hospitals in Pennsylvania. PARTICIPANTS: All 184 115 hospitalisations for primary diagnoses of AMI or CHF where the attending physician was a self-designated internist. PRIMARY OUTCOME MEASURE: In-hospital mortality. RESULTS: Of the 2575 physicians, 2238 had initial certification and 820 were eligible for MOC. After controlling for patient demographics and clinical characteristics, hospital-level factors and physicians' demographic and medical school characteristics, both initial certification and MOC were associated with lower mortality. The adjusted OR for initial certification was 0.835 (95% CI 0.756 to 0.922; p<0.001). Patients cared for by physicians with initial certification had a 15.87% decrease in mortality compared with those cared for by non-certified physicians (mortality rate difference of 5.09 per 1000 patients; 95% CI 2.12 to 8.05; p<0.001). The adjusted OR for MOC was 0.804 (95% CI 0.697 to 0.926; p=0.003). Patients cared for by physicians who completed MOC had an 18.91% decrease in mortality compared with those cared for by MOC lapsed physicians (mortality rate difference of 6.22 per 1000 patients; 95% CI 2.0 to 10.4; p=0.004). CONCLUSIONS: Initial certification was associated with lower mortality for AMI or CHF. Moreover, for patients whose physicians had initial certification, an additional advantage was associated with its maintenance.

Strengths, Weaknesses, and Suggestions for Improvement in Postgraduate Assessment in Community Medicine in India: A Delphi Study.

OBJECTIVES: It was to understand the strengths and weaknesses of the current postgraduate assessment system in community medicine in India, to identify recommendations for change, and to build a consensus around them. MATERIALS AND METHODS: A conventional Delphi technique was preferred for consensus building among experts. We completed three Delphi rounds over a period of 4 weeks, and 16 experts participated in the study. Content analysis was done for open-ended responses, and consensus analysis was done for Likert-type scale questionnaire. In round three, we obtained their top five preferences for change in assessment. RESULTS: The experts agreed to have an assessment system based on ongoing formative and one end-of-year summative assessment. Apart from this, they agreed on the various occasions for carrying out the formative assessment. Furthermore, they clearly agreed on measures such as blueprinting, improving test formats, and adequate briefing of test-taking students. CONCLUSION AND RECOMMENDATIONS: Most of the consensus items were found to be in alignment with the modern assessment theory. Regulating body and policymakers should revise the current postgraduate assessment system in community medicine to enhance its validity and reliability.

Assessment of Global Health Education: The Role of Multiple-Choice Questions.

Introduction: The standardization of global health education and assessment remains a significant issue among global health educators. This paper explores the role of multiple choice questions (MCQs) in global health education: whether MCQs are appropriate in written assessment of what may be perceived to be a broad curriculum packed with fewer facts than biomedical science curricula; what form the MCQs might take; what we want to test; how to select the most appropriate question format; the challenge of quality item-writing; and, which aspects of the curriculum MCQs may be used to assess. Materials and Methods: The Medical School for International Health (MSIH) global health curriculum was blue-printed by content experts and course teachers. A 30-question, 1-h examination was produced after exhaustive item writing and revision by teachers of the course. Reliability, difficulty index and discrimination were calculated and examination results were analyzed using SPSS software. Results: Twenty-nine students sat the 1-h examination. All students passed (scores above 67% - in accordance with University criteria). Twenty-three (77%) questions were found to be easy, 4 (14%) of moderate difficulty, and 3 (9%) difficult (using examinations department difficulty index calculations). Eight questions (27%) were considered discriminatory and 20 (67%) were non-discriminatory according to examinations department calculations and criteria. The reliability score was 0.27. Discussion: Our experience shows that there may be a role for single-best-option (SBO) MCQ assessment in global health education. MCQs may be written that cover the majority of the curriculum. Aspects of the curriculum may be better addressed by non-SBO format MCQs. MCQ assessment might usefully complement other forms of assessment that assess skills, attitude and behavior. Preparation of effective MCQs is an exhaustive process, but high quality MCQs in global health may serve as an important driver of learning.

Development, testing and generalizability of a standardized evaluation form for the assessment of patient-directed reports in the new final medical licensing examination in Germany.

Background: As doctors often fail to explain diagnoses and therapies to patients in an understandable and appropriate way, the improvement of doctor-patient communication is essential. The current medical training and examinations are focused on verbal rather than on written communication. Following the premise of "assessment drives learning", the final medical licensing examination in Germany has been further developed by the German National Institute for state examinations in Medicine, Pharmacy and Psychotherapy (IMPP). As part of the discharge management the candidates have to prepare a report for the patient that is understandable and provides them with all important information about their stay in hospital. Aim: A standardized evaluation form for formative and summative feedback has been developed and tested with regard to applicability and the assurance of test quality criteria, especially the reliability to assess the written communication skills of the students. Methodology: In an expert consensus procedure, a draft for a standardized evaluation form was developed. This form was revised after an initial trial run on patient-directed reports written by students in their last year of medical studies. Afterwards twenty-one patient-directed reports were evaluated by fourteen different examiners. The reliability was tested by calculating the generalizability-coefficient and by analysing the inter-rater reliability. Results: The first test on the evaluation of the patient-directed reports indicated the practicability of the application and the usefulness of the evaluation form as an instrument for assessing the written communication skills of students. The analyses of the inter-rater reliability showed that the degree of agreement in the evaluations was partly different between two groups of examiners. The calculated G-coefficient indicates a high reliability. The content validity of the evaluation form was given through the comprehensive medical expertise in the development process. Conclusion: Assessing written patient-directed communication is a benefit of the newly developed last part of the medical licensing examination in Germany. Continuous formative assessment and feedback based on the evaluation form is intended to improve the written communication skills of future doctors. Furthermore, a better understanding of their diagnosis and treatment as well as a trusting relationship with their doctor may empower patients in the medical decision process and lead to fewer dismissal errors in the future. For consistent use of the evaluation form a standardized training of examiners should be implemented.

International Medical Graduates: Promoting Equity and Belonging.

International medical school graduates (IMGs) play a vital role in the health care system of the United States. They constitute roughly one-quarter of the physician workforce, comprising a significant proportion of the primary care providers in high-need rural and urban areas, where they provide equal and, in some instances, better care than U.S. graduates. Nonetheless, they face a series of hurdles in entering U.S. residency programs and throughout their training experiences.IMGs must expend significant resources to obtain Education Commission for Foreign Medical Graduates certification, which includes Steps 1, 2 Clinical Knowledge and 2 Clinical Skills of the United States Medical Licensing Examination. They encounter the uncertainty of matching and, if successful, obtaining a visa to enter the United States. Once here, they need to adapt to the complexities of the health care system and familiarize themselves with the cultural nuances, professional behaviors, and communication skills of another country. They encounter biases and microaggressions and lack support groups and mentors. Those who choose an academic career are less likely to obtain leadership positions.This Perspective provides an overview of these challenges and highlights opportunities for change at local and national levels. Specifically, it identifies strategies that would assist IMGs before entry, at entry, during training, at the transition to practice, and in practice. The current COVID-19 pandemic highlights the shortage of physicians in the United States and illustrates the importance of ensuring that IMGs, who are essential health care workers, feel welcome, valued, and recognized for their contributions.

Implementing a Practical Global Health Curriculum: The Benefits and Challenges of Patient-Based Learning in the Community.

Background: A growing number of medical schools across the world have incorporated global health (GH) into their curricula. While several schools focus GH education on lecture-based courses, our premise is that global health education should embody a holistic approach to patient care and medical education in local communities. Medical students may learn global health by focusing on real patients, their families and communities as part of a practical curriculum. Aims and Objectives: A unique GH curriculum was devised to compare student learning outcomes on a practical vs. lecture-based course. The premise was that learning from patients would result in a greater breadth of coverage of the global health syllabus as compared to that from a lecture-based course. Methods: A teaching and learning program was developed over 3 years to provide medical students interaction with real patients in the community on a first-preclinical-year Introduction to Global Health and Medical Anthropology course. Learning outcomes on the practical vs. lecture-based course were compared using thematic analysis of the written assignments of both courses: global health case reports and literature reviews, respectively. All members of three cohorts of students undertaking the course in successive academic years were compared (Group A: literature review; Groups B and C: case reports; n = 87). Results: Case reports provided evidence of a greater breadth of learning outcomes when compared to the literature review (p < 0.001). The writing of the case report was enhanced by completion of a field journal and family health needs assessment tool (p < 0.001). Students demonstrated a closeness to their patients that added depth, understanding and motivation to assist patients in health activities and advocate for their needs. Discussion: Placements with patients in the community provided students with a rich learning environment and facilitated the formation of relationships with patients to better understand the social determinants of health and advocate for improvements in their living and working conditions and access to healthcare. Conclusions: Global health may be better learned experientially by following patients rather than from frontal lectures. Patient-based learning inspires a commitment to the individual and facilitates medical schools in meeting their obligations to the communities they serve.

Academic excellence in Latin America: Social accountability of medical schools.

Context: Social accountability of medical schools has emerged as a standard of excellence in medical education during the last decade. However, the lack of valid and reliable instruments to estimate social accountability has limited the possibility of measuring the impact that medical schools have in society. Our aim was to develop an instrument and validate its use for assessing social accountability in Latin American countries.Methods: We used a three-phase mixed methods research design to develop, validate and estimate social accountability in a diverse convenient sample of 49 medical schools from 16 Latin American countries. We used a qualitative framework approach and a Delphi consensus method to design an instrument with high content validity. Finally, we assessed the psychometric properties of the instrument.Results: The Social Accountability Instrument for Latin America (SAIL) contained 21 items in four domains: mission and quality improvement, public policy, community engagement, and professional integrity. Its reliability index, estimated using Cronbach's alpha, was very high (0.96). Most of the medical schools that had ranked over the 80th percentile on traditional national academic estimates did not reach the 80th percentile using SAIL.Conclusions: There are validity arguments (content and reliability) to support the measurement of social accountability using the SAIL instrument. Its application showed that it provides a complementary dimension to that traditionally obtained when estimating quality in medical schools.

Factors to be considered in designing a faculty development program for medical education: local experience from the Western region of Saudi Arabia.

BACKGROUND: Among the different aims of medical education, the provision of society with skilled, professional, and knowledgeable healthcare workers who maintain and develop their expertise over a lifetime career is important. The achievement of this goal is linked with the professional development of both faculty members and healthcare workers. This study aims to measure the perception of faculty members regarding their views about the goals of faculty development programs, practices and activities, and factors that determine their achievement. METHODS: A cross-sectional survey was conducted in multiple universities in the Western region of Saudi Arabia. The participants were given a pre-designed self-administered questionnaire generated from literature. The survey questionnaire consisted of three sections that were designed to assess the faculty members' perception on the faculty development program. RESULTS: A total of 210 faculty members participated in the study. The most important perceived goal was to motivate teachers to become better teachers. The most important perceived practice was establishing a positive climate for teaching and learning. The most important perceived factor was skilled and dedicated staff support. CONCLUSION: The results of this study demonstrate that faculty members have positive perceptions regarding all aspects of faculty development programs. This study will raise awareness regarding the importance of faculty development programs in sustaining educational vitality. We recommend the implementation and maintenance of comprehensive faculty development programs in Saudi universities.

Medical assessment in the age of digitalisation.

BACKGROUND: Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed. METHODS: In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified. RESULTS: The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions. CONCLUSIONS: It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.

Entrustment decisions: Implications for curriculum development and assessment.

With increased interest in the use of entrustable professional activities (EPAs) in undergraduate and postgraduate medical education, comes questions about their implications for curriculum development and assessment. This paper addresses some of those questions, discussed at a symposium of the 2017 conference of AMEE, by presenting the components of an EPA, describing their importance and application, identifying their implications for assessment, and pinpointing some of challenges they pose in undergraduate and postgraduate settings. It defines entrustment, describes the three levels of trust, and presents trainee and supervisor factors that influence it as well as perceived benefits, and risks. Two aspects of EPAs have implications for assessment: units of professional practice and decisions based on entrustment, which impact an assessment's blueprint, test methods, scores, and standards. In an undergraduate setting EPAs have great appeal, but work is needed to identify and develop a robust assessment system for core EPAs. At the postgraduate level, there is tension between the granularity of the competencies and the integrated nature of the EPAs. Even though work remains, EPAs offer an important step in the evolution of competency-based education.

The utility of mini-Clinical Evaluation Exercise in undergraduate and postgraduate medical education: A BEME review: BEME Guide No. 59.

Background: This BEME review aims at exploring, analyzing, and synthesizing the evidence considering the utility of the mini-CEX for assessing undergraduate and postgraduate medical trainees, specifically as it relates to reliability, validity, educational impact, acceptability, and cost.Methods: This registered BEME review applied a systematic search strategy in seven databases to identify studies on validity, reliability, educational impact, acceptability, or cost of the mini-CEX. Data extraction and quality assessment were carried out by two authors. Discrepancies were resolved by a third reviewer. Descriptive synthesis was mainly used to address the review questions. A meta-analysis was performed for Cronbach's alpha.Results: Fifty-eight papers were included. Only two studies evaluated all five utility criteria. Forty-seven (81%) of the included studies met seven or more of the quality criteria. Cronbach's alpha ranged from 0.58 to 0.97 (weighted mean = 0.90). Reported G coefficients, Standard error of measurement, and confidence interval were diverse and varied based on the number of encounters and the nested or crossed design of the study. The calculated number of encounters needed for a desirable G coefficient also varied greatly. Content coverage was reported satisfactory in several studies. Mini-CEX discriminated between various levels of competency. Factor analyses revealed a single dimension. The six competencies showed high levels of correlation with statistical significance with the overall competence. Moderate to high correlations between mini-CEX scores and other clinical exams were reported. The mini-CEX improved students' performance in other examinations. By providing a framework for structured observation and feedback, the mini-CEX exerts a favorable educational impact. Included studies revealed that feedback was provided in most encounters but its quality was questionable. The completion rates were generally above 50%. Feasibility and high satisfaction were reported.Conclusion: The mini-CEX has reasonable validity, reliability, and educational impact. Acceptability and feasibility should be interpreted given the required number of encounters.