Qualitative content analysis of the highly cited articles in medical education: Trends and characteristics associated with citation of published studies in medical education research.

PURPOSE: To explore and describe the highly cited articles' themes of research in medical education and to provide an insight into and reflection on which the elites of medical education society invested their energies from 2009 to 2018. METHODS: An in-depth content analysis as a research technique for the objective, systematic, and quantitative description of the manifest content of communication was used to quantitatively assess subject interests, methods, and other characteristics associated with citation of published studies in medical education research. Meaning units were compacted and coded with labels and categories in two phases. RESULTS: Among a variety of topics, methods, and strategies, 764 codes, 24 descriptive themes, and seven categories were extracted from the content analysis as the most prominent. Categories of medical education research were: modern technologies updating in medical education; learner performance improvement; sociological aspects of medical education; clinical reasoning; research methodology concerns of medical education; instructional design educational models; and professional aspects of medical education. CONCLUSIONS: Commitment to continuous revision of educational emphasis and concerns on technological, sociological, and methodological concerns were the most repeated components of the highly cited articles that were ascertained through increased structure course designs and instructional strategies of the flipped classrooms to realize clinical reasoning and performance improvement.[Box: see text].

Entrustment decisions: Implications for curriculum development and assessment.

With increased interest in the use of entrustable professional activities (EPAs) in undergraduate and postgraduate medical education, comes questions about their implications for curriculum development and assessment. This paper addresses some of those questions, discussed at a symposium of the 2017 conference of AMEE, by presenting the components of an EPA, describing their importance and application, identifying their implications for assessment, and pinpointing some of challenges they pose in undergraduate and postgraduate settings. It defines entrustment, describes the three levels of trust, and presents trainee and supervisor factors that influence it as well as perceived benefits, and risks. Two aspects of EPAs have implications for assessment: units of professional practice and decisions based on entrustment, which impact an assessment's blueprint, test methods, scores, and standards. In an undergraduate setting EPAs have great appeal, but work is needed to identify and develop a robust assessment system for core EPAs. At the postgraduate level, there is tension between the granularity of the competencies and the integrated nature of the EPAs. Even though work remains, EPAs offer an important step in the evolution of competency-based education.

Twelve tips on how to provide self-regulated learning (SRL) enhanced feedback on clinical performance.

The provision of self-regulated learning (SRL) enhanced feedback on performing clinical skills and making a clinical diagnosis recognizes the importance of feedback on the use of key SRL processes. In contrast to the broader concept of self-directed learning, SRL has a specific focus on the individual learner's approach to achieve a task, including their planning, self-monitoring, and future adaptations. The key SRL processes can be identified using structured microanalysis during the clinical task and feedback to the learner using the tips outlined in this article. It is essential that SRL enhanced feedback is integrated with best practice on providing feedback to ensure that its potential is achieved.

2018 Consensus framework for good assessment.

Introduction: In 2010, the Ottawa Conference produced a set of consensus criteria for good assessment. These were well received and since then the working group monitored their use. As part of the 2010 report, it was recommended that consideration be given in the future to preparing similar criteria for systems of assessment. Recent developments in the field suggest that it would be timely to undertake that task and so the working group was reconvened, with changes in membership to reflect broad global representation.Methods: Consideration was given to whether the initially proposed criteria continued to be appropriate for single assessments and the group believed that they were. Consequently, we reiterate the criteria that apply to individual assessments and duplicate relevant portions of the 2010 report.Results and discussion: This paper also presents a new set of criteria that apply to systems of assessment and, recognizing the challenges of implementation, offers several issues for further consideration. Among these issues are the increasing diversity of candidates and programs, the importance of legal defensibility in high stakes assessments, globalization and the interest in portable recognition of medical training, and the interest among employers and patients in how medical education is delivered and how progression decisions are made.

The development of a pragmatic, clinically driven ultrasound curriculum in a UK medical school.

Whether ultrasound (US) should be incorporated into a medical undergraduate curriculum remains a matter of debate within the medical education arena. There are clear potential benefits to its early introduction particularly with respect to the study of living anatomy and physiology in addition to the learning of clinical skills and procedures required for the graduate clinical practice. However, this needs to be balanced against what is perceived as an added value in addition to financial and time constraints which may potentially lead to the sacrifice of other aspects of the curriculum. Several medical schools have already reported their experiences of teaching US either as a standalone course or as a fully integrated vertical curriculum. This article describes and discusses the initial experience of a UK medical school that has taken the steps to develop its own pragmatic vertical US curriculum based on clinical endpoints with the intent of using US to enhance the learning experience of students and equipping them with the skills required for the safe practice as a junior doctor.

Qualitative and quantitative feedback in the context of competency-based education.

Research indicates the importance and usefulness of feedback, yet with the shift of medical curricula toward competencies, feedback is not well understood in this context. This paper attempts to identify how feedback fits within a competency-based curriculum. After careful consideration of the literature, the following conclusions are drawn: (1) Because feedback is predicated on assessment, the assessment should be designed to optimize and prevent inaccuracies in feedback; (2) Giving qualitative feedback in the form of a conversation would lend credibility to the feedback, address emotional obstacles and create a context in which feedback is comfortable; (3) Quantitative feedback in the form of individualized data could fulfill the demand for more feedback, help students devise strategies on how to improve, allow students to compare themselves to their peers, recognizing that big data have limitations; and (4) Faculty development needs to incorporate and promote cultural and systems changes with regard to feedback. A better understanding of the role of feedback in competency-based education could result in more efficient learning for students.

Trends in national licensing examinations in medicine.

CONTEXT: As a contribution to this special issue commemorating the journal's 50th volume, this paper seeks to explore directions for national licensing examinations (NLEs) in medicine. Increases in the numbers of new medical schools and the mobility of doctors across national borders mean that NLEs are becoming even more important to ensuring physician competence. OBJECTIVES: The purpose of this paper is to explore the use of NLEs in the future in the context of global changes in medical education and health care delivery. METHODS: Because the literature related to NLEs is so large, we have not attempted a comprehensive review, but have focused instead on a small number of topics on which we think we have something useful to say. The paper is organised around five predicted trends for NLEs. DISCUSSION: The first section discusses reasons why we think the use of NLEs will increase in the coming years. The second section discusses the ongoing problem of content specificity and its implications for the design of NLEs. The third section examines the evolution of large-scale, standardised cognitive assessments in NLEs and suggests some future directions. Reflecting the fact that NLEs are, increasingly, attempting to assess more than just knowledge, the fourth section addresses the future of large-scale clinical skills assessments in NLEs, predicting both increases in their use and some shifts in the nature of the stations used. The fifth section discusses workplace-based assessments, predicting increases in their use for formative assessment and identifying some limitations in their direct application in NLEs. The concluding section discusses the cost of NLEs and indulges in some further speculations about their evolution.

Assessing competencies using milestones along the way.

This paper presents perspectives and controversies surrounding the use of milestones to assess competency in outcomes-based medical education. Global perspectives (Canada, Europe, and the United States) and developments supporting their rationales are discussed. In Canada, there is a significant movement away from conceptualizing competency based on time, and a move toward demonstration of specific competencies. The success of this movement may require complex (rather than reductionist) milestones that reflect students' progression through complexity and context and a method to narrate their journey. European countries (United Kingdom, France, and Germany) have stressed the complexity associated with time and milestones for medical students to truly achieve competence. To meet the changing demands of medicine, they view time as actually providing students with knowledge and exposure to achieve various milestones. In the United States, milestones are based on sampling throughout professional development to initiate lifelong learning. However, the use of milestones may not imply overall competence (reductionism). Milestones must be developed alongside outcomes-based curriculum with use of faculty and competency committees. The perspectives outlined in this paper underscore emerging challenges for implementing outcomes-based medical education and call for new conceptualizations of competence.

Preparing leaders in health professions education.

In the past 15 years, the number of Master's degree programs in Health Professions Education (MHPE) has grown from 7 to 121 programs worldwide. New MHPE programs continue to be developed each year, due to increased demand for individuals with specialized knowledge concerning how to best educate future health professionals. During the 2012 Association of Medical Education in Europe (AMEE) meeting in Lyon, France, a symposium was organized to explore the reasons for the proliferation of MHPE programs worldwide. In particular, the issues explored included the need for such programs, their outcomes in developing education leaders and scholars in HPE, and facilitators, barriers and models for initiating such programs. This paper synthesizes the discussion during this symposium. Some of the reasons for enrolling in a Master's degree program in HPE include the formal credential, knowledge of a number of theories and frameworks, new approaches to problems and ways of thinking, the mentored project, and networking and working with faculty and students. The uniqueness of being a trainee in an MHPE program is the immersion in the medical education environment and the assimilation of a new approach to scholarship and a new approach to leadership.

Advancing the objective structured clinical examination: sequential testing in theory and practice.

CONTEXT: Models of short-term remediation for failing students are typically associated with improvements in candidate performance at retest. However, the process is costly to deliver, particularly for performance retests with objective structured clinical examinations (OSCEs), and there is increasing evidence that these traditional models are associated with the longitudinal underperformance of candidates. METHODS: Rather than a traditional OSCE model, sequential testing involves a shorter 'screening' format, with an additional 'sequential' test for candidates who fail to meet the screening standard. For those tested twice, overall pass/fail decisions are then based on results on the full sequence of tests. In this study, the impacts of sequential assessment on student performance, cost of assessment delivery and overall reliability were modelled using data sourced from a final graduating OSCE in an undergraduate medical degree programme. RESULTS: Initial modelling using pre-existing OSCE data predicted significant improvements in reliability in the critical area, reflected in pilot results: 13.5% of students (n = 228) were required to sit the sequential OSCE. One student (0.4%) was identified as representing a false positive result (i.e. under the previous system this student would have passed the OSCE but failed on extended testing). Nine students (3.9%) who would have required OSCE retests under the prior system passed the full sequence and were therefore able to graduate at the normal time without loss of earnings. Overall reliability was estimated as 0.79 for the full test sequence. Significant cost savings were realised. CONCLUSIONS: Sequential testing in OSCEs increases reliability for borderline students because the increased number of observations implies that 'observed' student marks are closer to 'true' marks. However, the station-level quality of the assessment needs to be sufficiently high for the full benefits in terms of reliability to be achieved. The introduction of such a system has financial benefits, good validity inferences and has proved acceptable to students and other stakeholders.

Estimating and comparing the reliability of a suite of workplace-based assessments: an obstetrics and gynaecology setting.

This paper reports on a study that compares estimates of the reliability of a suite of workplace based assessment forms as employed to formatively assess the progress of trainee obstetricians and gynaecologists. The use of such forms of assessment is growing nationally and internationally in many specialties, but there is little research evidence on comparisons by procedure/competency and form-type across an entire specialty. Generalisability theory combined with a multilevel modelling approach is used to estimate variance components, G-coefficients and standard errors of measurement across 13 procedures and three form-types (mini-CEX, OSATS and CbD). The main finding is that there are wide variations in the estimates of reliability across the forms, and that therefore the guidance on assessment within the specialty does not always allow for enough forms per trainee to ensure that the levels of reliability of the process is adequate. There is, however, little evidence that reliability varies systematically by form-type. Methodologically, the problems of accurately estimating reliability in these contexts through the calculation of variance components and, crucially, their associated standard errors are considered. The importance of the use of appropriate methods in such calculations is emphasised, and the unavoidable limitations of research in naturalistic settings are discussed.

Is short-term remediation after OSCE failure sustained? A retrospective analysis of the longitudinal attainment of underperforming students in OSCE assessments.

BACKGROUND: Significant improvements in the delivery of criterion-based assessment techniques have improved confidence in standard setting and assessment quality. However, for underperforming students, a lack of evidence about longitudinal performance of this group poses dilemmas to educators when making decisions about the timing and nature of remediation. AIM: To investigate the longitudinal performance of the UK undergraduate medical degree students, with a particular focus on comparing the poorly performing students (i.e. those with borderline or failing grades) with the main cohort of students. METHOD: Over a 5-year period, 3200-student objective structured clinical examination (OSCE) assessments from a single medical school were investigated. A poorly performing subgroup of 125 students was identified and their longitudinal performance in the final 3 years of the undergraduate medical degree analysed. RESULT: The relative performance of this student group declines across serial OSCEs, despite current methods of 'remediation and retest'. CONCLUSIONS: This analysis demonstrates that typically students in the poorly performing subgroup achieve only short-term success with traditional remediation and retest models, and critically show an absence of longitudinal improvement. There is a clear need for institutions to develop profiling models that can help identify this student group and develop effective, research led models of remediation.

Preparedness is not enough: understanding transitions as critically intensive learning periods.

OBJECTIVES: Doctors make many transitions whilst they are training and throughout their ensuing careers. Despite studies showing that transitions in other high-risk professions such as aviation have been linked to increased risk in the form of adverse outcomes, the effects of changes on doctors' performance and consequent implications for patient safety have been under-researched. The purpose of this project was to investigate the effects of transitions upon medical performance. METHODS: The project sought to focus on the inter-relationships between doctors and the complex work settings into which they transition. To this end, a 'collective' case study of doctors was designed. Key transitions for foundation year and specialist trainee doctors were studied. Four levels of the case were examined, pertaining to: the regulatory and policy context; employer requirements; the clinical teams in which doctors worked, and the doctors themselves. Data collection methods included interviews, observations and desk-based research. RESULTS: A number of problems with doctors' transitions that can all adversely affect performance were identified. (i) Transitions are regulated but not systematically monitored. (ii) Actual practice (as observed and reported) was determined much more by situational and contextual factors than by the formal (regulatory and management) frameworks. (iii) Trainees' and health professionals' accounts of their actual experiences of work showed how performance is dependent on the local learning environment. (iv) The increased regulation of clinical activity through protocols and care pathways helps to improve trainees' performance, whereas the less regulated aspects of work, such as rotas, induction and the making of multiple transitions within rotations, can impede performance during a period of transition. CONCLUSIONS: Transitions may be reframed as critically intensive learning periods (CILPs) in which doctors engage with the particularities of the setting and establish working relationships with other doctors and other professionals. Institutions and wards have their own learning cultures which may or may not recognise that transitions are CILPS. The extent to which these cultures take account of transitions as CILPs will contribute to the performance of new doctors. Thus, these findings have implications for practice and for policy, regulation and research.