Assessment of management reasoning: Design considerations drawn from analysis of simulated outpatient encounters.

PURPOSE: Management reasoning is a distinct subset of clinical reasoning. We sought to explore features to be considered when designing assessments of management reasoning. METHODS: This is a hybrid empirical research study, narrative review, and expert perspective. In 2021, we reviewed and discussed 10 videos of simulated (staged) physician-patient encounters, actively seeking actions that offered insights into assessment of management reasoning. We analyzed our own observations in conjunction with literature on clinical reasoning assessment, using a constant comparative qualitative approach. RESULTS: Distinguishing features of management reasoning that will influence its assessment include management scripts, shared decision-making, process knowledge, illness-specific knowledge, and tailoring of the encounter and management plan. Performance domains that merit special consideration include communication, integration of patient preferences, adherence to the management script, and prognostication. Additional facets of encounter variation include the clinical problem, clinical and nonclinical patient characteristics (including preferences, values, and resources), team/system characteristics, and encounter features. We cataloged several relevant assessment approaches including written/computer-based, simulation-based, and workplace-based modalities, and a variety of novel response formats. CONCLUSIONS: Assessment of management reasoning could be improved with attention to the performance domains, facets of variation, and variety of approaches herein identified.

A decade of faculty development for health professions educators: lessons learned from the Macy Faculty Scholars Program.

Faculty development (FD) programs are critical for providing the knowledge and skills necessary to drive positive change in health professions education, but they take many forms to attain the program goals. The Macy Faculty Scholars Program (MFSP), created by the Josiah Macy Jr. Foundation (JMJF) in 2010, intends to develop participants as leaders, scholars, teachers, and mentors. After a decade of implementation, an external review committee conducted a program evaluation to determine how well the program met its intended goals and defined options for ongoing improvement.The committee selected Stufflebeam's CIPP (context, input, process, products) framework to guide the program evaluation. Context and input components were derived from the MFSP description and demographic data, respectively. Process and product components were obtained through a mixed-methods approach, utilizing both quantitative and qualitative data obtained from participant survey responses, and curriculum vitae (CV).The evaluation found participants responded favorably to the program and demonstrated an overall increase in academic productivity, most pronounced during the two years of the program. Mentorship, community of practice, and protected time were cited as major strengths. Areas for improvement included: enhancing the diversity of program participants, program leaders and mentors across multiple sociodemographic domains; leveraging technology to strengthen the MFSP community of practice; and improving flexibility of the program.The program evaluation results provide evidence supporting ongoing investment in faculty educators and summarizes key strengths and areas for improvement to inform future FD efforts for both the MFSP and other FD programs.

An outcomes research perspective on medical education: Has anything changed in the last 18 years?

PURPOSE: Medical education research focused on patient-centered outcomes holds the promise of improved decision-making by medical educators. In 2001, Prystowsky and Bordage demonstrated that patient-centered outcomes were evaluated in fewer than one percent of studies published in a survey of major medical education journals. Though many have called for increased inclusion of patient-centered outcomes in medical education literature, it remains uncertain to what degree this need has been addressed systematically. METHODS: Using the same data sources as in the original report (Academic Medicine, Medical Education, and Teaching and Learning in Medicine), we sought to replicate Prystowsky and Bordage's study. We extracted data from original empirical research reports from these three journal sources for the years 2014-2016. We selected 652 articles that met the inclusion criteria for further analysis. RESULTS: Study participants were largely trainees (64% of studies) or faculty (25% of studies). Only 2% of studies included patients as active or passive participants. Study outcomes reported were satisfaction (40% of studies), performance (39%), professionalism (20%), and cost (1%). CONCLUSIONS: These results do not differ significantly from the original 2001 study. The medical education literature as represented in these three prominent journals has made little progress in placing a greater focus on patient-centered outcomes.

Development and validation of conflict management attitude questionnaire for medical students.

BACKGROUND: Medical students should effectively manage conflicts in teamwork and communication with other team members. This study aimed to develop and validate a tool to evaluate attitude of medical students and physicians toward conflict management. METHOD: A multi-step process was employed to develop and validate a Conflict Management Attitude Questionnaire (CMAQ) based on the steps recommended in AMEE Guide No. 87. First, the initial items were obtained from the literature review and focus group. After cognitive interviews with the medical students and revision of the questionnaire, content validity was performed by experts. The construct validity and reliability of the questionnaire were assessed using exploratory Factor Analysis (EFA) and Cronbach's alpha coefficient, respectively. RESULTS: This multi-step process resulted in a 12-item, five-point Likert-type questionnaire with satisfactory construct validity. Exploratory factor analysis revealed three factors, comprising the four items from the "perceived interactions in conflict management" subscale loading on the first factor, and five items from the "perceived value of learning conflict management" subscale loading on the second factor, along with three items from the "perceived application of conflict management" subscale loading on the third factor. All subscales described 56.48% of the variance. Validation results showed that Content Validity Index (CVI) and Content Validity Ratio (CVR) were greater than 0.75. Cronbach's alpha coefficient was 0.791. CONCLUSION: This study showed that CMAQ has valid evidence for assessing the attitude of medical students toward conflict management with favorable psychometric properties and strong evidence of construct validity. However, due to the lack of evidence on any specific questionnaire to evaluate the attitude towards conflict management, future studies should conduct a confirmatory investigation regarding other aspects of medical students' attitudes toward conflict management.

Assessing changes in the quality of quantitative health educations research: a perspective from communities of practice.

BACKGROUND: As a community of practice (CoP), medical education depends on its research literature to communicate new knowledge, examine alternative perspectives, and share methodological innovations. As a key route of communication, the medical education CoP must be concerned about the rigor and validity of its research literature, but prior studies have suggested the need to improve medical education research quality. Of concern in the present study is the question of how responsive the medical education research literature is to changes in the CoP. We examine the nature and extent of changes in the quality of medical education research over a decade, using a widely cited study of research quality in the medical education research literature as a benchmark to compare more recent quality indicators. METHODS: A bibliometric analysis was conducted to examine the methodologic quality of quantitative medical education research studies published in 13 selected journals from September 2013 to December 2014. Quality scores were calculated for 482 medical education studies using a 10-item Medical Education Research Study Quality Instrument (MERSQI) that has demonstrated strong validity evidence. These data were compared with data from the original study for the same journals in the period September 2002 to December 2003. Eleven investigators representing 6 academic medical centers reviewed and scored the research studies that met inclusion and exclusion criteria. Primary outcome measures include MERSQI quality indicators for 6 domains: study design, sampling, type of data, validity, data analysis, and outcomes. RESULTS: There were statistically significant improvements in four sub-domain measures: study design, type of data, validity and outcomes. There were no changes in sampling quality or the appropriateness of data analysis methods. There was a small but significant increase in the use of patient outcomes in these studies. CONCLUSIONS: Overall, we judge this as equivocal evidence for the responsiveness of the research literature to changes in the medical education CoP. This study identified areas of strength as well as opportunities for continued development of medical education research.

Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic.

Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy.

BACKGROUND: Using revised Bloom's taxonomy, some medical educators assume they can write multiple choice questions (MCQs) that specifically assess higher (analyze, apply) versus lower-order (recall) learning. The purpose of this study was to determine whether three key stakeholder groups (students, faculty, and education assessment experts) assign MCQs the same higher- or lower-order level. METHODS: In Phase 1, stakeholders' groups assigned 90 MCQs to Bloom's levels. In Phase 2, faculty wrote 25 MCQs specifically intended as higher- or lower-order. Then, 10 students assigned these questions to Bloom's levels. RESULTS: In Phase 1, there was low interrater reliability within the student group (Krippendorf's alpha = 0.37), the faculty group (alpha = 0.37), and among three groups (alpha = 0.34) when assigning questions as higher- or lower-order. The assessment team alone had high interrater reliability (alpha = 0.90). In Phase 2, 63% of students agreed with the faculty as to whether the MCQs were higher- or lower-order. There was low agreement between paired faculty and student ratings (Cohen's Kappa range .098-.448, mean .256). DISCUSSION: For many questions, faculty and students did not agree whether the questions were lower- or higher-order. While faculty may try to target specific levels of knowledge or clinical reasoning, students may approach the questions differently than intended.

Mapping clinical reasoning literature across the health professions: a scoping review.

BACKGROUND: Clinical reasoning is at the core of health professionals' practice. A mapping of what constitutes clinical reasoning could support the teaching, development, and assessment of clinical reasoning across the health professions. METHODS: We conducted a scoping study to map the literature on clinical reasoning across health professions literature in the context of a larger Best Evidence Medical Education (BEME) review on clinical reasoning assessment. Seven databases were searched using subheadings and terms relating to clinical reasoning, assessment, and Health Professions. Data analysis focused on a comprehensive analysis of bibliometric characteristics and the use of varied terminology to refer to clinical reasoning. RESULTS: Literature identified: 625 papers spanning 47 years (1968-2014), in 155 journals, from 544 first authors, across eighteen Health Professions. Thirty-seven percent of papers used the term clinical reasoning; and 110 other terms referring to the concept of clinical reasoning were identified. Consensus on the categorization of terms was reached for 65 terms across six different categories: reasoning skills, reasoning performance, reasoning process, outcome of reasoning, context of reasoning, and purpose/goal of reasoning. Categories of terminology used differed across Health Professions and publication types. DISCUSSION: Many diverse terms were present and were used differently across literature contexts. These terms likely reflect different operationalisations, or conceptualizations, of clinical reasoning as well as the complex, multi-dimensional nature of this concept. We advise authors to make the intended meaning of 'clinical reasoning' and associated terms in their work explicit in order to facilitate teaching, assessment, and research communication.

Exploring the characteristics and context that allow Master Adaptive Learners to thrive.

Because change is ubiquitous in healthcare, clinicians must constantly make adaptations to their practice to provide the highest quality care to patients. In a previous article, Cutrer et al. described a metacognitive approach to learning based on self-regulation, which facilitates the development of the Master Adaptive Learner (MAL). The MAL process helps individuals to cultivate and demonstrate adaptive expertise, allowing them to investigate new concepts (learn) and create new solutions (innovate). An individual's ability to learn in this manner is driven by several internal characteristics and is also impacted by numerous aspects of their context. In this article, the authors examine the important internal and contextual factors that can impede or foster Master Adaptive Learning.

Competency-based education: programme design and challenges to implementation.

CONTEXT: Competency-based education (CBE) has been widely cited as an educational framework for medical students and residents, and provides a framework for designing educational programmes that reflect four critical features: a focus on outcomes, an emphasis on abilities, a reduction of emphasis on time-based training, and promotion of learner centredness. Each of these features has implications and potential challenges for implementing CBE. METHODS: As an experiment in CBE programme design and implementation, the University of Michigan Master of Health Professions Education (UM-MHPE) degree programme was examined for lessons to be learned when putting CBE into practice. The UM-MHPE identifies 12 educational competencies and 20 educational entrustable professional activities (EPAs) that serve as the vehicle for both learning and assessment. The programme also defines distinct roles of faculty members as assessors, mentors and subject-matter experts focused on highly individualised learning plans adapted to each learner. CONCLUSIONS: Early experience with implementing the UM-MHPE indicates that EPAs and competencies can provide a viable alternative to traditional courses and a vehicle for rigorous assessment. A high level of individualisation is feasible but carries with it significant costs and makes intentional community building essential. Most significantly, abandoning a time-based framework is a difficult innovation to implement in a university structure that is predicated on time-based education.

How to write an educational research grant: AMEE Guide No. 101.

Writing an educational research grant in health profession education is challenging, not only for those doing it for the first time but also for more experienced scholars. The intensity of the competition, the peculiarities of the grant format, the risk of rejection, and the time required are among the many obstacles that can prevent educational researchers with interesting and important ideas from writing a grant, that could provide the funding needed to turn their scholarly ideas into reality. The aim of this AMEE Guide is to clarify the grant-writing process by (a) explaining the mechanics and structure of a typical educational research grant proposal, and (b) sharing tips and strategies for making the process more manageable.

Assessment challenges in competency-based education: A case study in health professions education.

There is a growing demand for health sciences faculty with formal training in education. Addressing this need, the University of Michigan Medical School created a Master in Health Professions Education (UM-MHPE). The UM-MHPE is a competency-based education (CBE) program targeting professionals. The program is individualized and adaptive to the learner's situation using personal mentoring. Critical to CBE is an assessment process that accurately and reliably determines a learner's competence in educational domains. The program's assessment method has two principal components: an independent assessment committee and a learner repository. Learners submit evidence of competence that is evaluated by three independent assessors. The assessments are presented to an Assessment Committee who determines whether the submission provides evidence of competence. The learner receives feedback on the submission and, if needed, the actions needed to reach competency. During the program's first year, six learners presented 10 submissions for review. Assessing learners in a competency-based program has created challenges; setting standards that are not readily quantifiable is difficult. However, we argue it is a more genuine form of assessment and that this process could be adapted for use within most competency-based formats. While our approach is demanding, we document practical learning outcomes that assess competence.

Twelve tips for developing, implementing, and sustaining medical education fellowship programs: Building on new trends and solid foundations.

Medical education fellowship programs (MEFPs) are a form of faculty development contributing to an organization's educational mission and participants' career development. Building an MEFP requires a systematic design, implementation, and evaluation approach which aligns institutional and individual faculty goals. Implementing an MEFP requires a team of committed individuals who provide expertise, guidance, and mentoring. Qualified MEFP directors should utilize instructional methods that promote individual and institutional short and long term growth. Directors must balance the use of traditional design, implementation, and evaluation methodologies with advancing trends that may support or threaten the acceptability and sustainability of the program. Drawing on the expertise of 28 MEFP directors, we provide twelve tips as a guide to those implementing, sustaining, and/or growing a successful MEFP whose value is demonstrated by its impacts on participants, learners, patients, teaching faculty, institutions, the greater medical education community, and the population's health.

The relationship between promotions committees' identification of problem medical students and subsequent state medical board actions.

Studies have found unprofessional behavior in medical school was associated with disciplinary action by state medical boards. For medical schools, promotions committees are responsible for identifying which students do not demonstrate academic performance and professional behavior acceptable for promotion and graduation. The objective of this study was to determine if student identification by promotions committees during medical school was associated with disciplinary actions by state medical boards later in practice. We reviewed 20 years of promotions committees' records from a single institution and noted students identified by promotions committees for performance or behavioral issues. These were compared with disciplinary action reports from the Federation of State Medical Boards (FSMB) for graduates. Over the two decades, 2,131 students matriculated and 2,078 of these graduated. The promotions committees identified 140 students for poor academic performance or behavioral problems (140/2,078, 6.7 %). Of these, 108 students graduated. FSMB records showed 29 of the 2,078 graduates had sanctions by state boards (29/2,078, 1.4 %). Only four students that had actions by state medical boards were among the 108 graduated students identified by medical school promotions committees (4/108, 3.7 %). Of the students not identified by promotions committees, 25 eventually had disciplinary actions (25/1,970, 1.3 %). The odds of having state medical board action if identified by promotions committees was 3.0 (CI 1.02-8.8, p < 0.05). In conclusion, identification of students by medical school promotions committees was later associated with state medical board actions. However, most graduates with state medical board actions were not identified by medical school promotions committees.

Humility and respect: core values in medical education.

CONTEXT: Many of the values and behaviours described in the original Hippocratic Oath are relevant to medical education. In particular, the values of intellectual humility and respect for one's colleagues are essential in all scientific disciplines. There are three contexts within medical education from which to consider humility and respect: uncertainty; theory, and colleagues. UNCERTAINTY: As medical education grows in scope and participation, we will be required to acknowledge that we 'know not' with increasing frequency. The uncertainty of what we do and do not know is compounded by uncertainty about whether ignorance is individual or corporate. As difficult as it is to admit that we 'know not', it is dangerous NOT to recognise the limits of our knowledge and experience. THEORY: Theories are critical tools in understanding complex phenomena. They identify constructs and relationships that are important and those that are irrelevant. We tend to forget that theories are models or simplified representations of reality and not in themselves 'truths'. Viewing problems from other theoretical perspectives can widen our horizons by allowing us to identify possibly important concepts and relationships that we have not considered. COLLEAGUES: Colleagues are invaluable for helping us respond to our 'knowing not' and for providing alternative perspectives when our theories lead us astray. However, colleagues come in many guises and include close colleagues, as well as those in distant fields. OBSTACLES TO HUMILITY AND RESPECT: As obviously desirable as humility and respect seem to be, there are conflicts that prevent us from being humble and respectful. Such conflicts include other salient professional values, such as critical scepticism, competition and confidence. HONOURING THE OATH: Adoption of the values of humility and respect in medical education can be fostered through intentional behaviours, both as individuals and as a discipline. We can deliberately seek to broaden our horizons to promote intellectual humility. We can foster collaboration among colleagues who come from different places, both geographically and intellectually. Finally, we can pursue 'academic cultural exchanges' with other fields and disciplines to learn and share in different ways of thinking.

The power of written word: Reflection reduces errors of omission.

BACKGROUND: Medical trainees are expected to perform complex tasks while experiencing interruptions, which increases susceptibility to errors of omission. In our study, we examine whether documentation of clinical encounters increases reflective thinking and reduces errors of omission among novice learners in a simulated setting. METHODS: In 2021, 56 senior medical students participated in a simulated paging curriculum involving urgent inpatient cross-cover scenarios (sepsis and atrial fibrillation). Students responded to pages from standardized registered nurses (SRNs) via telephone, gathered history, and discussed clinical decision-making. Following the phone encounter, students documented a brief note (documentation encounter). A 'phone' score (number of checklist items completed in the phone encounter) and a 'combined' score (number of checklist items completed in the phone and documentation encounters) were calculated. Data were analyzed for differences between the phone scores (control) and combined scores using T-tests and McNemar test of symmetry. FINDINGS: Fifty-four students (96%) participated. Combined scores were higher than phone scores for sepsis (72.8 ± 11.3% vs. 67.9 ± 11.9%, p < 0.001) and atrial fibrillation (74.0 ± 10.1% vs. 67.6 ± 10.0%, p < 0.001) cases. Important items, such as ordering blood cultures for sepsis (p = 0.023) and placing the patient on telemetry for atrial fibrillation (p = 0.013), were more likely to be present when a note was documented. DISCUSSION: This study suggests that documentation provides a mechanism for learners to reflect, which could increase important diagnostic and therapeutic interventions. CONCLUSION: Documentation by novice medical learners may improve patient care by allowing for reflection and reducing errors of omission.

Medical education: changes and perspectives.

As medical education undergoes significant internationalization, it is important for the medical education community to understand how different countries structure and provide medical education. This article highlights the current landscape of medical education in China, particularly the changes that have taken place in recent years. It also examines policies and offers suggestions about future strategies for medical education in China. Although many of these changes reflect international trends, Chinese medical education has seen unique transformations that reflect its particular culture and history.

Applying the Bookmark method to medical education: standard setting for an aseptic technique station.

INTRODUCTION: The purpose was to evaluate the Bookmark standard-setting method for use on a performance-based assessment in medical education. METHODS: We compared cutscores for Aseptic performance assessment using the modified Angoff, Hofstee and modified Bookmark methods. RESULTS: The Angoff produced a cutscore of 62%, SD=18 and a percent passing (pp)= 64%. The Hofstee cutscore was 71%, SD=7 and pp=46%. Bookmark mean cutscores were 65.9% SD=10.7 and pp=42% for advanced beginners; 83.6%, SD=9.2 and pp=17% for competent and the proficient category resulted in a cutscore of 96.4% SD=3.9 and pp=1%. Faculty judges found the Bookmark method to be an easy and acceptable method. CONCLUSIONS: The Bookmark method was acceptable to faculty, has reasonable quality metrics when compared to other methods and can be a practical tool for establishing standards in performance-based examinations. The Bookmark method could be useful for establishing multiple levels of competency using the Dreyfus criteria.