Assessment of management reasoning: Design considerations drawn from analysis of simulated outpatient encounters.

PURPOSE: Management reasoning is a distinct subset of clinical reasoning. We sought to explore features to be considered when designing assessments of management reasoning. METHODS: This is a hybrid empirical research study, narrative review, and expert perspective. In 2021, we reviewed and discussed 10 videos of simulated (staged) physician-patient encounters, actively seeking actions that offered insights into assessment of management reasoning. We analyzed our own observations in conjunction with literature on clinical reasoning assessment, using a constant comparative qualitative approach. RESULTS: Distinguishing features of management reasoning that will influence its assessment include management scripts, shared decision-making, process knowledge, illness-specific knowledge, and tailoring of the encounter and management plan. Performance domains that merit special consideration include communication, integration of patient preferences, adherence to the management script, and prognostication. Additional facets of encounter variation include the clinical problem, clinical and nonclinical patient characteristics (including preferences, values, and resources), team/system characteristics, and encounter features. We cataloged several relevant assessment approaches including written/computer-based, simulation-based, and workplace-based modalities, and a variety of novel response formats. CONCLUSIONS: Assessment of management reasoning could be improved with attention to the performance domains, facets of variation, and variety of approaches herein identified.

A decade of faculty development for health professions educators: lessons learned from the Macy Faculty Scholars Program.

Faculty development (FD) programs are critical for providing the knowledge and skills necessary to drive positive change in health professions education, but they take many forms to attain the program goals. The Macy Faculty Scholars Program (MFSP), created by the Josiah Macy Jr. Foundation (JMJF) in 2010, intends to develop participants as leaders, scholars, teachers, and mentors. After a decade of implementation, an external review committee conducted a program evaluation to determine how well the program met its intended goals and defined options for ongoing improvement.The committee selected Stufflebeam's CIPP (context, input, process, products) framework to guide the program evaluation. Context and input components were derived from the MFSP description and demographic data, respectively. Process and product components were obtained through a mixed-methods approach, utilizing both quantitative and qualitative data obtained from participant survey responses, and curriculum vitae (CV).The evaluation found participants responded favorably to the program and demonstrated an overall increase in academic productivity, most pronounced during the two years of the program. Mentorship, community of practice, and protected time were cited as major strengths. Areas for improvement included: enhancing the diversity of program participants, program leaders and mentors across multiple sociodemographic domains; leveraging technology to strengthen the MFSP community of practice; and improving flexibility of the program.The program evaluation results provide evidence supporting ongoing investment in faculty educators and summarizes key strengths and areas for improvement to inform future FD efforts for both the MFSP and other FD programs.

Dealing with uncertainty in clinical reasoning: A threshold model and the roles of experience and task framing.

INTRODUCTION: Uncertainty is integral to clinical practice and clinical reasoning but has proven difficult to study and model. Little is known about how clinicians manage uncertainty. According to evidence-based medicine theory, clinicians should utilise new information to reduce uncertainty until reaching action thresholds for further information gathering or treatment. We examined the impact of experience and task framing on uncertainty thresholds and the extent to which these thresholds guided clinical decisions. Finally, we sought to determine the impact of framing by having participants provide threshold responses as a range or as specific numbers. METHODS: One hundred sixty-eight fourth-year medical students, 93 residents and 72 faculty were presented a case of viral pneumonia with a suspected superimposed bacterial infection. Participants identified their testing and treatment thresholds with either a specific number or an inter-threshold range of probabilities that would compel them to test further. Afterwards, they were told the patient had a 20% pre-test probability of a superimposed infection and asked whether they would treat the patient with antibiotics, order additional testing or neither. Responses were compared with their previously stated threshold values to assess decision-making consistency. RESULTS: Testing thresholds were 15.8%, 20.6% and 25.8%, treatment thresholds were 78.5%, 71.6% and 73.4% and threshold spans (difference between testing and treatment thresholds) were 62.7, 51 and 47.6 for students, residents and faculty, respectively. Sixty-four percent of respondents made judgements consistent with their thresholds, 28% escalated their decision (doing more than their thresholds predicted) and 7.6% de-escalated their decision (doing less than their thresholds predicted). Framing had an impact on both faculty and resident decisions and a larger impact on students. DISCUSSION: These findings help us understand how clinical reasoning and threshold determinations vary with clinical experience. As uncertainty can lead to unnecessary testing and cognitive discomfort, examining decision thresholds helps us ascertain how diagnostic and treatment decisions are made.

An outcomes research perspective on medical education: Has anything changed in the last 18 years?

PURPOSE: Medical education research focused on patient-centered outcomes holds the promise of improved decision-making by medical educators. In 2001, Prystowsky and Bordage demonstrated that patient-centered outcomes were evaluated in fewer than one percent of studies published in a survey of major medical education journals. Though many have called for increased inclusion of patient-centered outcomes in medical education literature, it remains uncertain to what degree this need has been addressed systematically. METHODS: Using the same data sources as in the original report (Academic Medicine, Medical Education, and Teaching and Learning in Medicine), we sought to replicate Prystowsky and Bordage's study. We extracted data from original empirical research reports from these three journal sources for the years 2014-2016. We selected 652 articles that met the inclusion criteria for further analysis. RESULTS: Study participants were largely trainees (64% of studies) or faculty (25% of studies). Only 2% of studies included patients as active or passive participants. Study outcomes reported were satisfaction (40% of studies), performance (39%), professionalism (20%), and cost (1%). CONCLUSIONS: These results do not differ significantly from the original 2001 study. The medical education literature as represented in these three prominent journals has made little progress in placing a greater focus on patient-centered outcomes.

Development and validation of conflict management attitude questionnaire for medical students.

BACKGROUND: Medical students should effectively manage conflicts in teamwork and communication with other team members. This study aimed to develop and validate a tool to evaluate attitude of medical students and physicians toward conflict management. METHOD: A multi-step process was employed to develop and validate a Conflict Management Attitude Questionnaire (CMAQ) based on the steps recommended in AMEE Guide No. 87. First, the initial items were obtained from the literature review and focus group. After cognitive interviews with the medical students and revision of the questionnaire, content validity was performed by experts. The construct validity and reliability of the questionnaire were assessed using exploratory Factor Analysis (EFA) and Cronbach's alpha coefficient, respectively. RESULTS: This multi-step process resulted in a 12-item, five-point Likert-type questionnaire with satisfactory construct validity. Exploratory factor analysis revealed three factors, comprising the four items from the "perceived interactions in conflict management" subscale loading on the first factor, and five items from the "perceived value of learning conflict management" subscale loading on the second factor, along with three items from the "perceived application of conflict management" subscale loading on the third factor. All subscales described 56.48% of the variance. Validation results showed that Content Validity Index (CVI) and Content Validity Ratio (CVR) were greater than 0.75. Cronbach's alpha coefficient was 0.791. CONCLUSION: This study showed that CMAQ has valid evidence for assessing the attitude of medical students toward conflict management with favorable psychometric properties and strong evidence of construct validity. However, due to the lack of evidence on any specific questionnaire to evaluate the attitude towards conflict management, future studies should conduct a confirmatory investigation regarding other aspects of medical students' attitudes toward conflict management.

Insights on my future job: implementing near-peer shadowing program for operating room freshmen.

BACKGROUND: As a main challenge in paramedical faculties of medical sciences, freshmen lose interest in their academic field of study and then job motivation. Lack of developed knowledge about their academic field and unfamiliarity with their future job's tasks and roles contribute to freshmen's job motivation loss. Various interventional programs have been implemented to improve students' job motivation by familiarizing them with their future job's duties and responsibilities. METHODS: This was one-group pretest-posttest design study in 2019-2020. Students grouped into pairs of a freshman (shadowee) with a senior (shadower) in a clinical setting during shadowing program. This program helps freshmen to comprehend and discover realities of their academic field and can help them change their perspectives regarding their future job's duties and responsibilities. The shadowees' main task was reflective observation on operating room events and interactions and to be active in the program, several tasks e.g., how to wear gloves, guns, and disinfect equipment were assigned to them exclusively under the supervision of senior students. The Hackman and Oldham's Job Diagnostic Survey (JDS) questionnaire and a novel Job Motivation Survey (JMS) questionnaire were distributed among participants. RESULTS: Fifty freshmen majoring in operating room participated in the shadowing program from November 2019 to January 2020. Before and after the program, they completed Hackman and Oldham's job diagnostic survey and researcher-made job motivation survey questionnaires. Results were indicative of a significant difference in job diagnostic survey questionnaire score, where overall pre-test and post-test scores before and after the intervention were 57.78 (±9.78) and 68.58 (±5.02), respectively; the score difference was statistically significant (P < 0.001). Moreover, the overall pre-test and post-test scores of the job motivation survey questionnaire were 25.16 (± 9.75) and 39.80 (±5.18), respectively; this score difference was statistically significant (P < 0.001). CONCLUSION: Shadowing program improved freshmen's realistic perception of their future job's duties and responsibility, and hence enhancing their job motivation and job recognition. As future work, in various disciplines, further studies need to evaluate the impact of such interventional programs in providing early insights for freshmen as well as in providing guidance on their plans for education, and future job.

Reflection on near-peer shadowing program: impact on operating room student's perception of their future profession.

BACKGROUND: Reflection is a key element in learning from observation and experience of future profession's roles and responsibilities in clinical encounters. Moreover, reflection helps students cope with the challenges, complexities, and uncertainties of professional development. Students' written reflections on clinical exposure offer valuable information, and their analysis provides instructors with invaluable insight into students' experiences. This study evaluated Operating Room students' written reflections on their first clinical exposure experiences towards their future profession through the shadowing program. METHODS: This study was a qualitative analysis on Operating Room freshmen's reflections in the undergraduate program of Zahedan and Zabol University of Medical Sciences in Iran. After the shadowing program, all participants were asked to write an unstructured written reflection, and these fifty written reflections were de-identified and independently analyzed using the thematic analysis approach. RESULTS: Qualitative analysis extracted 10 subthemes and four main themes including (i) Moving towards the guiding realities of future profession, (ii) Discovering milestones of realizing professional identity, (iii) Managing the emotions affecting the perception of future profession's desirability, and (iv) Excellence in professional growth and development. CONCLUSION: Reflecting on the experience of the shadowing program, the participants described being in the OR environment as a stimulating and valuable learning opportunity. Moreover, this experience helped improve their perception of future profession's realities, as well as initiate realization of professional identity and planning for professional developments.

Assessing changes in the quality of quantitative health educations research: a perspective from communities of practice.

BACKGROUND: As a community of practice (CoP), medical education depends on its research literature to communicate new knowledge, examine alternative perspectives, and share methodological innovations. As a key route of communication, the medical education CoP must be concerned about the rigor and validity of its research literature, but prior studies have suggested the need to improve medical education research quality. Of concern in the present study is the question of how responsive the medical education research literature is to changes in the CoP. We examine the nature and extent of changes in the quality of medical education research over a decade, using a widely cited study of research quality in the medical education research literature as a benchmark to compare more recent quality indicators. METHODS: A bibliometric analysis was conducted to examine the methodologic quality of quantitative medical education research studies published in 13 selected journals from September 2013 to December 2014. Quality scores were calculated for 482 medical education studies using a 10-item Medical Education Research Study Quality Instrument (MERSQI) that has demonstrated strong validity evidence. These data were compared with data from the original study for the same journals in the period September 2002 to December 2003. Eleven investigators representing 6 academic medical centers reviewed and scored the research studies that met inclusion and exclusion criteria. Primary outcome measures include MERSQI quality indicators for 6 domains: study design, sampling, type of data, validity, data analysis, and outcomes. RESULTS: There were statistically significant improvements in four sub-domain measures: study design, type of data, validity and outcomes. There were no changes in sampling quality or the appropriateness of data analysis methods. There was a small but significant increase in the use of patient outcomes in these studies. CONCLUSIONS: Overall, we judge this as equivocal evidence for the responsiveness of the research literature to changes in the medical education CoP. This study identified areas of strength as well as opportunities for continued development of medical education research.

On the validity of summative entrustment decisions.

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

Capturing outcomes of competency-based medical education: The call and the challenge.

There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic.

Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

Outcomes of competency-based medical education: A taxonomy for shared language.

As the global transformation of postgraduate medical training continues, there are persistent calls for program evaluation efforts to understand the impact and outcomes of competency-based medical education (CBME) implementation. The measurement of a complex educational intervention such as CBME is challenging because of the multifaceted nature of activities and outcomes. What is needed, therefore, is an organizational taxonomy to both conceptualize and categorize multiple outcomes. In this manuscript we propose a taxonomy that builds on preceding works to organize CBME outcomes across three domains: focus (educational, clinical), level (micro, meso, macro), and timeline (training, transition to practice, practice). We also provide examples of how to conceptualize outcomes of educational interventions across medical specialties using this taxonomy. By proposing a shared language for outcomes of CBME, we hope that this taxonomy will help organize ongoing evaluation work and catalyze those seeking to engage in the evaluation effort to help understand the impact and outcomes of CBME.

What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy.

BACKGROUND: Using revised Bloom's taxonomy, some medical educators assume they can write multiple choice questions (MCQs) that specifically assess higher (analyze, apply) versus lower-order (recall) learning. The purpose of this study was to determine whether three key stakeholder groups (students, faculty, and education assessment experts) assign MCQs the same higher- or lower-order level. METHODS: In Phase 1, stakeholders' groups assigned 90 MCQs to Bloom's levels. In Phase 2, faculty wrote 25 MCQs specifically intended as higher- or lower-order. Then, 10 students assigned these questions to Bloom's levels. RESULTS: In Phase 1, there was low interrater reliability within the student group (Krippendorf's alpha = 0.37), the faculty group (alpha = 0.37), and among three groups (alpha = 0.34) when assigning questions as higher- or lower-order. The assessment team alone had high interrater reliability (alpha = 0.90). In Phase 2, 63% of students agreed with the faculty as to whether the MCQs were higher- or lower-order. There was low agreement between paired faculty and student ratings (Cohen's Kappa range .098-.448, mean .256). DISCUSSION: For many questions, faculty and students did not agree whether the questions were lower- or higher-order. While faculty may try to target specific levels of knowledge or clinical reasoning, students may approach the questions differently than intended.

Mapping clinical reasoning literature across the health professions: a scoping review.

BACKGROUND: Clinical reasoning is at the core of health professionals' practice. A mapping of what constitutes clinical reasoning could support the teaching, development, and assessment of clinical reasoning across the health professions. METHODS: We conducted a scoping study to map the literature on clinical reasoning across health professions literature in the context of a larger Best Evidence Medical Education (BEME) review on clinical reasoning assessment. Seven databases were searched using subheadings and terms relating to clinical reasoning, assessment, and Health Professions. Data analysis focused on a comprehensive analysis of bibliometric characteristics and the use of varied terminology to refer to clinical reasoning. RESULTS: Literature identified: 625 papers spanning 47 years (1968-2014), in 155 journals, from 544 first authors, across eighteen Health Professions. Thirty-seven percent of papers used the term clinical reasoning; and 110 other terms referring to the concept of clinical reasoning were identified. Consensus on the categorization of terms was reached for 65 terms across six different categories: reasoning skills, reasoning performance, reasoning process, outcome of reasoning, context of reasoning, and purpose/goal of reasoning. Categories of terminology used differed across Health Professions and publication types. DISCUSSION: Many diverse terms were present and were used differently across literature contexts. These terms likely reflect different operationalisations, or conceptualizations, of clinical reasoning as well as the complex, multi-dimensional nature of this concept. We advise authors to make the intended meaning of 'clinical reasoning' and associated terms in their work explicit in order to facilitate teaching, assessment, and research communication.

Diversity in Emergency Medicine: Are We Supporting a Career Interest in Emergency Medicine for Everyone?

STUDY OBJECTIVE: Women and students underrepresented in medicine are less likely to apply for residency in emergency medicine. The latter are from racial or ethnic populations that are underrepresented as physicians relative to the general population. The factors that result in lower application rates from women and groups underrepresented in medicine are inadequately described in the literature. This study's objective was to test whether female students and those underrepresented in medicine have lower interest in emergency medicine even after controlling for academic ability, student indebtedness, and common career values consistent with emergency medicine career interest. METHODS: Secondary data analyses were conducted on a cross section of all residency applicants from 2005 to 2010. Data sources included American Medical College Application Service, the Electronic Residency Application Service, and the Graduating Questionnaire. Data linkage was by the Association of American Medical Colleges and provided deidentified to the authors. A binary logistic regression model was fitted with the outcome variable planned career into emergency medicine versus another specialty on the Graduating Questionnaire. The binary logistic regression model independent variables included demographics, student attitudes, debt, grade point average, standardized tests, and medical school experiences. RESULTS: The binary logistic regression model included 17,067 individuals. Being a woman (odds ratio 0.75) and from a background underrepresented in medicine (odds ratio 0.68) independently correlated with lower emergency medicine interest. Age, medical debt, importance of work-life balance, confidence in specialty choice, and plan to care for underserved populations were positively associated with emergency medicine interest. Importance of specialty competitiveness and importance of mentorship advice were correlated with lower emergency medicine interest. CONCLUSION: Female medical students and those underrepresented in medicine were less likely to plan for a career in emergency medicine. This correlation remained significant even when other previously identified factors that have predicted a career in emergency medicine were controlled for.

Use of evidence in health professions education: Attitudes, practices, barriers and supports.

Introduction: Health professions educators are increasingly called upon to apply an evidence-informed approach to teaching and assessment practices. There is scant empirical research exploring educators' attitudes, practices, and perceived barriers and supports to using research evidence in educational practice. Methods: We conducted a survey of AMEE members to explore three domains related to evidence-informed health professions education: (1) attitudes; (2) practices; and (3) supports and barriers. Analyses involved descriptive statistics to characterize participants' responses per domain, exploratory factor analysis, and multivariate regression analyses. Results: Three hundred ninety-six (∼10%) participants representing health professions educators (HPEs) and non HPEs (e.g. PhDs) and different roles (e.g. teacher, administrator) completed the survey. Attitudes toward evidence-informed HPE were generally favorable. Several barriers preclude participants from engaging in evidence-informed approaches to health professions education (HPE). Discussion: This study provides preliminary evidence on the attitudes toward and perceived barriers and supports of research use in HPE from different groups of HP educators, clinicians, and administrators. The findings for each of the three domains require additional exploration using qualitative methodologies. Conclusion: Targeted interventions designed to increase the uptake of research in HPE should consider different stakeholder groups' perceptions regarding these approaches, current vs. best practices, and factors that may impede evidence-informed approaches.

The terminology of clinical reasoning in health professions education: Implications and considerations.

Introduction: Clinical reasoning is considered to be at the core of health practice. Here, we report on the diversity and inferred meanings of the terms used to refer to clinical reasoning and consider implications for teaching and assessment. Methods: In the context of a Best Evidence Medical Education (BEME) review of 625 papers drawn from 18 health professions, we identified 110 terms for clinical reasoning. We focus on iterative categorization of these terms across three phases of coding and considerations for how terminology influences educational practices. Results: Following iterative coding with 5 team members, consensus was possible for 74, majority coding was possible for 16, and full team disagreement existed for 20 terms. Categories of terms included: purpose/goal of reasoning, outcome of reasoning, reasoning performance, reasoning processes, reasoning skills, and context of reasoning. Discussion: Findings suggest that terms used in reference to clinical reasoning are non-synonymous, not uniformly understood, and the level of agreement differed across terms. If the language we use to describe, to teach, or to assess clinical reasoning is not similarly understood across clinical teachers, program directors, and learners, this could lead to confusion regarding what the educational or assessment targets are for "clinical reasoning."

The impact of informed self-assessment on the development of medical students' learning goals.

PURPOSE: This study investigates the contributions of self-assessment (SA) and external feedback on the development of learning goals (LG) and the influence on LG recall and implementation in medical students. METHODS: Following a standardized patient (SP) assessment, 168 pre-clinical medical students completed a SA, received SP feedback and created a LG. LG were categorized by source. Two weeks later, students recalled LG and described implementation. Chi-squared analyses were used to test the associations. RESULTS: SA influenced LG for 82.8% of students whereas SP feedback influenced LG for 45.9%. Students rarely generated LG based on SA when they received discordant feedback (5.4%), but sometimes incorporated feedback discordant from their SA into LG (14.9%). Students who created LG based on SP feedback were more likely to recall LG than those who created LG based on SA, 89.7 versus 67.6%, p < 0.05 and implement their LG, 72.4 versus 48.9%, χ2(1) = 5.3, p = 0.017. Students who reported receiving effective feedback were more likely to implement their LG than those reporting adequate feedback, 60.9 versus 37.9%, χ2(1) = 8.0, p = 0.01. CONCLUSIONS: SA is an essential part of goal setting and subsequent action. Perception of feedback plays a crucial role in LG implementation.

Exploring the characteristics and context that allow Master Adaptive Learners to thrive.

Because change is ubiquitous in healthcare, clinicians must constantly make adaptations to their practice to provide the highest quality care to patients. In a previous article, Cutrer et al. described a metacognitive approach to learning based on self-regulation, which facilitates the development of the Master Adaptive Learner (MAL). The MAL process helps individuals to cultivate and demonstrate adaptive expertise, allowing them to investigate new concepts (learn) and create new solutions (innovate). An individual's ability to learn in this manner is driven by several internal characteristics and is also impacted by numerous aspects of their context. In this article, the authors examine the important internal and contextual factors that can impede or foster Master Adaptive Learning.