What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study.

PURPOSE: The purpose of this study was to enrich understanding about the perceived benefits and drawbacks of constructed response short-answer questions (CR-SAQs) in preclerkship assessment using Norcini's criteria for good assessment as a framework. METHODS: This multi-institutional study surveyed students and faculty at three institutions. A survey using Likert scale and open-ended questions was developed to evaluate faculty and student perceptions of CR-SAQs using the criteria of good assessment to determine the benefits and drawbacks. Descriptive statistics and Chi-square analyses are presented, and open responses were analyzed using directed content analysis to describe benefits and drawbacks of CR-SAQs. RESULTS: A total of 260 students (19%) and 57 faculty (48%) completed the survey. Students and faculty report that the benefits of CR-SAQs are authenticity, deeper learning (educational effect), and receiving feedback (catalytic effect). Drawbacks included feasibility, construct validity, and scoring reproducibility. Students and faculty found CR-SAQs to be both acceptable (can show your reasoning, partial credit) and unacceptable (stressful, not USMLE format). CONCLUSIONS: CR-SAQs are a method of aligning innovative curricula with assessment and could enrich the assessment toolkit for medical educators.

Using a boundary crossing lens to understand basic science educator and clinical educator collaboration in instructional design.

PURPOSE: Collaborations between basic science educators (BE) and clinical educators (CE) in medical education are common and necessary to create integrated learning materials. However, few studies describe experiences of or processes used by educators engaged in interdisciplinary teamwork. We use the lens of boundary crossing to explore processes described by BE and CE that support the co-creation of integrated learning materials, and the impact that this work has on them. MATERIALS AND METHODS: We conducted qualitative content analysis on program evaluation data from 27 BE and CE who worked on 12 teams as part of a multi-institutional instructional design project. RESULTS: BE and CE productively engaged in collaboration using boundary crossing mechanisms. These included respecting diverse perspectives and expertise and finding efficient processes for completing shared work that allow BE and CE to build on each other's contributions. BE and CE developed confidence in connecting clinical concepts with causal explanations, and willingness to engage in and support such collaborations at their own institutions. CONCLUSIONS: BE and CE report the use of boundary crossing mechanisms that support collaboration in instructional design. Such practices could be harnessed in future collaborations between BE and CE.

Connecting Biochemistry Knowledge to Patient Care in the Clinical Workplace: Senior Medical Students' Perceptions about Facilitators and Barriers.

Phenomenon: Medical students have difficulties applying knowledge about biomedical mechanisms learned before clerkships to patient care activities. Many studies frame this challenge as a problem of basic science knowledge transfer predominantly influenced by students' individual cognitive processes. Social cognitive theory would support extending this framing to the interplay between the individual's cognition, the environment, and their behaviors. This study investigates senior medical students' experiences of biochemistry knowledge use during workplace learning and examines how their experiences were influenced by interactions with people and other elements of the clinical learning environment. Approach: The authors used a qualitative approach with a constructivist orientation. From September to November 2020 they conducted semi-structured interviews with 11 fourth-year medical students at one institution who had completed the pre-clerkship curriculum, core clinical clerkships, and the United States Medical Licensing Exam Step 1. The authors identified themes using thematic analysis. Findings: Participants reported that they infrequently used or connected to biochemistry knowledge in workplace patient care activities, yet all had examples of such connections that they found valuable to learning. Most participants felt the responsibility for making connections between biochemistry knowledge and activities in the clinical workplace should be shared between themselves and supervisors, but connections were often recognized and acted on only by the student. Connections that participants described prompted their effort to retrieve knowledge or fill a perceived learning gap. Participants identified multiple barriers and facilitators to connecting, including supervisors' behaviors and perceived knowledge, and "patients seen" in clerkships. Participants also reported learning biochemistry during USMLE Step 1 study that did not connect to patient care activities, underscoring a perception of disconnect. Insights: This study identifies specific personal, social, and physical environmental elements that influence students' perceived use of biochemistry during patient care activities. Though these findings may be most significant for biochemistry, they likely extend to other basic science disciplines. Students' self-directed efforts to connect to their biochemistry knowledge could be augmented by increased social support from clinical supervisors, which in turn likely requires faculty development. Opportunities for connection could be enhanced by embedding into the environment instructional strategies or technologies that build on known authentic connections between biochemistry and "patients seen" in clerkships. These efforts could strengthen student learning, improve clinical supervisors' self-efficacy, and better inform curriculum design.

A Multi-institutional Study of the Feasibility and Reliability of the Implementation of Constructed Response Exam Questions.

PROBLEM: Some medical schools have incorporated constructed response short answer questions (CR-SAQs) into their assessment toolkits. Although CR-SAQs carry benefits for medical students and educators, the faculty perception that the amount of time required to create and score CR-SAQs is not feasible and concerns about reliable scoring may impede the use of this assessment type in medical education. INTERVENTION: Three US medical schools collaborated to write and score CR-SAQs based on a single vignette. Study participants included faculty question writers (N = 5) and three groups of scorers: faculty content experts (N = 7), faculty non-content experts (N = 6), and fourth-year medical students (N = 7). Structured interviews were performed with question writers and an online survey was administered to scorers to gather information about their process for creating and scoring CR-SAQs. A content analysis was performed on the qualitative data using Bowen's model of feasibility as a framework. To examine inter-rater reliability between the content expert and other scorers, a random selection of fifty student responses from each site were scored by each site's faculty content experts, faculty non-content experts, and student scorers. A holistic rubric (6-point Likert scale) was used by two schools and an analytic rubric (3-4 point checklist) was used by one school. Cohen's weighted kappa (κw) was used to evaluate inter-rater reliability. CONTEXT: This research study was implemented at three US medical schools that are nationally dispersed and have been administering CR-SAQ summative exams as part of their programs of assessment for at least five years. The study exam question was included in an end-of-course summative exam during the first year of medical school. IMPACT: Five question writers (100%) participated in the interviews and twelve scorers (60% response rate) completed the survey. Qualitative comments revealed three aspects of feasibility: practicality (time, institutional culture, teamwork), implementation (steps in the question writing and scoring process), and adaptation (feedback, rubric adjustment, continuous quality improvement). The scorers' described their experience in terms of the need for outside resources, concern about lack of expertise, and value gained through scoring. Inter-rater reliability between the faculty content expert and student scorers was fair/moderate (κw=.34-.53, holistic rubrics) or substantial (κw=.67-.76, analytic rubric), but much lower between faculty content and non-content experts (κw=.18-.29, holistic rubrics; κw=.59-.66, analytic rubric). LESSONS LEARNED: Our findings show that from the faculty perspective it is feasible to include CR-SAQs in summative exams and we provide practical information for medical educators creating and scoring CR-SAQs. We also learned that CR-SAQs can be reliably scored by faculty without content expertise or senior medical students using an analytic rubric, or by senior medical students using a holistic rubric, which provides options to alleviate the faculty burden associated with grading CR-SAQs.

Using a Curricular Vision to Define Entrustable Professional Activities for Medical Student Assessment.

BACKGROUND: The new UCSF Bridges Curriculum aims to prepare students to succeed in today's health care system while simultaneously improving it. Curriculum redesign requires assessment strategies that ensure that graduates achieve competence in enduring and emerging skills for clinical practice. AIM: To design entrustable professional activities (EPAs) for assessment in a new curriculum and gather evidence of content validity. SETTING: University of California, San Francisco, School of Medicine. PARTICIPANTS: Nineteen medical educators participated; 14 completed both rounds of a Delphi survey. PROGRAM DESCRIPTION: Authors describe 5 steps for defining EPAs that encompass a curricular vision including refining the vision, defining draft EPAs, developing EPAs and assessment strategies, defining competencies and milestones, and mapping milestones to EPAs. A Q-sort activity and Delphi survey involving local medical educators created consensus and prioritization for milestones for each EPA. PROGRAM EVALUATION: For 4 EPAs, most milestones had content validity indices (CVIs) of at least 78 %. For 2 EPAs, 2 to 4 milestones did not achieve CVIs of 78 %. DISCUSSION: We demonstrate a stepwise procedure for developing EPAs that capture essential physician work activities defined by a curricular vision. Structured procedures for soliciting faculty feedback and mapping milestones to EPAs provide content validity.

When students' words hurt: 12 tips for helping faculty receive and respond constructively to student evaluations of teaching.

Student evaluations of curricular experiences and instructors are employed by institutions to obtain feedback and guide improvement. However, to be effective, evaluations must prompt faculty action. Unfortunately, evaluative comments that engender strong reactions may undermine the process by hindering innovation and improvement steps. The literature suggests that faculty interpret evaluation feedback as a judgment not just on their teaching ability but on their personal and professional identity. In this context, critical evaluations, even when constructively worded, can result in disappointment, hurt, and shame. The COVID pandemic has challenged institutions and faculty to repeatedly adapt curricula and educational practices, heightening concerns for faculty burnout. In this context, the risk of 'words that hurt' is higher than ever. This article offers guidance for faculty and institutions to support effective responses to critical feedback and ameliorate counterproductive effects of learner evaluations.

Professional Identity Formation of Basic Science Medical Educators: A Qualitative Study of Identity Supports and Threats.

PURPOSE: Basic science medical educators (BSME) play a vital role in the training of medical students, yet little is known about the factors that shape their professional identities. This multi-institutional qualitative study investigated factors that support and threaten the professional identity formation (PIF) of these medical educators. METHOD: A qualitative descriptive study was conducted with a purposive sample of 58 BSME from 7 allopathic medical schools in the U.S. In-depth semi-structured interviews of individual BSME were conducted between December 2020 and February 2021 to explore the facilitators and barriers shaping the PIF of BSME. Thematic analysis was conducted. RESULTS: Factors shaping PIF were grouped into 3 broad domains: personal, social, and structural. Interrelated themes described a combination of factors that pushed BSME into teaching (early or positive teaching experiences) and kept them there (satisfaction and rewards of teaching, communities of like-minded people), as well as factors that challenged their PIF (misunderstanding from medical students, clinical, and research faculty, lack of formal training programs, and lack of tenure-track educator positions). The structural environment was reported to be crucial for PIF and determined whether BSME felt that they belonged and were valued. CONCLUSIONS: This study shows that although most BSME derive a sense of fulfillment and meaning from their role as medical educators, they face considerable obstacles during their PIF. Structural change and support are needed to increase recognition, value, promotion, and belonging for BSME to improve the satisfaction and retention of this important group of faculty.

The design and utility of institutional teaching awards: a literature review.

BACKGROUND: Institutional teaching awards have been used widely in higher education since the 1970s. Nevertheless, a comprehensive review of the literature on such awards has not been published since 1997. AIM: We conducted a literature review to learn as much as possible about the design (e.g., formats, selection processes) and utility (e.g., impact on individuals and institutions) of teaching awards in order to provide information for use in designing, implementing, or evaluating award programs. METHODS: We searched electronic databases for English-language publications on awards for exemplary teaching. Targeted publications included descriptions and/or investigations of award programs, their impact, and theoretical or conceptual models for awards programs. Screening was conducted by dual review; a third reviewer was assigned for disagreements. Data were analyzed qualitatively. Results were summarized descriptively. RESULTS: We identified 1302 publications for initial relevancy screening by title and abstract. We identified an additional 23 publications in a follow-up search. The full text of 126 publications was reviewed for further relevance. A total of 62 publications were identified as relevant, and of these 43 met our criteria for inclusion. Of the 43, 19 described the design features of 24 awards; 20 reports discussed award utility. Nomination and selection processes and benefits (e.g., plaques) varied as did perceived impact on individuals and institutions. CONCLUSION: Limited evidence exists regarding design and utility of teaching awards. Awards are perceived as having potential for positive impact, including promotions, but may also have unintended negative consequences. Future research should investigate the impact of awards on personal and professional development, and how promotion and tenure committees perceive awards.

Thinking Slow More Quickly: Development of Integrated Illness Scripts to Support Cognitively Integrated Learning and Improve Clinical Decision-Making.

Illness scripts describe the mental model used by experienced clinicians to store and recall condition-specific knowledge when making clinical decisions. Studies demonstrate that novice clinicians struggle to develop and apply strong illness scripts. We developed the Integrated Illness Script and Mechanism of Disease (IIS-MOD) map framework to address this challenge.

Medical Biochemistry Without Rote Memorization: Multi-Institution Implementation and Student Perceptions of a Nationally Standardized Metabolic Map for Learning and Assessment.

Despite the growing number of patients worldwide with metabolism-related chronic diseases, medical biochemistry education is commonly perceived as focusing on recall of facts irrelevant for patient care. The authors suggest that this focus on rote memorization of pathways creates excessive cognitive load that may interfere with learners' development of an integrated understanding of metabolic regulation and dysregulation. This cognitive load can be minimized by providing appropriate references during learning and assessment. Biochemistry educators collaborated to develop a medically relevant pathways of human metabolism map (MetMap) that is now being used at many medical schools as a nationally standardized resource during learning and assessments. To assess impact, students from three medical schools were surveyed about its benefits and disadvantages. Responses were obtained from 481 students (84%) and were examined using thematic analysis. Five main themes emerged as perceived benefits of using the MetMap: (1) aids visual and mental organization, (2) promotes deep learning and applied understanding, (3) decreases emphasis on memorization, (4) reduces anxiety on exams, and (5) aids recall. Perceived disadvantages were (1) fear of underpreparation for licensing exams, (2) overwhelming nature of the map, and (3) reduced motivation for and time spent studying. Results affirm that students' perceive use of the MetMap promotes focus on broader metabolic concepts and deep versus surface learning, supporting a shift in cognitive load toward desired goals. Although the long-term impact on learning needs to be further studied, the use of the MetMap represents a step toward open-reference exams that reflect "real-world" practice.

Toward Defining the Foundation of the MD Degree: Core Entrustable Professional Activities for Entering Residency.

Currently, no standard defines the clinical skills that medical students must demonstrate upon graduation. The Liaison Committee on Medical Education bases its standards on required subject matter and student experiences rather than on observable educational outcomes. The absence of such established outcomes for MD graduates contributes to the gap between program directors' expectations and new residents' performance.In response, in 2013, the Association of American Medical Colleges convened a panel of experts from undergraduate and graduate medical education to define the professional activities that every resident should be able to do without direct supervision on day one of residency, regardless of specialty. Using a conceptual framework of entrustable professional activities (EPAs), this Drafting Panel reviewed the literature and sought input from the health professions education community. The result of this process was the publication of 13 core EPAs for entering residency in 2014. Each EPA includes a description, a list of key functions, links to critical competencies and milestones, and narrative descriptions of expected behaviors and clinical vignettes for both novice learners and learners ready for entrustment.The medical education community has already begun to develop the curricula, assessment tools, faculty development resources, and pathways to entrustment for each of the 13 EPAs. Adoption of these core EPAs could significantly narrow the gap between program directors' expectations and new residents' performance, enhancing patient safety and increasing residents', educators', and patients' confidence in the care these learners provide in the first months of their residency training.

Learning objectives in radiology education: why you need them and how to write them.

Learning objectives are a critical step in the creation and implementation of a radiology curriculum. Their use is mandated by the Liaison Committee on Medical Education, the Accreditation Council on Graduate Medical Education, and the Accreditation Council on Continuing Medical Education, but more importantly they can have a significant beneficial impact on quality of radiology education programs. Learning objectives guide student learning, help clarify our teaching goals, and simplify learner testing and evaluation. This article will review the components of a proper learning objective and provide a simple, straightforward approach to writing them effectively.