How Teachers Can Promote Assessment for Learning During All Phases of Learning.

Assessment for Learning (AFL) includes all educational activities designed to improve teaching and learning through gathering, sharing, and acting on information. AFL expands on the concept of formative assessment-which focuses mainly on an in-the-moment assessment activity-to include all processes that facilitate teachers and learners actively seeking and interpreting evidence to guide learning. AFL involves teachers and learners as partners and uses evidence to identify what the learner needs to learn (planning), review where the learner is in their learning (observing), and strategize how to maximize learning (supporting). 1 We provide the following guidelines for teachers to implement AFL within these 3 phases of a learning activity. Teachers should tailor their choice of AFL strategies to suit their relationship with the learner and learning environment context, aiming to support the development of self-regulated learning and metacognitive skills.

"Finding My Piece in That Puzzle": A Qualitative Study Exploring How Medical Students at Four U.S. Schools Envision Their Future Professional Identity in Relation to Health Systems.

PURPOSE: Health systems science (HSS) curricula equip future physicians to improve patient, population, and health systems outcomes (i.e., to become "systems citizens"), but the degree to which medical students internalize this conception of the physician role remains unclear. This study aimed to explore how students envision their future professional identity in relation to the system and identify experiences relevant to this aspect of identity formation. METHOD: Between December 2018 and September 2019, authors interviewed 48 students at 4 U.S. medical schools with HSS curricula. Semistructured interviews were audiorecorded, transcribed, and analyzed iteratively using inductive thematic analysis. Interview questions explored how students understood the health system, systems-related activities they envisioned as future physicians, and experiences and considerations shaping their perspectives. RESULTS: Most students anticipated enacting one or more systems-related roles as a future physician, categorized as "bottom-up" efforts enacted at a patient or community level (humanist, connector, steward) or "top-down" efforts enacted at a system or policy level (system improver, system scholar, policy advocate). Corresponding activities included attending to social determinants of health or serving medically underserved populations, connecting patients with team members to address systems-related barriers, stewarding health care resources, conducting quality improvement projects, researching/teaching systems topics, and advocating for policy change. Students attributed systems-related aspirations to experiences beyond HSS curricula (e.g., low-income background; work or volunteer experience; undergraduate studies; exposure to systems challenges affecting patients; supportive classmates, faculty, and institutional culture). Students also described future-oriented considerations promoting or undermining identification with systems-related roles (responsibility, affinity, ability, efficacy, priority, reality, consequences). CONCLUSIONS: This study illuminates systems-related roles medical students at 4 schools with HSS curricula envisioned as part of their future physician identity and highlights past/present experiences and future-oriented considerations shaping identification with such roles. These findings inform practical strategies to support professional identity formation inclusive of systems engagement.

Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships.

PURPOSE: To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation. METHOD: This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system. RESULTS: Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning. CONCLUSIONS: Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.

Are You Thinking What I'm Thinking? Exploring Response Process Validity Evidence for a Workplace-based Assessment for Operative Feedback.

OBJECTIVE: Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assessment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery. DESIGN: Semi-structured interviews explored the reasonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measuring Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment questions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualitative content analysis to generate themes. SETTING: Single tertiary academic medical center. PARTICIPANTS: Eight residents and 13 faculty within the Department of Otolaryngology-Head and Neck Surgery participating in a 6-month pilot of SIMPL. RESULTS: We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty. CONCLUSIONS: Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a continued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment.

Exploring perspectives on health professions education scholarship units from sub-Saharan Africa.

INTRODUCTION: There has been a marked increase in institutional structures developed to support health professions education scholarship recently. These health professions education scholarship units (HPESUs) engage in a diverse range of activities. Previous work provided insight into factors that influence the functioning of such units, but data from European, Asian, Latin American, and African contexts was absent, potentially leading to a single world-view informing international standards for HPESUs. This aim of this study was to explore perspectives from sub-Saharan Africa (SSA) in response to this omission. METHODS: Situated within an interpretivist paradigm, the research team conducted semi-structured interviews with nine HPESU leaders in SSA, exploring how participants experienced and understood the functioning of their units. Despite efforts to have representation from across the region, most participants were from South Africa. The researchers analysed data thematically using the theory of institutional logics as an analytical frame. RESULTS: Several aspects of the HPESUs aligned with the previously identified logics of academic research, service and teaching; and of a cohesive education continuum. By contrast, leaders described financial sustainability as a more prominent logic than financial accountability. DISCUSSION: The similarities identified in this study may reflect isomorphism-a process which sees institutions within a similar field becoming more alike, particularly as newer institutions seek to acquire legitimacy within that field. An important caveat, however, is that isomorphism tends to occur across similar institutional contexts, which was not the case in this study. Understanding these differences is key as these HPESUs move to foster scholarship that can respond to the region's unique context.

Boyer and Beyond: An Interview Study of Health Professions Education Scholarship Units in the United States and a Synthetic Framework for Scholarship at the Unit Level.

PURPOSE: Health professions education scholarship units (HPESUs) in the United States are large in number and diverse in purpose, activities, and contributions. Although each of these units shares a commitment to scholarship, there is no synthetic framework to accurately represent and evaluate their activities and contributions. This study aimed to provide such a framework. METHOD: The authors examined data collected from 11 U.S. HPESU directors. Interviews occurred between April 2015 and February 2016. The research team used a combination of deductive and inductive qualitative techniques to analyze the interview transcripts. The deductive portion drew on Boyer's four-part framework of scholarship; the inductive portion produced a new conceptualization of scholarship at the HPESU level. RESULTS: The scholarly activities of HPESUs generally align with Boyer's four types of scholarship-discovery, integration, application, and teaching. However, this categorization fails to capture the interconnectedness and variety of purposes served by these activities. Both are important when considering how best to represent the scholarly contributions made by HPESUs. From their analysis of interviews, the authors developed a three-part framework characterizing HPESU scholarly activities: supporting a scholarly approach to education, supporting educational scholarship within the institution, and supporting HPESU members' scholarship. CONCLUSIONS: The authors contend that the three-part, unit-level framework for scholarship constructed in this study brings clarity and understanding to the purpose, activities, and contributions made by HPESUs in the United States. The proposed framework may allow unit directors to better justify and advocate for the resources needed to further promote the work of HPESUs.

Understanding self-assessment as an informed process: residents' use of external information for self-assessment of performance in simulated resuscitations.

Self-directed learning requires self-assessment of learning needs and performance, a complex process that requires collecting and interpreting data from various sources. Learners' approaches to self-assessment likely vary depending on the learner and the context. The aim of this study was to gain insight into how learners process external information and apply their interpretation of this information to their self-assessment and learning during a structured educational activity. The study combined quantitative performance data with qualitative interview data. Pediatric residents led video-recorded simulated resuscitations and rated their crisis resource management skills on a validated 6-item instrument. Three independent observers rated the videos using the same instrument. During semi-structured interviews, each resident reviewed the video, rerated performance, discussed the self-assessment process, and interpreted feedback and observer scores. Transcripts were analyzed for themes. Sixteen residents participated. Residents' self-assessed scores ranged widely but usually fell within two points of the observers. They almost universally lowered their scores when self-assessing after the video review. Five major themes emerged from qualitative analysis of their interviews: (1) residents found self-assessment important and useful in certain contexts and conditions; (2) residents varied in their self-directed learning behaviors after the simulated resuscitation; (3) quantitative observer assessment had limited usefulness; (4) video review was difficult but useful; and (5) residents focused on their weaknesses and felt a need for constructive feedback to enhance learning. The residents in our study almost uniformly embraced the importance of self-assessment for all medical professionals. Even though video review had a negative impact on their self-assessment scores and was perceived as painful, residents saw this as the most useful aspect of the study exercises residents. They were less accepting of the quantitative assessment by observers. Residents explained their tendency to focus on weaknesses as a way to create an incentive for learning, demonstrating that self-assessment is closely linked to self-directed learning. How learners can use video review and external assessment most effectively to guide their self-directed learning deserves further study.