Is There a Problem With Evidence in Health Professions Education?

ABSTRACT: What constitutes evidence, what value evidence has, and how the needs of knowledge producers and those who consume the knowledge produced as evidence might be better aligned are questions that continue to challenge the health sciences. In health professions education (HPE), debates on these questions have ebbed and flowed with little sense of resolution or progress. In this article, the authors explore whether there is a problem with evidence in HPE using thought experiments anchored in Argyris' learning loops framework.From a single-loop perspective ("How are we doing?"), there may be many problems with evidence in HPE, but little is known about how research evidence is being used in practice and policy. A double-loop perspective ("Could we do better?") suggests expectations of knowledge producers and knowledge consumers might be too high, which suggests more systemwide approaches to evidence-informed practice in HPE are needed. A triple-loop perspective ("Are we asking the right questions?") highlights misalignments between the dynamics of research and decision-making, such that scholarly inquiry may be better approached as a way of advancing broader conversations, rather than contributing to specific decision-making processes.The authors ask knowledge producers and consumers to be more attentive to the translation from knowledge to evidence. They also argue for more systematic tracking and audit of how research knowledge is used as evidence. Given that research does not always have to serve practical purposes or address the problems of a particular program or institution, the relationship between knowledge and evidence should be understood in terms of changing conversations, as well as influencing decisions.

How to discuss transferability of qualitative research in health professions education.

Transferability is commonly identified as a quality criterion for qualitative research. This criterion was introduced by Lincoln and Guba to describe the degree to which a study's findings can be transferred to other contexts, settings or respondents. In this How To paper, we present a more nuanced, multidimensional view of transferability and explain relevant concepts, reflexive approaches and specific techniques to guide researchers in discussing transferability. We identify three dimensions of transferability for use in many approaches to qualitative research: applicability, resonance and theoretical engagement. Transferability as applicability relates to providing sufficient information for readers to evaluate the relevance of findings to other contexts. Transferability as resonance requires the researcher to present the research in a way that evokes a sense of familiarity or shared experience. Transferability as theoretical engagement refers to ways the researcher uses theory to frame a problem, connects findings to existing constructs and/or proposes a model or theory that could explain a process or phenomenon. We encourage researchers to consider all three dimensions of transferability when developing and presenting their work, keeping in mind that some dimensions may be more relevant than others based on study methodology and project goals.

Researching lived experience in health professional education.

INTRODUCTION: Qualitative approaches have flourished in medical education research. Many research articles use the term 'lived experience' to describe the purpose of their study, yet we have noticed contradictory uses and misrepresentations of this term. In this conceptual paper, we consider three sources of these contradictions and misrepresentations: (1) the conflation of perspectives with experiences; (2) the conflation of experience with lived experience; and (3) the conflation of researching lived experience with phenomenology. We offer suggestions to facilitate more precise use of terminology. ARGUMENT: Our starting point is to free researchers from unnecessary shackles: Not every problem in medical education should be studied through experience, nor should every study of experience be phenomenological. Data based on participants' perceptions, beliefs, opinions and thoughts, while based on reflections of experiences, are not in and of themselves accounts of experience. Lived experiences are situated, primal and pre-reflective; perspectives are more abstract. Lived experience-as opposed to experiences as such-deeply attune to bodies, relationality, space and time. There is also a difference between experiences as lived, how a person makes sense of these and what the researcher interprets and represents. Phenomenology is a meaningful approach to the study of lived experience, but other approaches, such as narrative inquiry and self-study, can also offer useful avenues for undertaking this type of research. DISCUSSION: We aim to broaden researchers' scope with this paper and equip researchers with the information they need to be clear about the meaning and use of the terms experience and lived experience. We also hope to open new methodological possibilities for researching experiences as lived and, through highlighting tensions, to prompt researchers of lived experience to strive for ontological closeness and resonance.

A Scoping Review of Interprofessional Simulation-Based Team Training Programs.

SUMMARY STATEMENT: Interprofessional simulation-based team training (ISBTT) is promoted as a strategy to improve collaboration in healthcare, and the literature documents benefits on teamwork and patient safety. Teamwork training in healthcare is traditionally grounded in crisis resource management (CRM), but it is less clear whether ISBTT programs explicitly take the interprofessional context into account, with complex team dynamics related to hierarchy and power. This scoping review examined key aspects of published ISBTT programs including (1) underlying theoretical frameworks, (2) design features that support interprofessional learning, and (3) reported behavioral outcomes. Of 4854 titles identified, 58 articles met inclusion criteria. Most programs were based on CRM and related frameworks and measured CRM outcomes. Only 12 articles framed ISBTT as interprofessional education and none measured all interprofessional competencies. The ISBTT programs may be augmented by integrating theoretical concepts related to power and intergroup relations in their design to empower participants to navigate complex interprofessional dynamics.

How Primary Care Clinicians Process Patient Death: Logistics, Emotions, and Opportunities for Structural Support.

BACKGROUND: Navigating the logistics and emotional processing of a patient's death is an inevitable part of many physicians' roles. While research has primarily examined how inpatient clinicians cope with patient loss, little work has explored how primary care clinicians (PCCs) handle patient death in the outpatient setting, and what support resources could help PCCs process loss. OBJECTIVE: To explore PCCs' experiences with the logistics and emotional processing of patient deaths and suggestions for supportive resources. DESIGN: Qualitative study using semi-structured interviews conducted between March and May 2023. PARTICIPANTS: Recruitment emails were sent to 136 PCCs (physicians and nurse practitioners) at three San Francisco academic primary care clinics. Twelve clinicians participated in the study. APPROACH: This study used a template analysis approach. Interview transcripts were analyzed in an iterative fashion to identify themes for how PCCs navigate patient death. RESULTS: Participants (n=12) described outpatient death notification as inconsistent, delayed, and rife with uncertainty regarding subsequent actions. They felt various emotions, notably sadness and guilt, especially with deaths of young, vulnerable patients or those from preventable illnesses. Participants identified strategies for emotional processing and recommended improvements including clear procedural guidance, peer debriefings, and formal acknowledgements of deceased patients. CONCLUSIONS: Interviewing PCCs about their experiences following a patient death revealed key themes in logistical and emotional processing, and clinic resource recommendations to better support PCCs. Given the distinct characteristics of primary care-such as enduring patient relationships, greater isolation in ambulatory settings compared to inpatient environments, and rising burnout rates-enhancing guidance and support for PCCs is crucial to mitigate administrative burdens and grief after patient loss.

Building on Strengths: An Affirmational and Systems-Level Approach to Revisiting Character in Medicine.

ABSTRACT: Although U.S. medical education has continued to place increased emphasis on defining competency standards and ensuring accountability to the public, health care inequities have persisted, several basic health outcomes have worsened, public trust in the health care system has eroded, and moral distress, burnout, and attrition among practicing physicians have escalated. These opposing trends beg the question of how the "good doctor" concept may be strengthened. In this perspective, the authors argue that revisiting the construct of physician character from an affirmational perspective could meaningfully improve medical education's impact on overall health by more holistically conceptualizing what-and who-a good doctor is. The authors introduce positive psychology's framework of character strengths, probe the distinction between character strengths and medical professionalism, and summarize the role of character strengths in promoting physician engagement and well-being in health care work. They contend that a systems-level approach to cultivating character strengths will foster physician moral agency and well-being and, by extension, transformational change in health care. Consistent with best practice in modern character education, the authors propose that institutions mindfully cultivate moral community among all stakeholders (students, faculty, staff, postgraduate trainees, and patients) and that moral community interaction centers on each member's personal aspirations with respect to living a good life, guided by the character strengths framework and informed by patient perspectives.

How Teachers Can Promote Assessment for Learning During All Phases of Learning.

Assessment for Learning (AFL) includes all educational activities designed to improve teaching and learning through gathering, sharing, and acting on information. AFL expands on the concept of formative assessment-which focuses mainly on an in-the-moment assessment activity-to include all processes that facilitate teachers and learners actively seeking and interpreting evidence to guide learning. AFL involves teachers and learners as partners and uses evidence to identify what the learner needs to learn (planning), review where the learner is in their learning (observing), and strategize how to maximize learning (supporting). 1 We provide the following guidelines for teachers to implement AFL within these 3 phases of a learning activity. Teachers should tailor their choice of AFL strategies to suit their relationship with the learner and learning environment context, aiming to support the development of self-regulated learning and metacognitive skills.

Using a boundary crossing lens to understand basic science educator and clinical educator collaboration in instructional design.

PURPOSE: Collaborations between basic science educators (BE) and clinical educators (CE) in medical education are common and necessary to create integrated learning materials. However, few studies describe experiences of or processes used by educators engaged in interdisciplinary teamwork. We use the lens of boundary crossing to explore processes described by BE and CE that support the co-creation of integrated learning materials, and the impact that this work has on them. MATERIALS AND METHODS: We conducted qualitative content analysis on program evaluation data from 27 BE and CE who worked on 12 teams as part of a multi-institutional instructional design project. RESULTS: BE and CE productively engaged in collaboration using boundary crossing mechanisms. These included respecting diverse perspectives and expertise and finding efficient processes for completing shared work that allow BE and CE to build on each other's contributions. BE and CE developed confidence in connecting clinical concepts with causal explanations, and willingness to engage in and support such collaborations at their own institutions. CONCLUSIONS: BE and CE report the use of boundary crossing mechanisms that support collaboration in instructional design. Such practices could be harnessed in future collaborations between BE and CE.

The Problem and Power of Professionalism: A Critical Analysis of Medical Students' and Residents' Perspectives and Experiences of Professionalism.

PURPOSE: Professionalism has historically been viewed as an honorable code to define core values and behaviors of physicians, but there are growing concerns that professionalism serves to control people who do not align with the majority culture of medicine. This study explored how learners, particularly those from historically marginalized groups, view the purpose of professionalism and how they experience professionalism as both an oppressive and valuable force. METHOD: The authors conducted a qualitative study with a critical orientation. In 2021 and 2022, they interviewed fourth-year medical students and senior residents at 3 institutions about their perceptions and experiences of professionalism. After cataloguing participants' stories, the authors combined critical theory with narrative and thematic analysis to identify mechanisms by which professionalism empowered or disempowered individuals or groups based on identities. RESULTS: Forty-nine trainees (31 medical students and 18 senior residents from multiple specialties) participated in interviews; 17 identified as a race/ethnicity underrepresented in medicine and 15 as people of color not underrepresented in medicine. Their stories, especially those of participants underrepresented in medicine, identified professionalism as an oppressive, homogenizing force that sometimes encoded racism through various mechanisms. These mechanisms included conflating differences with unprofessionalism, enforcing double standards of professionalism, and creating institutional policies that regulated appearance or hindered advocacy. Participants described deleterious consequences of professionalism on their learning and mental health. However, participants also described useful aspects of professionalism as a means of advocating for marginalized groups. Additionally, participants described how they reconceived professionalism to include their own identities and values. CONCLUSIONS: Trainees, especially those from historically marginalized groups, experience professionalism as a restrictive, assimilative force while also finding value in and constructive adaptations for professionalism. Understanding both the destructive and empowering aspects of professionalism on individual and institutional levels can help improve the framing of professionalism in medical education.

Learning Together: Co-Learning Among Faculty and Trainees in the Clinical Workplace.

PURPOSE: Studies of workplace learning largely focus on how individuals (clinicians and trainees) learn in the clinical workplace. By focusing on individuals, these studies miss opportunities to explore the co-constructive nature of learning that may result in clinician-educators and trainees learning alongside each other (co-learning). The purpose of this study was to explore whether and how co-learning occurs in the workplace and what value co-learning holds. METHOD: Using a constructivist grounded theory approach, the authors interviewed 34 faculty clinician-educators from 10 specialties at one academic institution to explore whether and how co-learning occurred in their practice and how faculty perceived its value for faculty and for trainees. Interviews were conducted and recorded June to December 2019. Through iterative analysis, the authors refined a conceptual model of co-learning. RESULTS: In a 3-part model of co-learning derived from faculty interviews, faculty and trainees jointly identify learning opportunities during work-related activities and choose learning strategies for learning that lead to interpretation and construction of meaning; these activities produce learning outcomes, such as understanding or insight. The model also accounts for asymmetry in experience and position between faculty and trainees. All faculty participants valued co-learning and elucidated barriers and facilitators to using it in the workplace. How faculty managed asymmetries shaped the nature of the co-learning. When faculty worked to lessen asymmetry between teacher and trainee, co-learning was often collaborative. CONCLUSIONS: The model of co-learning makes explicit the concept of asymmetry in workplace learning and aids consideration of how related dynamics affect the nature of learning. Faculty must be intentional in managing asymmetries to fully leverage learning opportunities afforded by the workplace.

Connecting Biochemistry Knowledge to Patient Care in the Clinical Workplace: Senior Medical Students' Perceptions about Facilitators and Barriers.

Phenomenon: Medical students have difficulties applying knowledge about biomedical mechanisms learned before clerkships to patient care activities. Many studies frame this challenge as a problem of basic science knowledge transfer predominantly influenced by students' individual cognitive processes. Social cognitive theory would support extending this framing to the interplay between the individual's cognition, the environment, and their behaviors. This study investigates senior medical students' experiences of biochemistry knowledge use during workplace learning and examines how their experiences were influenced by interactions with people and other elements of the clinical learning environment. Approach: The authors used a qualitative approach with a constructivist orientation. From September to November 2020 they conducted semi-structured interviews with 11 fourth-year medical students at one institution who had completed the pre-clerkship curriculum, core clinical clerkships, and the United States Medical Licensing Exam Step 1. The authors identified themes using thematic analysis. Findings: Participants reported that they infrequently used or connected to biochemistry knowledge in workplace patient care activities, yet all had examples of such connections that they found valuable to learning. Most participants felt the responsibility for making connections between biochemistry knowledge and activities in the clinical workplace should be shared between themselves and supervisors, but connections were often recognized and acted on only by the student. Connections that participants described prompted their effort to retrieve knowledge or fill a perceived learning gap. Participants identified multiple barriers and facilitators to connecting, including supervisors' behaviors and perceived knowledge, and "patients seen" in clerkships. Participants also reported learning biochemistry during USMLE Step 1 study that did not connect to patient care activities, underscoring a perception of disconnect. Insights: This study identifies specific personal, social, and physical environmental elements that influence students' perceived use of biochemistry during patient care activities. Though these findings may be most significant for biochemistry, they likely extend to other basic science disciplines. Students' self-directed efforts to connect to their biochemistry knowledge could be augmented by increased social support from clinical supervisors, which in turn likely requires faculty development. Opportunities for connection could be enhanced by embedding into the environment instructional strategies or technologies that build on known authentic connections between biochemistry and "patients seen" in clerkships. These efforts could strengthen student learning, improve clinical supervisors' self-efficacy, and better inform curriculum design.

Identities and roles through clinician-educator transitions: A systematic narrative review.

OBJECTIVE: To synthesise the literature exploring nurse and other clinicians' conceptualisations and experiences of roles and identities as they transition to educator positions and to identify facilitators/barriers to, and consequences of, successful transitions. DESIGN: A systematic narrative review of empirical research reporting clinician-educator transitions was conducted from database inception to December 2020. DATA SOURCES: Our search employed ERIC, CINAHL, PsycINFO, Scopus, Ovid MEDLINE® Plus, ERIC (ProQuest), and Sociological Abstracts (ProQuest). We used search terms and synonyms relating to 'identity', 'role' and 'transitions'. REVIEW METHODS: Using the PRISMA protocol for systematic reviews, we reviewed titles and abstracts for inclusion, then used the Critical Appraisal Skills Program tool to evaluate article quality. We extracted evidence from included articles, synthesising data thematically by exploring similarities and differences between studies. RESULTS: We screened 2753 articles. 23 studies (across 25 articles) matched our inclusion criteria, 20 of which included nursing clinician-educator transitions. We identified four themes describing clinician-educator transitions: (i) multiple ways of conceptualising roles and identities, (ii) clinician-educator transitions as complex, emotion-laden processes, (iii) personal, interpersonal, and organisational facilitators and barriers of clinician-educator transitions, and (iv) positive and negative consequences of clinician-educator transitions. CONCLUSION: Our review supports deeper understandings of transition processes that can be used by organisations to better support clinicians as they adapt to their new educator roles and identities.

"Finding My Piece in That Puzzle": A Qualitative Study Exploring How Medical Students at Four U.S. Schools Envision Their Future Professional Identity in Relation to Health Systems.

PURPOSE: Health systems science (HSS) curricula equip future physicians to improve patient, population, and health systems outcomes (i.e., to become "systems citizens"), but the degree to which medical students internalize this conception of the physician role remains unclear. This study aimed to explore how students envision their future professional identity in relation to the system and identify experiences relevant to this aspect of identity formation. METHOD: Between December 2018 and September 2019, authors interviewed 48 students at 4 U.S. medical schools with HSS curricula. Semistructured interviews were audiorecorded, transcribed, and analyzed iteratively using inductive thematic analysis. Interview questions explored how students understood the health system, systems-related activities they envisioned as future physicians, and experiences and considerations shaping their perspectives. RESULTS: Most students anticipated enacting one or more systems-related roles as a future physician, categorized as "bottom-up" efforts enacted at a patient or community level (humanist, connector, steward) or "top-down" efforts enacted at a system or policy level (system improver, system scholar, policy advocate). Corresponding activities included attending to social determinants of health or serving medically underserved populations, connecting patients with team members to address systems-related barriers, stewarding health care resources, conducting quality improvement projects, researching/teaching systems topics, and advocating for policy change. Students attributed systems-related aspirations to experiences beyond HSS curricula (e.g., low-income background; work or volunteer experience; undergraduate studies; exposure to systems challenges affecting patients; supportive classmates, faculty, and institutional culture). Students also described future-oriented considerations promoting or undermining identification with systems-related roles (responsibility, affinity, ability, efficacy, priority, reality, consequences). CONCLUSIONS: This study illuminates systems-related roles medical students at 4 schools with HSS curricula envisioned as part of their future physician identity and highlights past/present experiences and future-oriented considerations shaping identification with such roles. These findings inform practical strategies to support professional identity formation inclusive of systems engagement.

Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships.

PURPOSE: To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation. METHOD: This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system. RESULTS: Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning. CONCLUSIONS: Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.

How do internal medicine subspecialty societies support clinician-educator careers? A qualitative exploratory study.

BACKGROUND: Internal Medicine (IM) subspecialty professional societies can provide valuable community, recognition, resources, and leadership opportunities that promote career success. Historically, this support focused on clinical and research dimensions of academic careers, but educational dimensions have gained more attention recently. This study explores how IM subspecialty professional societies support their clinician-educator members. METHODS: Using a qualitative study with two phases, the authors collected information from each IM subspecialty society's website about support for medical education. Using information from the first phase, we developed an interview guide for subspecialty society leaders. We used inductive thematic analysis to analyze interview transcripts. RESULTS: Website analysis identified various mechanisms used by several IM subspecialty societies to promote medical education. These included websites focused on medical education, dedicated medical education poster/abstract sessions at annual meetings, and strategies to promote networking among clinician-educators. Interviews with eight subspecialty society leaders about the professional societies' roles with respect to medical education yielded four main themes: [1] varying conceptions of "medical education" in relation to the society [2] strategies to advance medical education at the society level [3] barriers to recognizing medical education [4] benefits of clinician-educators to the societies. Integrating these themes, we describe recommended strategies for professional societies to better serve clinician-educators. CONCLUSIONS: We explore how IM subspecialty societies attend to a growing constituency of clinician-educators, with increasing recognition and support of the career path but persistent barriers to its formalization. These conversations shed light on opportunities for professional subspecialty societies to better serve the needs of their clinician-educator members while also enabling these members to make positive contributions in return.

Bridging medical education goals and health system outcomes: An instrumental case study of pre-clerkship students' improvement projects.

INTRODUCTION: Many medical schools engage students in health system improvement (HSI) efforts. Evaluation of these efforts often focuses on students' learning outcomes and rarely considers the impact on health systems, despite the significant commitment health systems make to these efforts. Our study identified and evaluated system-level outcomes of pre-clerkship medical students' engagement in HSI efforts. METHODS: We used an instrumental case study approach to examine the effects of pre-clerkship medical students' engagement in HSI projects as part of a 15-month experiential curriculum. We extracted data from 53 project summaries and posters completed during the 2017-18 academic year and follow-up survey data collected in May 2019 from physician coaches and health system professionals who mentored students, contributed to these projects, and worked in the clinical microsystems where the projects occurred. RESULTS: We identified three categories and ten indicators of health system outcomes relevant to medical student engagement in HSI. Using these indicators, our evaluation found multiple benefits to the microsystems in which projects occurred. These included achievement of project aims, perceived immediate and sustained project impact on the health system, and development and implementation of projects with aims that aligned with national and health system priorities. CONCLUSION: Evaluation of HSI curricula needs to include effects on health systems so that program design can optimize the experience for all involved. Our study offers a framework others can use to evaluate system-level effects of project-based HSI curricula and shows several ways in which students' engagement can add value to health systems.

Exploring the role of power during debriefing of interprofessional simulations.

Interprofessional simulation aims to improve teamwork and patient care by bringing participants from multiple professions together to practice simulated patient care scenarios. Yet, power dynamics may influence interprofessional learning during simulation, which typically occurs during the debriefing. This issue has received limited attention to date but may explain why communication breakdowns and conflicts among healthcare teams persist despite widespread adoption of interprofessional simulation. This study explores the role of power during interprofessional simulation debriefings. We collected data through observations of seven interprofessional simulation sessions and debriefings, four focus groups with simulation participants, and four interviews with simulation facilitators. We identified ways in which power dynamics influenced discussions during debriefing and sometimes limited participants' willingness to share feedback and speak up. We also found that issues related to power that arose during interprofessional simulations often went unacknowledged during the debriefing, leaving healthcare professionals unprepared to navigate power discrepancies with other members of healthcare teams in practice. Given that the goal of interprofessional simulation is to allow professionals to learn together about each other, explicitly addressing power in debriefing after interprofessional simulation may enhance learning.

Are You Thinking What I'm Thinking? Exploring Response Process Validity Evidence for a Workplace-based Assessment for Operative Feedback.

OBJECTIVE: Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assessment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery. DESIGN: Semi-structured interviews explored the reasonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measuring Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment questions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualitative content analysis to generate themes. SETTING: Single tertiary academic medical center. PARTICIPANTS: Eight residents and 13 faculty within the Department of Otolaryngology-Head and Neck Surgery participating in a 6-month pilot of SIMPL. RESULTS: We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty. CONCLUSIONS: Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a continued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment.