Improving patient-centered care for HIV and mental illness: exploring hospital and community integration through education.

Current models of care delivery are failing patients with complexity, like those living with HIV, mental illness and other psychosocial challenges. These patients often require resource-intensive personalized care across hospital and community settings, but available supports can be fragmented and challenging to access and navigate. To improve this, the authors created a program to enhance integrated, trauma-informed care through an innovative educational role for a HIV community caseworker embedded in an academic HIV Psychiatry clinic, called the Mental Health Clinical Fellowship. Through qualitative interviews with 21 participants (patients, physicians, clinicians and Mental Health Clinical Fellows) from October 2020-March 2023, the authors explore how implementation of this program affects patient experiences and satisfaction with care. Patients described their care experiences as less stigmatizing, more accessible, holistic and coordinated. They often attributed this to the integration between fellow and psychiatrist, and specifically the accessible stance of community organizations embedded within a hospital, which helped build trust. Interchangeable and integrated support by caseworker and psychiatrist improved patient engagement in psychiatric management and patient satisfaction with their care. Cross-context and cross-disciplinary care provision that includes providers from community and hospital working directly together to deliver care can improve care for patients with significant complexity.

Exploring adaptive expertise in residency: the (missed) opportunity of uncertainty.

Preparing novice physicians for an unknown clinical future in healthcare is challenging. This is especially true for emergency departments (EDs) where the framework of adaptive expertise has gained traction. When medical graduates start residency in the ED, they must be supported in becoming adaptive experts. However, little is known about how residents can be supported in developing this adaptive expertise. This was a cognitive ethnographic study conducted at two Danish EDs. The data comprised 80 h of observations of 27 residents treating 32 geriatric patients. The purpose of this cognitive ethnographic study was to describe contextual factors that mediate how residents engage in adaptive practices when treating geriatric patients in the ED. Results showed that all residents fluidly engaged in both adaptive and routine practices, but they were challenged when engaging in adaptive practices in the face of uncertainty. Uncertainty was often observed when residents' workflows were disrupted. Furthermore, results highlighted how residents construed professional identity and how this affected their ability to shift between routine and adaptive practices. Residents reported that they thought that they were expected to perform on par with their more experienced physician colleagues. This negatively impacted their ability to tolerate uncertainty and hindered the performance of adaptive practices. Thus, aligning clinical uncertainty with the premises of clinical work, is imperative for residents to develop adaptive expertise.

Preparing future physicians for complexity: a post-graduate elective in HIV psychiatry.

BACKGROUND: Patients with complex care needs have multiple concurrent conditions (medical, psychiatric, social vulnerability or functional impairment), interfering with achieving desired health outcomes. Their care often requires coordination and integration of services across hospital and community settings. Physicians feel ill-equipped and unsupported to navigate uncertainty and ambiguity caused by multiple problems. A HIV Psychiatry resident elective was designed to support acquisition of integrated competencies to navigate uncertainty and disjointed systems of care - necessary for complex patient care. METHODS: Through qualitative thematic analysis of pre- and post-interviews with 12 participants - residents and clinic staff - from December 2019 to September 2022, we explored experiences of this elective. RESULTS: This educational experience helped trainees expand their understanding of what makes patients complex. Teachers and trainees emphasize the importance of an approach to "not knowing" and utilizing integrative competencies for navigating uncertainty. Through perspective exchange and collaboration, trainees showed evidence of adaptive expertise: the ability to improvise while drawing on past knowledge. CONCLUSIONS: Postgraduate training experiences should be designed to facilitate skills for caring for complex patients. These skills help residents fill in practice gaps, improvise when standardization fails, and develop adaptive expertise. Going forward, findings will be used to inform this ongoing elective.

Being and Becoming an Evidence-Based Practitioner: Occupational Therapists' Journey Toward Expertise.

IMPORTANCE: Despite mounting evidence for the management of various health care conditions, the uptake of scientific evidence in occupational therapy is often suboptimal. Although positive attitudes and self-efficacy are key to evidence-based practices (EBPs), how one becomes an evidence-based practitioner, and how expertise in EBP manifests in practice, remains unclear. OBJECTIVE: To describe how expert evidence-based practitioners conceptualize and enact their expertise in stroke rehabilitation. DESIGN: Qualitative interpretive descriptive study using in-depth semistructured interviews with eight occupational therapists identified as expert evidence-based practitioners in a previous study. Transcripts were analyzed using an inductive thematic content analysis, and emergent themes were identified. SETTINGS: Rehabilitation settings across Canada. RESULTS: Six overarching themes emerged: (1) relying on personal attributes to engage in practice improvement, (2) acting on factors that motivate and trigger EBP, (3) achieving better outcomes because of engagement in EBP, (4) using an adaptive decision-making process, (5) participating in professional activities that contribute to practice improvement, and (6) working in a practice area with a large body of evidence. CONCLUSIONS AND RELEVANCE: Expertise in stroke rehabilitation EBP appears to be a function of several personal attributes and habits of mind in addition to being influenced by a commitment to client-centered practice. Expertise requires a combination of deliberate effort and motivation to improve client outcomes, always in a context conducive to reflection, adaptation, and openness to innovation. Experts who model these traits and articulate the processes used to develop their expertise can be viewed as promising educational and continuing professional development resources. What This Article Adds: Expert evidence-based occupational therapists develop their expertise in this domain through commitment to client-centered practice, efforts toward honing this aspect of practice, and a willingness to innovate and adapt to challenging situations.

"It's Complicated": Using Education to Bridge Essential Care Between Hospital and Community for Complex Patients with HIV.

OBJECTIVE: For the most complex patients, like those with HIV and mental illness, integrated care occurs across diverse community and hospital contexts. There is a need for cross-discipline and cross-context educational opportunities for diverse providers to learn integrated care skillsets in real practice settings. The authors developed a Continuing Professional Development (CPD) experience for frontline case workers to be embedded in a hospital-based HIV psychiatry clinic that aims to enhance collaborative skills across hospital and community settings, called the Mental Health Clinical Fellowship. METHODS: Through qualitative pre- and post-interviews with 16 participants from October 2020 to October 2021, the authors explored patient, physician, clinician, resident learner, and the Mental Health Clinical Fellow's learning experiences and the impact on patient care. RESULTS: Preliminary findings elucidate some common challenges providers experience in providing care to this complex population, including facing uncertainty of diagnosis and management, and not having enough time or resources to navigate this uncertainty. The opportunity to work and learn across disciplines through the fellowship reduced challenges, and also facilitated adaptive expertise development. CONCLUSIONS: Cross-context and cross-discipline education opportunities facilitate perspective-sharing and enhanced ability to develop adaptive expertise in caring for complex populations. There is also promise for improving care and decreasing fragmentation because of the educational experience.

The imperative for (and opportunities of) research on adaptive expertise in health professions education.

In this editorial, three Advances in Health Sciences Education editors argue for the importance and impact of adaptive expertise on the future of health professions education and work. They present a sample of the broad range of theory-informed research currently contributing to understanding and applying adaptive expertise in health professions education. They reflect on the unique opportunities that interdisciplinarity offers this endeavour. Finally they offer potential ways forward for continued efforts to advance collective understanding of education, expert development and health professions practice.

Balancing closure and discovery: adaptive expertise in the workplace.

Residents must develop knowledge, skills, and attitudes to handle a rapidly developing clinical environment. To address this need, adaptive expertise has been suggested as an important framework for health professions education. However, research has yet to explore the relationship between workplace learning and adaptive expertise. This study sought to investigate how clinical supervision might support the development of adaptive expertise. The present study used a focused ethnography in two emergency departments. We observed 75 supervising situations with the 27 residents resulting in 116 pages of field notes. The majority of supervision was provided by senior physicians, but also included other healthcare professionals. We found that supervision could serve two purposes: closure and discovery. Supervision aimed at discovery included practices that reflected instructional approaches said to promote adaptive expertise, such as productive struggle. Supervision aimed at closure-included practices with instructional approaches deemed important for efficient and safe patient care, such as verifying information. Our results suggest that supervision is a shared practice and responsibility. We argue that setting and aligning expectations before engaging in supervision is important. Furthermore, results demonstrated that supervision was a dynamic process, shifting between both orientations, and that supervision aimed at discovery could be an an appropriate mode of supervision, even in the most demanding clinical situations.

Combining adaptive expertise and (critically) reflective practice to support the development of knowledge, skill, and society.

Adaptive expertise (AE) and reflective practice (RP), two influential and resonant theories of professional expertise and practice in their own right, may further benefit health professions education if carefully combined. The current societal and systemic context is primed for both AE and RP. Both bodies of work position practitioners as agentive, learning continually and thoughtfully throughout their careers, particularly in order to manage unprecedented situations well. Similar on the surface, the roots and practices of AE and RP diverge at key junctures and we will focus on RP's movement toward critically reflective practice. The roots of AE and RP, and how they relate to or diverge from present-day applications matter because in health professions education, as in all education, paradigmatic mixing should be undertaken purposefully. This paper will explore the need for AE and RP, their shared commitments, distinctive histories, pedagogical possibilities both individually and combined, and next steps for maximizing their potential to positively impact the field. We argue that this exploration is urgently needed because both AE and RP hold much promise for improving health care and yet employing them optimally-whether alone or together-requires understanding and intent. We build an interprofessional education case situated in long-term care, throughout the paper, to demonstrate the potential that AE and RP might offer to health professions education individually and combined. This exploration comes just in time. Within the realities of uncertain practice emphasized by the pandemic, practitioners were also called to act in response to complex and urgent social movements. A combined AE and RP approach, with focus on critically reflective practice in particular, would potentially prepare professionals to respond effectively, compassionately, and equitably to future health and social crises and challenges.

Applying activity theory to undergraduate medical curriculum reform: Lessons in contradictions from multiple stakeholders' perspectives.

PURPOSE: Medical school curricula require regular updating. We adopted an activity theory lens to conduct a holistic, multiple stakeholder-informed analysis of curricular reform, aiming to understand how the social relations between groups contribute to unanticipated tensions and outcomes. METHODS: A research assistant conducted semi-structured interviews with purposively sampled (N = 19) administrative staff, faculty course leads, faculty tutors, curriculum developers, change leaders and student leaders. The team applied a framework analysis to guide within and between stakeholder comparisons. RESULTS: Participants reported unique (N = 21) and cross-cutting (N = 17) contradictions underscoring emerging drivers of current and potential change. Unique contradictions raised by 1-2 groups represented seeds of change that had the potential to spread across all groups. By contrast, two general types of cross-cutting contradictions arose when one group had a dominant, confirming voice or two or more groups had contrasting perspectives. CONCLUSIONS: While finding contradictions was expected, our analysis profiled their nature and some of the specific tensions they raised across and within stakeholder groups. The activity theory lens provided an accessible way to unravel curricular reform into manageable units of analysis. Systematically identifying contradictions arising from curricular reform will help stakeholders collaborate with a shared purpose toward positive, sustained change.

Later is too late: Exploring student experiences of diversity and inclusion in medical school orientation.

BACKGROUND: While there is increasing effort among medical schools to recruit diverse students, there is a paucity of research into the unique experiences of these students during their transition to medicine. This study explored how experiences during medical school orientation influence students' transition into the medical profession. METHODS: Semi-structured interviews were conducted (April-August 2019) with 16 first-year Canadian medical students. We applied descriptive thematic analysis using a constant comparative approach. Verbatim transcripts were coded and analyzed to elucidate themes. RESULTS: Participants highlighted the importance of social orientation during their transition into medical school and noted experiencing complex social pressures during this time. They shared how incoming students were introduced to the dominant medical professional identity during orientation. Participants noted tensions during this period, many of which revolved around the dominant identity and their past, present and future selves. CONCLUSIONS: Longstanding issues of diversity and inclusion in medicine manifest from day one of medical school. While orientation may be intended as a transition period to welcome students into the profession, it is a crucial period for medical schools to intentionally establish a commitment to an inclusive culture. Waiting to do so after identity formation has already begun is a missed opportunity.

How supervision and educational supports impact medical students' preparation for future learning of endotracheal intubation skills: a non-inferiority experimental trial.

BACKGROUND: Professional education cannot keep pace with the rapid advancements of knowledge in today's society. But it can develop professionals who can. 'Preparation for future learning' (PFL) has been conceptualized as a form of transfer whereby learners use their previous knowledge to learn about and adaptively solve new problems. Improved PFL outcomes have been linked to instructional approaches targeting learning mechanisms similar to those associated with successful self-regulated learning (SRL). We expected training that includes evidence-based SRL-supports would be non-inferior to training with direct supervision using the outcomes of a 'near transfer' test, and a PFL assessment of simulated endotracheal intubation skills. METHOD: This study took place at the University of Toronto from October 2014 to August 2015. We randomized medical students and residents (n = 54) into three groups: Unsupervised, Supported; Supervised, Supported; and Unsupervised, Unsupported. Two raters scored participants' test performances using a Global Rating Scale with strong validity evidence. We analyzed participants' near transfer and PFL outcomes using two separate mixed effects ANCOVAs. RESULTS: For the Unsupervised, Supported group versus the Supervised, Supported group, we found that the difference in mean scores was 0.20, with a 95% Confidence Interval (CI) of - 0.17 to 0.57, on the near transfer test, and was 0.09, with a 95% CI of - 0.28 to 0.46, on the PFL assessment. Neither mean score nor their 95% CIs exceeded the non-inferiority margin of 0.60 units. Compared to the two Supported groups, the Unsupervised, Unsupported group was non-inferior on the near transfer test (differences in mean scores were 0.02 and - 0.22). On the PFL assessment, however, the differences in mean scores were 0.38 and 0.29, and both 95% CIs crossed the non-inferiority margin. CONCLUSIONS: Training with SRL-supports was non-inferior to training with a supervisor. Both interventions appeared to impact PFL assessment outcomes positively, yet inconclusively when compared to the Unsupervised and Unsupported group, By contrast, the Unsupervised, Supported group did not score well on the near transfer test. Based on the observed sensitivity of the PFL assessment, we recommend researchers continue to study how such assessments may measure learners' SRL outcomes  during structured learning experiences.

Resident learning trajectories in the workplace: A self-regulated learning analysis.

CONTEXT: Research in workplace learning has emphasised trainees' active role in their education. By focusing on how trainees fine-tune their strategic learning, theories of self-regulated learning (SRL) offer a unique lens to study workplace learning. To date, studies of SRL in the workplace tend to focus on listing the factors affecting learning, rather than on the specific mechanisms trainees use to regulate their goal-directed activities. To inform the design of workplace learning interventions that better support SRL, we asked: How do residents navigate their exposure to and experience performing invasive procedures in intensive care units? METHODS: In two academic hospitals, we conducted post-call debriefs with residents coming off shift and later sought their elaborated perspectives via semi-structured interviews. We used a constant comparative methodology to analyse the data, to iteratively refine data collection, and to inform abductive coding of the data, using SRL principles as sensitising concepts. RESULTS: We completed 29 debriefs and nine interviews with 24 trainees. Participants described specific mechanisms: identifying, creating, avoiding, missing and competing for opportunities to perform invasive procedures. While using these mechanisms to engage with procedures (or not), participants reported: distinguishing trajectories (i.e. becoming attuned to task-relevant factors), navigating trajectories (i.e. creating and interacting with opportunities to perform procedures), and co-constructing trajectories with their peers, supervisors and interprofessional team members. CONCLUSIONS: We identified specific SRL mechanisms trainees used to distinguish and navigate possible learning trajectories. We also confirmed previous findings, including that trainees become attuned to interactions between personal, behavioural and environmental factors (SRL theory), and that their resulting learning behaviours are constrained and guided by interactions with peers, supervisors and colleagues (workplace learning theory). Making learning trajectories explicit for clinician teachers may help them support trainees in prioritising certain trajectories, in progressing along each trajectory, and in co-constructing their plans for navigating them.

Critically reflective practice and its sources: A qualitative exploration.

CONTEXT: Critical reflection may improve health professionals' performance of the social roles of care (eg collaboration) in indeterminate zones of practice that are ambiguous, unique, unstable or value-conflicted. Research must explore critical reflection in practice and how it is developed. In this study, we explored what critical reflection consisted of in a context known for indeterminacy, and to what sources participants attributed their critically reflective insights and approaches. METHODS: The study context was the interface between health care and education for children with chronic conditions or disabilities necessitating health-related recommendations and supports (eg accommodations or equipment) at school. We conducted a secondary analysis of 42 interview transcripts from an institutional ethnographic study involving health professionals, school-based educators and parents of children with chronic conditions or disabilities. We coded all transcripts for instances of critical reflection, moments that seemed to lack but could benefit from critical reflection, and participant-attributed sources of critically reflective insights. RESULTS: Critically reflective practice involved getting to know the other, valuing and leveraging different forms and sources of knowledge, identifying and communicating workarounds (ie strategies to circumvent imperfect systems), seeing inequities, and advocating as collaborators, not adversaries. Participants invariably attributed critically reflective insights to personal experiences such as former careers or close personal relationships. CONCLUSIONS: This study shows that personal experiences and connections inspire critically reflective views, and that being critically reflective is not a binary trait possessed (or not) by individuals. It is learnable through personally meaningful experiences. Health professions education could aim to preserve philosophical space for personal experience as a source of learning and integrate evidence-informed approaches to foster critically reflective practice.

Exploring why we learn from productive failure: insights from the cognitive and learning sciences.

Advances in Health Sciences Education (AHSE) has been at the forefront of the cognitive wave in health professions education for the past 25 years. One example is research on productive failure, a teaching strategy that asks learners to attempt to generate solutions to difficult problems before receiving instruction. This study compared the effectiveness of productive failure with indirect failure to further characterize the underpinning cognitive mechanisms of productive failure. Year one pharmacy students (N = 42) were randomly assigned to a productive failure or an indirect failure learning condition. The problem of estimating renal function based on serum creatinine was described to participants in the productive failure learning condition, who were then asked to generate a solution. Participants in the indirect failure condition learned about the same problem and were given incorrect solutions that other students had created, as well as the Cockcroft-Gault formula, and asked to compare and contrast the equations. Immediately thereafter all participants completed a series of tests designed to assess acquisition, application, and preparation for future learning (PFL). The tests were repeated after a 1-week delay. Participants in the productive failure condition outperformed those in the indirect failure condition, both on the immediate PFL assessment, and after a 1-week delay. These results emphasize the crucial role of generation in learning. When preparing novice students to learn new knowledge in the future, generating solutions to problems prior to instruction may be more effective than simply learning about someone else's mistakes. Struggle and failure are most productive when experienced personally by a learner because it requires the learner to engage in generation, which deepens conceptual understanding.

Developing a Framework of Integrated Competencies for Adaptive Expertise in Integrated Physical and Mental Health Care.

Phenomenon: Despite the emergence of the integrated care (IC) model, IC is variably taught and is challenged by current siloed competency domains. This study aimed to define IC competencies spanning multiple competency domains. Approach: Iterative facilitated discussions were conducted at a half-day education retreat with 25 key informants including clinician educators and education scientists. Seven one-on-one semistructured interviews were subsequently conducted with different interprofessional providers in IC settings within a Canadian context. Data collection grounded in patient cases with a physical illness and concurrent mental illness (medical psychiatry) were used to elicit identification of complex patient needs and the key medical psychiatry knowledge and skills required to address these needs. A thematic analysis of transcripts was performed using constant comparison to iteratively identify themes. Findings: Participants described 4 broad competency domains necessary for expertise in IC: (a) extensive integrated knowledge of biopsychosocial aspects of disease, systems of care, and social determinants of care; (b) skills to establish a longitudinal alliance with the patient and functional relationships with colleagues; (c) constructing a comprehensive understanding of individual patients' complex needs and how these can be met within their health and social systems; and (d) the ability to effectively meet the patient's needs using IC models. These 4 domains were linked by an overarching philosophy of care encompassing key enabling attitudes such as proactively pursuing depth to understand patient and system complexity while maintaining a patient-centered approach. Insights: The study addresses how development of IC expertise can be fostered by integration of individual IC competency domains. The findings align with previous research suggesting that competencies from existing frameworks are being enacted jointly in expert capabilities to meet the complex needs of patients, in this case with comorbid physical and mental health concerns.

Examining grounded theory through the lens of rationalist epistemology.

The objective of scientific, or more broadly, academic knowledge is to provide an understanding of the social and natural world that lies beyond common sense and everyday thinking. Academics use an array of techniques, methods and conceptual apparatuses to achieve this goal. The question we explore in this essay is the following: Does the grounded theory approach, in the constructivist version developed by Kathy Charmaz, provide the necessary methodological tools for the creation of knowledge and theories beyond everyday thinking? To conduct our analysis, we have drawn on the rationalist epistemology originally developed by Gaston Bachelard and taken up a few decades later by Pierre Bourdieu and colleagues to look at the epistemological foundation of the CGT methods as defined by Charmaz. We focussed on two distinctive epistemological features characterising constructivist grounded theory (CGT): the use of inductive reasoning to generate interpretative theory; and the primacy of subjectivity over objectivity as the preferred path to knowledge making. While the usefulness of CGT for conducting qualitative research and understanding the perspective of social actors has been acknowledged by scholars in health professions education research and other research areas, the inductivist logic on which it draws raises questions concerning the nature of the knowledge yielded by this approach. As we argue in this article, it is still unclear in what way the interpretative theory generated by CGT is not a duplication of everyday thinking expressed through meta-narratives. It is also unclear how the understanding of social phenomena can be refined if the use of inductive procedures logically implies the creation of a new theory each time a study is conducted. We engage with these questions to broaden the epistemological conversation within the health professions education research community. It is our hope that scholars in the field will engage in this epistemological conversation and advance it in new directions.

Productive failure as an instructional approach to promote future learning.

Productive failure is an instructional approach that requires learners to struggle as they attempt to generate solutions to problems before, rather than after, receiving direct instruction on a targeted concept. Studies demonstrate that productive failure prepares students for later learning of new, related knowledge. Our study explored the effectiveness of productive failure as an instructional intervention in health professions education with respect to (a) acquisition and application of a novel concept, and (b) learners' preparation for future learning of new, related content. Forty year-one students enrolled in the Doctor of Pharmacy program at the University of Toronto were randomly assigned to a productive failure (i.e. attempt to generate solutions before receiving instruction) or direct instruction only learning condition. After a practice phase, participants completed a series of tests designed to measure knowledge acquisition, knowledge application, and preparation for future learning (new learning is required for successful problem solving). As expected, no difference in performance was seen between participants on the acquisition and application tests. However, participants in the productive failure condition outperformed those in the direct instruction condition on the preparation for future learning test. These results emphasize the role of struggle in learning and support the theory that engaging students in solving problems that are beyond their abilities can be a productive exercise in failure. The results suggest that productive failure assists learners in acquiring the conceptual knowledge needed to facilitate learning in the future.

Using simulation to explore medical students' understanding of integrated care within geriatrics.

BACKGROUND: Given the increasing evidence and expansion of integrated care (IC) in healthcare, new IC curricula introduced early in undergraduate medical education (UME) are needed. Building on a pilot IC simulation called "Getting to Know Patients' System of Care" (GPS-Care), we aimed to explore students' understanding of patients' complex physical and mental health needs, and to increase our understanding of how students learned in this simulation. METHODS: 177 of 259 first-year medical students participated in GPS-Care at the University of Toronto. Students role-played an elderly patient or caregiver within 5 simulated healthcare professional appointments. Students completed written reflections and 7 students participated in one-on-one interviews. A thematic analysis of the reflections and transcripts was conducted and descriptive data was generated for questionnaires. RESULTS: Data saturation was reached at 43 reflections and 7 transcripts and the following themes emerged: a) students reflected on patients' complex care experiences, b) students reflected on of the healthcare system needs care, c) students increased understanding of IC, and d) students desire to improve the care of IC patients within the healthcare system. CONCLUSIONS: In addition to confirming previous pilot study themes, the results from this study identified the role of productive struggle to provide students with a deeper understanding of patients' IC care needs. Moreover, GPS-Care resulted in a transformative learning experience resulting in new insights into the importance of IC early in UME training.

Resident and attending perceptions of direct observation in internal medicine: a qualitative study.

OBJECTIVES: Direct observation is the foundation of assessment and learning in competency-based medical education (CBME). Despite its importance, there is significant uncertainty about how to effectively implement frequent and high-quality direct observation. This is particularly true in specialties where observation of non-procedural skills is highly valued and presents unique challenges. It is therefore important to understand perceptions of direct observation to ensure successful acceptance and implementation. In this study, we explored perceptions of direct observation in internal medicine. METHODS: We interviewed internal medicine attending physicians (n = 9) and residents (n = 8) at the University of Toronto, purposively sampled for diversity. Using a constructivist grounded theory approach, constant comparative analysis was performed to develop a framework to understand perceptions of direct observation on the clinical teaching units. RESULTS: Participants articulated a narrow perception of what constitutes direct observation, in contrast to their own descriptions of skills that were observed. This resulted in the perception that certain valuable skills that participants felt were routinely observed were nonetheless not 'directly observable', such as clinical reasoning, observed through case presentations and patient care discussions. Differentiating direct observation from informal observation led to overestimation of the time and resource requirements needed to enhance direct observation, which contributed to scepticism and lack of engagement related to CBME implementation. CONCLUSIONS: In an internal medicine training programme, perceptions of what constitutes direct observation can lead to under-recognition and hinder acceptance in workplace-based assessment and learning. Our results suggest a reframing of 'direct observation' for residents and attending physicians, by explicitly identifying desired skills in non-procedurally-based specialties. These findings may help CBME-based training programmes improve the process of direct observation, leading to enhanced assessment and learning.

How can systems engineering inform the methods of programme evaluation in health professions education?

CONTEXT: We evaluate programmes in health professions education (HPE) to determine their effectiveness and value. Programme evaluation has evolved from use of reductionist frameworks to those addressing the complex interactions between programme factors. Researchers in HPE have recently suggested a 'holistic programme evaluation' aiming to better describe and understand the implications of 'emergent processes and outcomes'. FRAMEWORK: We propose a programme evaluation framework informed by principles and tools from systems engineering. Systems engineers conceptualise complexity and emergent elements in unique ways that may complement and extend contemporary programme evaluations in HPE. We demonstrate how the abstract decomposition space (ADS), an engineering knowledge elicitation tool, provides the foundation for a systems engineering informed programme evaluation designed to capture both planned and emergent programme elements. METHODS: We translate the ADS tool to use education-oriented language, and describe how evaluators can use it to create a programme-specific ADS through iterative refinement. We provide a conceptualisation of emergent elements and an equation that evaluators can use to identify the emergent elements in their programme. Using our framework, evaluators can analyse programmes not as isolated units with planned processes and planned outcomes, but as unfolding, complex interactive systems that will exhibit emergent processes and emergent outcomes. Subsequent analysis of these emergent elements will inform the evaluator as they seek to optimise and improve the programme. CONCLUSION: Our proposed systems engineering informed programme evaluation framework provides principles and tools for analysing the implications of planned and emergent elements, as well as their potential interactions. We acknowledge that our framework is preliminary and will require application and constant refinement. We suggest that our framework will also advance our understanding of the construct of 'emergence' in HPE research.