Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program.

BACKGROUND: Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design. METHODS: We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program's use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs. RESULTS: Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a "numbers game" residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning. CONCLUSION: Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities.

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

"The Most Crushing Thing": Understanding Resident Assessment Burden in a Competency-Based Curriculum.

Background: Competency-based medical education (CBME) was expected to increase the workload of assessment for graduate training programs to support the development of competence. Learning conditions were anticipated to improve through the provision of tailored learning experiences and more frequent, low-stakes assessments. Canada has adopted an approach to CBME called Competence by Design (CBD). However, in the process of implementation, learner anxiety and assessment burden have increased unexpectedly. To mitigate this unintended consequence, we need a stronger understanding of how resident assessment burdens emerge and function. Objective: This study investigates contextual factors leading to assessment burden on residents within the framework of CBD. Methods: Residents were interviewed about their experiences of assessment using constructivist grounded theory. Participants (n=21) were a purposive sample from operative and perioperative training programs, recruited from 6 Canadian medical schools between 2019 and 2020. Self-determination theory was used as a sensitizing concept to categorize findings on types of assessment burden. Results: Nine assessment burdens were identified and organized by threats to psychological needs for autonomy, relatedness, and competence. Burdens included: missed opportunities for self-regulated learning, lack of situational control, comparative assessment, lack of trust, constraints on time and resources, disconnects between teachers and learners, lack of clarity, unrealistic expectations, and limitations of assessment forms for providing meaningful feedback. Conclusions: This study contributes a contextual understanding of how assessment burdens emerged as unmet psychological needs for autonomy, relatedness, and competence, with unintended consequences for learner well-being and intrinsic motivation.

Scut to Scholarship: Can Operative Notes be Educationally Useful?

OBJECTIVE: Efforts to implement competency-based medical education require new sources of workplace-based evidence of growth in learning. We used qualitative analysis of operative notes to explore procedural variation in a simple surgical procedure. DESIGN: We used a grounded theory-based mixed methods approach to depict intersurgeon procedural variation. Our grounded theory approach to analysis included follow up interviews with surgeons and residents to probe their understandings of the reasons for variation in the dictated notes and the current and potential utility of operative notes as a reliable source of data for learning and assessment. SETTING: Publicly funded tertiary care otolaryngology-head & neck surgery residency program in Ontario, Canada PARTICIPANTS: Using maximum variability sampling, all surgeons performing tonsillectomy in the department (n = 6) contributed operative notes from 65 tonsillectomies, 5 intraoperative observations, and 4 semi-structured interviews. An additional 3 residents from various levels of training contributed semistructured interviews. RESULTS: Intersurgeon procedural variations persist even in simple surgical procedures such as tonsillectomy. Operative notes appear to capture procedural variations in a limited way. Surgeons and resident make informal educational use of the clerical work of writing and assessing operative notes, but optimization will be required to shift such hidden work into the formal educational domain. CONCLUSIONS: The implementation of competency-based medical education requires surgical educators to both eliminate low-yield tasks for learning and to find new opportunities for multiple low-stakes assessment. Analysis of operative notes may become a high-yield strategy for learning and assessment if residents and surgeons are coached to use operative notes more reliably and efficiently.

Trainee-environment interactions that stimulate motivation: A rich pictures study.

CONTEXT: Staying motivated when working and learning in complex workplaces can be challenging. When complex environments exceed trainees' aptitude, this may reduce feelings of competence, which can hamper motivation. Motivation theories explain how intrapersonal and interpersonal aspects influence motivation. Clinical environments include additional aspects that may not fit into these theories. We used a systems approach to explore how the clinical environment influences trainees' motivation and how they are intertwined. METHODS: We employed the rich pictures drawing method as a visual tool to capture the complexities of the clinical environment. A total of 15 trainees drew a rich picture representing a motivating situation in the workplace and were interviewed afterwards. Data collection and analysis were performed iteratively, following a constructivist grounded theory approach, using open, focused and selective coding strategies as well as memo writing. Both drawings and the interviews were used to reach our results. RESULTS: Trainees drew situations pertaining to tasks they enjoyed doing and that mattered for their learning or patient care. Four dimensions of the environment were identified that supported trainees' motivation. First, social interactions, including interpersonal relationships, supported motivation through close collaboration between health care professionals and trainees. Second, organisational features, including processes and procedures, supported motivation when learning opportunities were provided or trainees were able to influence their work schedule. Third, technical possibilities, including tools and artefacts, supported motivation when tools were used to provide trainees with feedback or trainees used specific instruments in their training. Finally, physical space supported motivation when the actual setting improved the atmosphere or trainees were able to modify the environment to help them focus. CONCLUSIONS: Different clinical environment dimensions can support motivation and be modified to create optimal motivating situations. To understand motivational dynamics and support trainees to navigate through postgraduate medical education, we need to take all clinical environment dimensions into account.

Considering the interdependence of clinical performance: implications for assessment and entrustment.

INTRODUCTION: Our ability to assess independent trainee performance is a key element of competency-based medical education (CBME). In workplace-based clinical settings, however, the performance of a trainee can be deeply entangled with others on the team. This presents a fundamental challenge, given the need to assess and entrust trainees based on the evolution of their independent clinical performance. The purpose of this study, therefore, was to understand what faculty members and senior postgraduate trainees believe constitutes independent performance in a variety of clinical specialty contexts. METHODS: Following constructivist grounded theory, and using both purposive and theoretical sampling, we conducted individual interviews with 11 clinical teaching faculty members and 10 senior trainees (postgraduate year 4/5) across 12 postgraduate specialties. Constant comparative inductive analysis was conducted. Return of findings was also carried out using one-to-one sessions with key informants and public presentations. RESULTS: Although some independent performances were described, participants spoke mostly about the exceptions to and disclaimers about these, elaborating their sense of the interdependence of trainee performances. Our analysis of these interdependence patterns identified multiple configurations of coupling, with the dominant being coupling of trainee and supervisor performance. We consider how the concept of coupling could advance workplace-based assessment efforts by supporting models that account for the collective dimensions of clinical performance. CONCLUSION: These findings call into question the assumption of independent performance, and offer an important step toward measuring coupled performance. An understanding of coupling can help both to better distinguish independent and interdependent performances, and to consider revising workplace-based assessment approaches for CBME.

'Oh my God, I can't handle this!': trainees' emotional responses to complex situations.

CONTEXT: Dealing with emotions is critical for medical trainees' professional development. Taking a sociocultural and narrative approach to understanding emotions, we studied complex clinical situations as a specific context in which emotions are evoked and influenced by the social environment. We sought to understand how medical trainees respond to emotions that arise in those situations. METHODS: In an international constructivist grounded theory study, 29 trainees drew two rich pictures of complex clinical situations, one exciting and one frustrating. Rich pictures are visual representations that capture participants' perceptions about the people, situations and factors that create clinical complexity. These pictures were used to guide semi-structured, individual interviews. We analysed visual materials and interviews in an integrated way, starting with looking at the drawings, doing a 'gallery walk', and using the interviews to inform the aesthetic analysis. RESULTS: Participants' drawings depicted a range of personal emotions in response to complexity, and disclosed unsettling feelings and behaviours that might be considered unprofessional. When trainees felt confident, they were actively participating, engaged in creative problem-solving strategies, and emphasised their personal involvement. When trainees felt the situation was beyond their control, they described how they were running away from the situation, hiding themselves behind others or distancing themselves from patients or families. CONCLUSIONS: A sense of control seems to be a key factor influencing trainees' emotional and behavioural responses to complexity. This is problematic, as complex situations are by their nature emergent and dynamic, which limits possibilities for control. Following a social performative approach to emotions, we should help students understand that feeling out of control is an inherent property of participating in complex clinical situations, and, by extension, that it is not something they will 'grow out of' with expertise.

The validation of an instrumented simulator for the assessment of performance and outcome of knot tying skill: a pilot study.

The construct validity of a surgical bench-top simulator with built-in computer acquired assessments was examined. It features two parallel elastic tubes instrumented with flexion sensors that simulate the walls of a wound. Participants from three groups (9 novices, 7 intermediates, 9 experts) performed 10 two-handed, double square knots. The peak tensions at the initiation of the first knot, the completion of the first knot and the completion of the second knot, as well as measures of movement economy indicated technical performance. Product quality was indicated by knot stability defined as the amount of slippage of the knot under the tension. There were significant differences between experts and novices for peak tension on first knot (p=.03), movement economy (p=.02), and knot stability (p=.002). The results support the construct validity of these objective measures.