Defining entrustable professional activities for first year anesthesiology residents: A Delphi study.

INTRODUCTION: The use of entrustable professional activities (EPAs) as a basis for assessment may bridge the gap between the theory of competency-based education and clinical practice. The purpose of this study was to develop and validate EPAs for United States (US) first-year clinical anesthesia (CA-1) residents for anesthesiology residency programs to use as the basis for curriculum development and workplace assessment. METHODS: From a list of EPAs abstracted from the literature, an expert panel through a modified Delphi consensus process established EPAs for the CA1 curriculum. RESULTS: The final list of EPAs after group consensus had 28 EPAs, with 14 (50%) considered to be applicable to the CA-1 year. An 80% consensus rate was used to accept or reject the final list. CONCLUSION: This study applied a construct validity lens to EPA development providing assurance that the EPAs adopted are appropriate for use in workplace-based assessment and entrustment decision-making.

Exploring the Impact of Entrustable Professional Activities on Feedback Culture: A Qualitative Study of Anesthesiology Residents and Attendings.

PURPOSE: Entrustable professional activities (EPAs) were introduced as a potential way to optimize workplace-based assessments. Yet, recent studies suggest that EPAs have not yet overcome all of the challenges to implementing meaningful feedback. The aim of this study was to explore the extent to which the introduction of EPAs via mobile app impacts feedback culture as experienced by anesthesiology residents and attending physicians. METHOD: Using a constructivist grounded theory approach, the authors interviewed a purposive and theoretical sample of residents (n = 11) and attendings (n = 11) at the Institute of Anaesthesiology, University Hospital of Zurich, where EPAs had recently been implemented. Interviews took place between February and December 2021. Data collection and analysis were conducted iteratively. The authors used open, axial, and selective coding to gain knowledge and understanding on the interplay of EPAs and feedback culture. RESULTS: Participants reflected on a number of changes in their day-to-day experience of feedback culture with the implementation of EPAs. Three main mechanisms were instrumental in this process: lowering the feedback threshold, change in feedback focus, and gamification. Participants felt a lower threshold to feedback seeking and giving and that the frequency of feedback conversations increased and tended to be more focused on a specific topic and shorter, while feedback content tended to focus more on technical skills and more attention was given to average performances. Residents indicated that the app-based approach fostered a game-like motivation to "climb levels," while attendings did not perceive a game-like experience. CONCLUSIONS: EPAs may offer a solution to problems of infrequent occurrence of feedback and invite attention to average performances and technical competencies, but may come at the expense of feedback on nontechnical skills. This study suggests that feedback culture and feedback instruments have a mutually interacting influence on each other.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154.

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

Surgery goes EPA (Entrustable Professional Activity) - how a strikingly easy to use app revolutionizes assessments of clinical skills in surgical training.

OBJECTIVE: Entrustable Professional Activities (EPAs) are increasingly being used in competency-based medical education approaches. A general lack of time in clinical settings, however, prevents supervisors from providing their trainees with adequate feedback. With a willingness for more administrative tasks being low in both trainees and educators, the authors developed a radical user-friendly mobile application based on the EPA concept called "Surg-prEPAred". DESIGN: Surg-prEPAred is designed to collect micro-assessment data for building competency profiles for surgical residents according to their curriculum. The goal of Surg-prEPAred is to facilitate the performance and documentation of workplace-based assessments. Through aggregated data the app generates a personalized competency profile for every trainee. During a pilot run of 4 months, followed by ongoing usage of the application with a total duration of 9 months (August 2019 to April 2020), 32 residents and 33 consultants made daily use of the application as a rating tool. Every rating included knowledge, skills and professional attitudes of the trainees. Before the initiation of the App and after the 9-month trial period trainees and supervisors where both sent questionnaires to evaluate the user friendliness and effectiveness of the App. RESULTS: Five hundred ten App based assessments were generated. Out of 40 pre-defined EPAs, 36 were assessed. 15 trainees and 16 supervisors returned the questionnaires and stated the surg-prEPAred App as very valuable, effective and feasible to evaluate trainees in a clinical setting providing residents with an individual competence portfolio to receive precision medical education. CONCLUSIONS: The authors expectation is that the Surg-prEPAred App will contribute to an improvement of quality of medical education and thus to the quality of patient care and safety. In the future the goal is to have the App become an integral part of the official Swiss surgical curriculum accepted by the Swiss professional surgical society.

A mobile application to facilitate implementation of programmatic assessment in anaesthesia training.

BACKGROUND: Programmatic assessment is a concept to support learning through continuously providing information about learner progress to trainees and supervisors. Central to the concept are multiple low-stakes workplace-based assessments and meaningful feedback opportunities. Mobile technology may facilitate frequent and concise workplace-based assessments and trigger meaningful feedback. We designed a mobile application (app) for real-time use at the workplace utilising the concept of entrustable professional activities. As the primary outcome, we analysed completion times and as the secondary outcome the quality of documented learning goals. METHODS: The prEPAred app requests trainees and supervisors to rate level of supervision of a professional activity directly after completion. Subsequently, ratings are compared, and supervisors may provide feedback via the app. We tested the app in five anaesthesiology departments at major teaching hospitals, analysing completion times, agreement on ratings, and quality of documented learning goals. RESULTS: We recorded 1518 assessments from 159 trainees and 89 supervisors. Median time for level of supervision rating was 56 (inter-quartile range: 39-85) s for trainees and 17 (11-30) s for supervisors. Learning goals via the app were documented in 767 cases (50.5%). Median feedback time was 2 min, 31 s (confidence interval [CI]: 1 min, 20 s to 5 min, 20 s). In 443 (29%) cases, a specific learning goal was documented. A post hoc analysis revealed that the odds of documenting learning goals increased if trainees rated the level of supervision higher than their supervisors (odds ratio 1.39; CI: 1.03-1.87). CONCLUSIONS: The prEPAred mobile app enabled frequent and concise documentation of workplace-based assessments. Disagreement in level of supervision rating stimulated documentation of specific learning goals indicating more meaningful feedback. Thus, the tool could advance workplace-based assessments towards programmatic assessment.

Bottom-up feedback to improve clinical teaching: validation of the Swiss System for Evaluation of Teaching Qualities (SwissSETQ).

AIMS OF THE STUDY: Clinical teaching is essential in preparing trainees for independent practice. To improve teaching quality, clinical teachers should be provided with meaningful and reliable feedback from trainees (bottom-up feedback) based on up-to-date educational concepts. For this purpose, we designed a web-based instrument, "Swiss System for Evaluation of Teaching Qualities" (SwissSETQ), building on a well-established tool (SETQsmart) and expanding it with current graduate medical education concepts. This study aimed to validate the new instrument in the field of anaesthesiology training. METHODS: Based on SETQsmart, we developed an online instrument (primarily including 34 items) with generic items to be used in all clinical disciplines. We integrated the recent educational frameworks of CanMEDS 2015 (Canadian Medical Educational Directives for Specialists), and of entrustable professional activities (EPAs). Newly included themes were "Interprofessionalism", "Patient centredness", "Patient safety", "Continuous professional development', and "Entrustment decisions". We ensured content validity by iterative discussion rounds between medical education specialists and clinical supervisors. Two think-aloud rounds with residents investigated the response process. Subsequently, the instrument was pilot-tested in the anaesthesia departments of four major teaching hospitals in Switzerland, involving 220 trainees and 120 faculty. We assessed the instrument's internal structure (to determine the factorial composition) using exploratory factor analysis, internal statistical consistency (by Cronbach's alpha as an estimate of reliability, regarding alpha >0.7 as acceptable, >0.8 as good, >0.9 as excellent), and inter-rater reliability (using generalisability theory in order to assess the minimum number of ratings necessary for a valid feedback to one single supervisor). RESULTS: Based on 185 complete ratings for 101 faculty, exploratory factor analysis revealed four factors explaining 72.3% of the variance (individual instruction 33.8%, evaluation of trainee performance 20.9%, teaching professionalism 12.8%; entrustment decisions 4.7%). Cronbach's alpha for the total score was 0.964. After factor analysis, we removed one item to arrive at 33 items for the final instrument. Generalisability studies yielded a minimum of five to six individual ratings to provide reliable feedback to one supervisor. DISCUSSION: The SwissSETQ possesses high content validity and an "excellent" internal structure for integrating up-to-date graduate medical education concepts. Thereby, the tool allows reliable bottom-up feedback by trainees to support clinical teachers in improving their teaching. Transfer to disciplines other than anaesthesiology needs to be further explored.

Development and Pilot Testing of Entrustable Professional Activities for US Anesthesiology Residency Training.

BACKGROUND: Modern medical education requires frequent competency assessment. The Accreditation Council for Graduate Medical Education (ACGME) provides a descriptive framework of competencies and milestones but does not provide standardized instruments to assess and track trainee competency over time. Entrustable professional activities (EPAs) represent a workplace-based method to assess the achievement of competency milestones at the point-of-care that can be applied to anesthesiology training in the United States. METHODS: Experts in education and competency assessment were recruited to participate in a 6-step process using a modified Delphi method with iterative rounds to reach consensus on an entrustment scale, a list of EPAs and procedural skills, detailed definitions for each EPA, a mapping of the EPAs to the ACGME milestones, and a target level of entrustment for graduating US anesthesiology residents for each EPA and procedural skill. The defined EPAs and procedural skills were implemented using a website and mobile app. The assessment system was piloted at 7 anesthesiology residency programs. After 2 months, faculty were surveyed on their attitudes on usability and utility of the assessment system. The number of evaluations submitted per month was collected for 1 year. RESULTS: Participants in EPA development included 18 education experts from 11 different programs. The Delphi rounds produced a final list of 20 EPAs, each differentiated as simple or complex, a defined entrustment scale, mapping of the EPAs to milestones, and graduation entrustment targets. A list of 159 procedural skills was similarly developed. Results of the faculty survey demonstrated favorable ratings on all questions regarding app usability as well as the utility of the app and EPA assessments. Over the 2-month pilot period, 1636 EPA and 1427 procedure assessments were submitted. All programs continued to use the app for the remainder of the academic year resulting in 12,641 submitted assessments. CONCLUSIONS: A list of 20 anesthesiology EPAs and 159 procedural skills assessments were developed using a rigorous methodology to reach consensus among education experts. The assessments were pilot tested at 7 US anesthesiology residency programs demonstrating the feasibility of implementation using a mobile app and the ability to collect assessment data. Adoption at the pilot sites was variable; however, the use of the system was not mandatory for faculty or trainees at any site.

An analysis of core EPAs reveals a gap between curricular expectations and medical school graduates' self-perceived level of competence.

BACKGROUND: Entrustable Professional Activities (EPAs) are being implemented worldwide as a means to promote competency-based medical education. In Switzerland, the new EPA-based curriculum for undergraduate medical education will be implemented in 2021. The aim of our study was to analyze the perceived, self-reported competence of graduates in 2019. The data represent a pre-implementation baseline and will provide guidance for curriculum developers. METHODS: Two hundred eighty-one graduates of the Master of Human Medicine program of the University of Zurich who had passed the Federal Licensing Exam in September 2019 were invited to complete an online survey. They were asked to rate their needed level of supervision ("observe only", "direct, proactive supervision", "indirect, reactive supervision") for 46 selected EPAs. We compared the perceived competence with the expected competence of the new curriculum. RESULTS: The response rate was 54%. The need for supervision expressed by graduates varied considerably by EPA. The proportion of graduates rating themselves at expected level was high for "history taking", "physical examination" "and documentation"; medium for "prioritizing differential diagnoses", "interpreting results" and "developing and communicating a management plan"; low for "practical skills"; and very low for EPAs related to "urgent and emergency care". CONCLUSIONS: Currently, there are significant gaps between the expectations of curriculum developers and the perceived competences of students. This is most obvious for practical skills and emergency situations. The new curriculum will either need to fill this gap or expectations might need to be revised.

[Entrustable Professional Activities in Graduate Medical Education in Psychiatry: A Promising Concept]. / Entrustable Professional Activities in der psychiatrischen Weiterbildung: Ein vielversprechendes Konzept.

Entrustable Professional Activities in Graduate Medical Education in Psychiatry: A Promising Concept Abstract. Entrustable Professional Activities (EPAs) are competency-based learning goals derived from observable clinical activities. In undergraduate medical education, they have now been adopted throughout Switzerland as part of the so-called PROFILES catalog (Principal Relevant Objectives and Framework for Integrated Learning and Education in Switzerland). The nine core EPAs to be mastered in undergraduate medical education can serve as a basis for introducing EPAs in graduate medical education as well. We will discuss this approach in the context of graduate medical education in psychiatry and psychotherapy from the perspective of different training contexts and a pilot example. In this position paper, we describe a promising opportunity to improve graduate medical training through the implementation of EPAs, both in terms of the quality of training and thus of patient care, as well as in terms of the attractiveness of the specialty for future residents.

Agreement between trainees and supervisors on first-year entrustable professional activities for anaesthesia training.

BACKGROUND: Entrustable professional activities (EPAs) are commonly developed by senior clinicians and education experts. However, if postgraduate training is conceptualised as an educational alliance, the perspective of trainees should be included. This raises the question as to whether the views of trainees and supervisors on entrustability of specific EPAs differ, which we aimed to explore. METHODS: A working group, including all stakeholders, selected and drafted 16 EPAs with the potential for unsupervised practice within the first year of training. For each EPA, first-year trainees, advanced trainees, and supervisors decided whether it should be possible to attain trust for unsupervised practice by the end of the first year of anaesthesiology training (i.e. whether the respective EPA qualified as a 'first-year EPA'). RESULTS: We surveyed 23 first-year trainees, 47 advanced trainees, and 51 supervisors (overall response rate: 68%). All groups fully agreed upon seven EPAs as 'first-year EPAs' and on four EPAs that should not be entrusted within the first year. For all five remaining EPAs, a significantly higher proportion of first-year trainees thought these should be entrusted as first-year EPAs compared with advanced trainees and supervisors. We found no differences between advanced trainees and supervisors. CONCLUSIONS: The views of first-year trainees, advanced trainees, and supervisors showed high agreement. Differing views of young trainees disappeared after the first year. This finding provides a fruitful basis to involve trainees in negotiations of autonomy.

Barriers and aids to routine neuromuscular monitoring and consistent reversal practice-A qualitative study.

BACKGROUND: Neuromuscular monitoring is recommended whenever a neuromuscular blocking agent is administered, but surveys have demonstrated inconsistent monitoring practices. Using qualitative methods, we aimed to explore barriers and aids to routine neuromuscular monitoring and consistent reversal practice. METHODS: Focus group interviews were conducted to obtain insights into the thoughts and attitudes of individual anaesthetists, as well as the influence of colleagues and department culture. Interviews were conducted at five Danish and one US hospital. Data were analysed using template analysis. RESULTS: Danish anaesthetists used objective neuromuscular monitoring when administering a non-depolarizing relaxant, but had challenges with calibrating the monitor and sometimes interpreting measurements. Residents from the US institution used subjective neuromuscular monitoring, objective neuromuscular monitoring was generally not available and most had not used it. Danish anaesthetists used neuromuscular monitoring to assess readiness for extubation, whereas US residents used subjective neuromuscular monitoring, clinical tests like 5-second head lift and ventilatory parameters. The residents described a lack of consensus between senior anaesthesiologists in reversal practice and monitoring use. Barriers to consistent and correct neuromuscular monitoring identified included unreliable equipment, time pressure, need for training, misconceptions about pharmacokinetics of neuromuscular blocking agents and residual block, lack of standards and guidelines and departmental culture. CONCLUSION: Using qualitative methods, we found that though Danish anaesthetists generally apply objective neuromuscular monitoring routinely and residents at the US institution often apply subjective neuromuscular monitoring, barriers to consistent and correct use still exist.

What Makes for Good Anesthesia Teaching by Faculty in the Operating Room? The Perspective of Anesthesiology Residents.

Background Teaching during patient care is an important competency for faculty. Little is known about anesthesiology resident preferences for teaching by anesthesiology faculty in the operating room (OR). If the behaviors and characteristics of anesthesia teaching in the OR that are most valued by residents were identified, faculty could incorporate that best practice to teach residents during OR cases. The objective of this phenomenological study was to interview anesthesiology residents to determine what they perceive the best faculty teachers are doing in the OR to educate residents. Methods Thirty randomly selected anesthesiology residents (10 in each post-graduate year class) were interviewed using a semi-structured approach with a predetermined question: "Based on your experiences as a resident, when you think about the best-attending teachers in the OR, what are the best-attending teachers doing in the OR to teach that other faculty maybe are not doing?" Interviews were recorded, transcribed, converted into codes, and grouped into themes derived from the cognitive apprenticeship framework, which includes content, teaching methods, sequencing, and social characteristics. Results Resident responses were separated into a total of 134 answers, with similar answers grouped into one of 27 different codes. The most commonly mentioned codes were: autonomy - step back and let resident work through (mentioned by 13 residents), reasoning - explain why attending does things (12), context - teach something relevant to the case (8), commitment - take time to teach (8), literature - bring relevant papers (8), prior knowledge - assess the baseline level (7), flexibility - be open to trying different approaches (7), focus on just a few learning points (6), reflection - ask resident questions (6), provide real-time feedback (6), teach back - ask residents to explain what they were taught in their own words (5), belonging - facilitate communication with the OR team (5), psychological safety - be open and approachable (5), equanimity - stay calm and collected (5), select proper timing for instruction when the resident is not occupied with patient care (5), visualization - use graphs or diagrams (5), and specify learning goals ahead of time (5). Conclusion The best practice for OR teaching, as perceived by anesthesia residents, includes social characteristics, such as context, commitment, psychological safety, equanimity, and proper timing, as well as teaching methods, such as autonomy, reasoning, literature, prior knowledge, flexibility, reflection, real-time feedback, and teach back. Further studies can determine if training anesthesiology faculty to incorporate these elements increases the caliber of daily teaching in the OR.

Development of an Objective Structured Clinical Examination Using the American Board of Anesthesiology Content Outline for the Objective Structured Clinical Examination Component of the APPLIED Certification Examination.

The goal of this study was to use the American Board of Anesthesiology Objective Structured Clinical Examination (OSCE) content outline as a blueprint to develop and administer a 9-station mock OSCE with station-specific checklists to senior residents (n = 14). The G- and Ф-coefficient reliability estimates were 0.76 and 0.61, respectively. Residents judged the scenarios as either extremely or somewhat realistic (88%). It is feasible to develop and administer a mock OSCE with rigorous psychometric characteristics.

How to debrief teamwork interactions: using circular questions to explore and change team interaction patterns.

We submit that interaction patterns within healthcare teams should be more comprehensively explored during debriefings in simulation-based training because of their importance for clinical performance. We describe how circular questions can be used for that purpose. Circular questions are based on social constructivism. They include a variety of systemic interviewing methods. The goals of circular questions are to explore the mutual dependency of team members' behavior and recurrent behavior patterns, to generate information, to foster perspective taking, to "fluidize" problems, and to put actions into relational contexts. We describe the nature of circular questions, the benefits they offer, and ways of applying them during debriefings.

Simulation With PARTS (Phase-Augmented Research and Training Scenarios): A Structure Facilitating Research and Assessment in Simulation.

INTRODUCTION: Assessment in simulation is gaining importance, as are scenario design methods increasing opportunity for assessment. We present our approach to improving measurement in complex scenarios using PARTS [Phase-Augmented Research and Training Scenarios], essentially separating cases into clearly delineated phases. METHODS: We created 7 PARTS with real-time rating instruments and tested these in 63 cases during 4 weeks of simulation. Reliability was tested by comparing real-time rating with postsimulation video-based rating using the same instrument. Validity was tested by comparing preintervention and postintervention total results, by examining the difference in improvement when focusing on the phase-specific results addressed by the intervention, and further explored by trying to demonstrate the discrete improvement expected from proficiency in the rare occurrence of leader inclusive behavior. RESULTS: Intraclass correlations [3,1] between real-time and postsimulation ratings were 0.951 (95% confidence interval [CI], 0.794-0.990), 1.00 (95% CI, --to--), 0.948 (95% CI, 0.783-0.989), and 0.995 (95% CI, 0.977-0.999) for 3 phase-specific scores and total scenario score, respectively. Paired t tests of prelecture-postlecture performance showed an improvement of 14.26% (bias-corrected and accelerated bootstrap [BCa] 95% CI, 4.71-23.82; P = 0.009) for total performance but of 28.57% (BCa 95% CI, 13.84-43.30; P = 0.002) for performance in the respective phase. The correlation of total scenario performance with leader inclusiveness was not significant (rs = 0.228; BCa 95% CI. -0.082 to 0.520; P = 0.119) but significant for specific phase performance (rs = 0.392; BCa 95% CI, 0.118-0.632; P = 0.006). CONCLUSIONS: The PARTS allowed for improved reliability and validity of measurements in complex scenarios.

Detection of Haemophilus influenzae type b by real-time PCR.

A real-time PCR assay targeting the capsulation locus of Haemophilus influenzae type b (Hib) was developed. The linear detection range was from 1 to 10(6) microorganisms per reaction mixture. No H. influenzae other than Hib or any other control bacteria typically found in the upper respiratory tract was detected.