Exploring the Impact of Entrustable Professional Activities on Feedback Culture: A Qualitative Study of Anesthesiology Residents and Attendings.

PURPOSE: Entrustable professional activities (EPAs) were introduced as a potential way to optimize workplace-based assessments. Yet, recent studies suggest that EPAs have not yet overcome all of the challenges to implementing meaningful feedback. The aim of this study was to explore the extent to which the introduction of EPAs via mobile app impacts feedback culture as experienced by anesthesiology residents and attending physicians. METHOD: Using a constructivist grounded theory approach, the authors interviewed a purposive and theoretical sample of residents (n = 11) and attendings (n = 11) at the Institute of Anaesthesiology, University Hospital of Zurich, where EPAs had recently been implemented. Interviews took place between February and December 2021. Data collection and analysis were conducted iteratively. The authors used open, axial, and selective coding to gain knowledge and understanding on the interplay of EPAs and feedback culture. RESULTS: Participants reflected on a number of changes in their day-to-day experience of feedback culture with the implementation of EPAs. Three main mechanisms were instrumental in this process: lowering the feedback threshold, change in feedback focus, and gamification. Participants felt a lower threshold to feedback seeking and giving and that the frequency of feedback conversations increased and tended to be more focused on a specific topic and shorter, while feedback content tended to focus more on technical skills and more attention was given to average performances. Residents indicated that the app-based approach fostered a game-like motivation to "climb levels," while attendings did not perceive a game-like experience. CONCLUSIONS: EPAs may offer a solution to problems of infrequent occurrence of feedback and invite attention to average performances and technical competencies, but may come at the expense of feedback on nontechnical skills. This study suggests that feedback culture and feedback instruments have a mutually interacting influence on each other.

A mobile application to facilitate implementation of programmatic assessment in anaesthesia training.

BACKGROUND: Programmatic assessment is a concept to support learning through continuously providing information about learner progress to trainees and supervisors. Central to the concept are multiple low-stakes workplace-based assessments and meaningful feedback opportunities. Mobile technology may facilitate frequent and concise workplace-based assessments and trigger meaningful feedback. We designed a mobile application (app) for real-time use at the workplace utilising the concept of entrustable professional activities. As the primary outcome, we analysed completion times and as the secondary outcome the quality of documented learning goals. METHODS: The prEPAred app requests trainees and supervisors to rate level of supervision of a professional activity directly after completion. Subsequently, ratings are compared, and supervisors may provide feedback via the app. We tested the app in five anaesthesiology departments at major teaching hospitals, analysing completion times, agreement on ratings, and quality of documented learning goals. RESULTS: We recorded 1518 assessments from 159 trainees and 89 supervisors. Median time for level of supervision rating was 56 (inter-quartile range: 39-85) s for trainees and 17 (11-30) s for supervisors. Learning goals via the app were documented in 767 cases (50.5%). Median feedback time was 2 min, 31 s (confidence interval [CI]: 1 min, 20 s to 5 min, 20 s). In 443 (29%) cases, a specific learning goal was documented. A post hoc analysis revealed that the odds of documenting learning goals increased if trainees rated the level of supervision higher than their supervisors (odds ratio 1.39; CI: 1.03-1.87). CONCLUSIONS: The prEPAred mobile app enabled frequent and concise documentation of workplace-based assessments. Disagreement in level of supervision rating stimulated documentation of specific learning goals indicating more meaningful feedback. Thus, the tool could advance workplace-based assessments towards programmatic assessment.

Bottom-up feedback to improve clinical teaching: validation of the Swiss System for Evaluation of Teaching Qualities (SwissSETQ).

AIMS OF THE STUDY: Clinical teaching is essential in preparing trainees for independent practice. To improve teaching quality, clinical teachers should be provided with meaningful and reliable feedback from trainees (bottom-up feedback) based on up-to-date educational concepts. For this purpose, we designed a web-based instrument, "Swiss System for Evaluation of Teaching Qualities" (SwissSETQ), building on a well-established tool (SETQsmart) and expanding it with current graduate medical education concepts. This study aimed to validate the new instrument in the field of anaesthesiology training. METHODS: Based on SETQsmart, we developed an online instrument (primarily including 34 items) with generic items to be used in all clinical disciplines. We integrated the recent educational frameworks of CanMEDS 2015 (Canadian Medical Educational Directives for Specialists), and of entrustable professional activities (EPAs). Newly included themes were "Interprofessionalism", "Patient centredness", "Patient safety", "Continuous professional development', and "Entrustment decisions". We ensured content validity by iterative discussion rounds between medical education specialists and clinical supervisors. Two think-aloud rounds with residents investigated the response process. Subsequently, the instrument was pilot-tested in the anaesthesia departments of four major teaching hospitals in Switzerland, involving 220 trainees and 120 faculty. We assessed the instrument's internal structure (to determine the factorial composition) using exploratory factor analysis, internal statistical consistency (by Cronbach's alpha as an estimate of reliability, regarding alpha >0.7 as acceptable, >0.8 as good, >0.9 as excellent), and inter-rater reliability (using generalisability theory in order to assess the minimum number of ratings necessary for a valid feedback to one single supervisor). RESULTS: Based on 185 complete ratings for 101 faculty, exploratory factor analysis revealed four factors explaining 72.3% of the variance (individual instruction 33.8%, evaluation of trainee performance 20.9%, teaching professionalism 12.8%; entrustment decisions 4.7%). Cronbach's alpha for the total score was 0.964. After factor analysis, we removed one item to arrive at 33 items for the final instrument. Generalisability studies yielded a minimum of five to six individual ratings to provide reliable feedback to one supervisor. DISCUSSION: The SwissSETQ possesses high content validity and an "excellent" internal structure for integrating up-to-date graduate medical education concepts. Thereby, the tool allows reliable bottom-up feedback by trainees to support clinical teachers in improving their teaching. Transfer to disciplines other than anaesthesiology needs to be further explored.

Development and Pilot Testing of Entrustable Professional Activities for US Anesthesiology Residency Training.

BACKGROUND: Modern medical education requires frequent competency assessment. The Accreditation Council for Graduate Medical Education (ACGME) provides a descriptive framework of competencies and milestones but does not provide standardized instruments to assess and track trainee competency over time. Entrustable professional activities (EPAs) represent a workplace-based method to assess the achievement of competency milestones at the point-of-care that can be applied to anesthesiology training in the United States. METHODS: Experts in education and competency assessment were recruited to participate in a 6-step process using a modified Delphi method with iterative rounds to reach consensus on an entrustment scale, a list of EPAs and procedural skills, detailed definitions for each EPA, a mapping of the EPAs to the ACGME milestones, and a target level of entrustment for graduating US anesthesiology residents for each EPA and procedural skill. The defined EPAs and procedural skills were implemented using a website and mobile app. The assessment system was piloted at 7 anesthesiology residency programs. After 2 months, faculty were surveyed on their attitudes on usability and utility of the assessment system. The number of evaluations submitted per month was collected for 1 year. RESULTS: Participants in EPA development included 18 education experts from 11 different programs. The Delphi rounds produced a final list of 20 EPAs, each differentiated as simple or complex, a defined entrustment scale, mapping of the EPAs to milestones, and graduation entrustment targets. A list of 159 procedural skills was similarly developed. Results of the faculty survey demonstrated favorable ratings on all questions regarding app usability as well as the utility of the app and EPA assessments. Over the 2-month pilot period, 1636 EPA and 1427 procedure assessments were submitted. All programs continued to use the app for the remainder of the academic year resulting in 12,641 submitted assessments. CONCLUSIONS: A list of 20 anesthesiology EPAs and 159 procedural skills assessments were developed using a rigorous methodology to reach consensus among education experts. The assessments were pilot tested at 7 US anesthesiology residency programs demonstrating the feasibility of implementation using a mobile app and the ability to collect assessment data. Adoption at the pilot sites was variable; however, the use of the system was not mandatory for faculty or trainees at any site.

Agreement between trainees and supervisors on first-year entrustable professional activities for anaesthesia training.

BACKGROUND: Entrustable professional activities (EPAs) are commonly developed by senior clinicians and education experts. However, if postgraduate training is conceptualised as an educational alliance, the perspective of trainees should be included. This raises the question as to whether the views of trainees and supervisors on entrustability of specific EPAs differ, which we aimed to explore. METHODS: A working group, including all stakeholders, selected and drafted 16 EPAs with the potential for unsupervised practice within the first year of training. For each EPA, first-year trainees, advanced trainees, and supervisors decided whether it should be possible to attain trust for unsupervised practice by the end of the first year of anaesthesiology training (i.e. whether the respective EPA qualified as a 'first-year EPA'). RESULTS: We surveyed 23 first-year trainees, 47 advanced trainees, and 51 supervisors (overall response rate: 68%). All groups fully agreed upon seven EPAs as 'first-year EPAs' and on four EPAs that should not be entrusted within the first year. For all five remaining EPAs, a significantly higher proportion of first-year trainees thought these should be entrusted as first-year EPAs compared with advanced trainees and supervisors. We found no differences between advanced trainees and supervisors. CONCLUSIONS: The views of first-year trainees, advanced trainees, and supervisors showed high agreement. Differing views of young trainees disappeared after the first year. This finding provides a fruitful basis to involve trainees in negotiations of autonomy.

Barriers and aids to routine neuromuscular monitoring and consistent reversal practice-A qualitative study.

BACKGROUND: Neuromuscular monitoring is recommended whenever a neuromuscular blocking agent is administered, but surveys have demonstrated inconsistent monitoring practices. Using qualitative methods, we aimed to explore barriers and aids to routine neuromuscular monitoring and consistent reversal practice. METHODS: Focus group interviews were conducted to obtain insights into the thoughts and attitudes of individual anaesthetists, as well as the influence of colleagues and department culture. Interviews were conducted at five Danish and one US hospital. Data were analysed using template analysis. RESULTS: Danish anaesthetists used objective neuromuscular monitoring when administering a non-depolarizing relaxant, but had challenges with calibrating the monitor and sometimes interpreting measurements. Residents from the US institution used subjective neuromuscular monitoring, objective neuromuscular monitoring was generally not available and most had not used it. Danish anaesthetists used neuromuscular monitoring to assess readiness for extubation, whereas US residents used subjective neuromuscular monitoring, clinical tests like 5-second head lift and ventilatory parameters. The residents described a lack of consensus between senior anaesthesiologists in reversal practice and monitoring use. Barriers to consistent and correct neuromuscular monitoring identified included unreliable equipment, time pressure, need for training, misconceptions about pharmacokinetics of neuromuscular blocking agents and residual block, lack of standards and guidelines and departmental culture. CONCLUSION: Using qualitative methods, we found that though Danish anaesthetists generally apply objective neuromuscular monitoring routinely and residents at the US institution often apply subjective neuromuscular monitoring, barriers to consistent and correct use still exist.