Bringing competency-based communication training to scale: A multi-institutional virtual simulation-based mastery learning curriculum for Emergency Medicine residents.

PURPOSE: Serious illness communication skills are essential for physicians, yet competency-based training is lacking. We address scalability barriers to competency-based communication skills training by assessing the feasibility of a multi-center, virtual simulation-based mastery learning (vSBML) curriculum on breaking bad news (BBN). METHODS: First-year emergency medicine residents at three academic medical centers participated in the virtual curriculum. Participants completed a pretest with a standardized patient (SP), a workshop with didactics and small group roleplay with SPs, a posttest with an SP, and additional deliberate practice sessions if needed to achieve the minimum passing standard (MPS). Participants were assessed using a previously published BBN assessment tool that included a checklist and scaled items. Authors compared pre- and posttests to evaluate the impact of the curriculum. RESULTS: Twenty-eight (90%) of 31 eligible residents completed the curriculum. Eighty-nine percent of participants did not meet the MPS at pretest. Post-intervention, there was a statistically significant improvement in checklist performance (Median= 93% vs. 53%, p < 0.001) and on all scaled items assessing quality of communication. All participants ultimately achieved the MPS. CONCLUSIONS: A multi-site vSBML curriculum brought all participants to mastery in the core communication skill of BBN and represents a feasible, scalable model to incorporate competency-based communication skills education in a widespread manner.

Simulation Based Mastery Learning of Transesophageal Echocardiography.

Transesophageal echocardiography (TEE) education is part of pediatric cardiology fellow training. Simulation-based mastery learning (SBML) is an efficient and valuable education experience. The aim of this project was to equip trainees with the basic knowledge and skill required to perform a pediatric TEE. The secondary aim was to assess the utility of using SBML for pediatric TEE training. The target group is trainees from pediatric cardiology and cardiac anesthesia who participated in a TEE bootcamp. A baseline knowledge pretest was obtained. The knowledge session consisted of preparation via reading material, viewing recorded lectures and completing an iterative multiple-choice examination, which was repeated until a minimum passing score of 90% was achieved. The skills session involved a review of TEE probe manipulation and image acquisition, followed by rapid cycle deliberate practice using simulation to acquire TEE skills at 3 levels, advancing in complexity from level 1 to level 3. Eight individuals (7 pediatric cardiology fellows at varying training levels and one anesthesia attending) participated in the TEE bootcamp. All reached a minimum knowledge post test score of at least 90% before the skills session. All subjects reached mastery in TEE probe manipulation. All reached mastery in image acquisition for the skill level that they attempted (level 1-8/8, level 2-8/8, level 3-4/4, with 4 participants not attempting level 3). A TEE bootcamp using SBML is a powerful medical education strategy. SBML is a rigorous approach that can be used to achieve high and uniform TEE learning outcomes among trainees of different training levels and backgrounds.

Effect of Ventricular Assist Device Self-care Simulation-Based Mastery Learning on Driveline Exit Site Infections: A Pilot Study.

BACKGROUND: Ventricular assist device simulation-based mastery learning (SBML) results in better patient and caregiver self-care skills compared with usual training. OBJECTIVE: The aim of this study was to evaluate the effect of SBML on driveline exit site infections. METHODS: We compared the probability of remaining infection free at 3 and 12 months between patients randomized to SBML or usual training. RESULTS: The SBML-training group had no infections at 3 months and 2 infections at 12 months, yielding a Kaplan-Meier estimate of the probability of remaining infection free of 0.857 (95% confidence interval [CI], 0.692-1.00) at 12 months. The usual-training group had 6 infections at 3 months with no additional infections by 12 months. Kaplan-Meier estimates of remaining infection free at 3 and 12 months were 0.878 (95% CI, 0.758-1.00) and 0.748 (95% CI, 0.591-0.946), respectively. Time-to-infection distributions for SBML versus usual training showed a difference in 12-month infection rates of 0.109 (P = .07). CONCLUSIONS: Ventricular assist device self-care SBML resulted in fewer 12-month infections.

A mastery learning approach to education about fall risk and gait assessment.

Mastery learning is a form of competency-based education in which learning time varies but outcomes are uniform. Trainees must meet a minimum passing standard (MPS) before completing a mastery learning curriculum. The objective of this study was to establish a curriculum for fall risk and gait assessment for medical students, determine an MPS for a fall risk and gait assessment clinical skills examination (CSE), and apply the MPS to a sample of medical students completing a fall risk and gait assessment CSE. Medical students completed an interactive session about fall risk and gait assessment including the Timed Up and Go (TUG) test and completed deliberate practice with 3 patients. Skills were evaluated using an 18-item skills checklist. A panel of clinical experts set the MPS at 82%. Eighty-seven medical students participated. The average score on the checklist was 14.7 of 18 (81.4%.) Although almost all performed the TUG correctly, only 61% met the MPS for the checklist. Our results suggest that a mastery learning approach may better prepare the 39% of students that did not meet MPS to complete a fall risk and gait assessment.

Development and evaluation of a simulation-based mastery learning maintenance of certification course.

BACKGROUND/OBJECTIVES: To develop and evaluate a post-acute care simulation-based mastery learning (SBML) continuing medical education (CME)/maintenance of certification (MOC) procedure course. DESIGN: Pretest-posttest study of the SBML intervention. SETTING: A 2-day post-acute care procedures course. PARTICIPANTS: Sixteen practicing clinicians (5 physicians,11 advanced practice providers). Participants engaged in a skills pretest on knee aspiration/injection, gastrostomy tube removal/replacement, tracheostomy tube exchange, and basic suturing using a checklist created for each procedure. Participants received a didactic on each procedure followed by deliberate practice with feedback. Using the same checklists, participants completed a skills posttest and were required to meet a minimum passing standard (MPS) to obtain CME/MOC credit. MEASUREMENTS: The MPS for each skills checklist was determined by a multidisciplinary panel of 11 experts. Participants completed surveys on procedure self-confidence and a course evaluation. RESULTS: There was statistically significant improvement between pre- and posttests for all four procedures (p < .001). All participants were able to meet or exceed the MPS for each skill during the 2-day course. Participants' self-confidence regarding each procedure improved significantly (p < .001). CONCLUSION: An SBML training course granting CME/MOC credit for post-acute care providers significantly improves performance of knee aspiration/injection, gastrostomy tube removal/replacement, tracheostomy tube exchange, and basic suturing.

Improving cardiology fellow education of right heart catheterization using a simulation based curriculum.

BACKGROUND: Medical procedures are traditionally taught informally at patients' bedside through observation and practice using the adage "see one, do one, teach one." This lack of formalized training can cause trainees to be unprepared to perform procedures independently. Simulation based education (SBE) increases competence, reduces complications, and decreases costs. We developed, implemented, and evaluated the efficacy of a right heart catheterization (RHC) SBE curriculum. METHODS: The RHC curriculum consisted of a pretest, video didactics, deliberate practice, and a posttest. Pre-and posttest skills examinations consisted of a dichotomous 43-item checklist on RHC skills and a 14-item hemodynamic waveform quiz. We enrolled two groups of fellows: 6 first-year, novice cardiology fellows at Northwestern University in their first month of training, and 11 second- and third-year fellows who had completed traditional required, level I training in RHC. We trained the first-year fellows at the beginning of the 2018-2019 year using the SBE curriculum and compared them to the traditionally-trained cardiology fellows who did not complete SBE. RESULTS: The SBE-trained fellows significantly improved RHC skills, hemodynamic knowledge, and confidence from pre- to posttesting. SBE-trained fellows performed similarly to traditionally-trained fellows on simulated RHC skills checklists (88.4% correct vs. 89.2%, p = .84), hemodynamic quizzes (94.0% correct vs. 86.4%, p = .12), and confidence (79.4 vs. 85.9 out of 100, p = .15) despite less clinical experience. CONCLUSIONS: A SBE curriculum for RHC allowed novice cardiology fellows to achieve level I skills and knowledge at the beginning of fellowship and can train cardiology fellows before patient contact.

Use of a simulation-based mastery learning curriculum for neurology residents to improve the identification and management of status epilepticus.

BACKGROUND: Appropriate and timely treatment of status epilepticus (SE) decreases morbidity and mortality. Therefore, skill-based training in the identification and management of SE is crucial for clinicians. OBJECTIVE: The objective of the study was to develop and evaluate the impact of a simulation-based mastery learning (SBML) curriculum to train neurology residents on the identification and management of SE. METHODS: We used pretest-posttest design with a retention test on SE skills for this study. Neurology residents in the second postgraduate year (PGY-2) were eligible to participate in the SE SBML curriculum. Learners completed a baseline-simulated SE skills assessment (pretest) using a 26-item dichotomous skills checklist. Next, they participated in a didactic session about the identification and management of SE, followed by deliberate skills practice. Subsequently, participants completed another skills assessment (posttest) using the same 26-item checklist. All participants were required to meet or exceed a minimum passing standard (MPS) determined by a panel of 14 SE experts using the Mastery Angoff standard setting method. After meeting the MPS at posttest, participants were reassessed during an unannounced in situ simulation session on the medical wards. We compared pretest with posttest simulated SE skills performance and posttest with reassessment in situ performance. RESULTS: The MPS was set at 88% (23/26) checklist items correct. Sixteen neurology residents participated in the intervention. Participant performance improved from a median of 44.23% (Interquartile range (IQR): 34.62-55.77) at pretest to 94.23% (IQR: 92.13-100) at the posttest after SBML (p < .001). There was no significant difference in scores between the posttest and in situ test up to 8 months later (94.23%; IQR: 92.31-100 vs. 92.31%; IQR: 88.46-96.15; p = .13). CONCLUSIONS: Our SBML curriculum significantly improved residents' SE identification and management skills that were largely retained during an unannounced simulated encounter in the hospital setting.

Patient, Caregiver, and Clinician Perceptions of Ventricular Assist Device Self-care Education Inform the Development of a Simulation-based Mastery Learning Curriculum.

BACKGROUND: Patients who undergo ventricular assist device (VAD) implantation and their caregivers must rapidly learn a significant amount of self-care skills and knowledge. OBJECTIVE: The aim of this study was to explore patient, caregiver, VAD coordinator, and physician perspectives and perceptions of existing VAD self-care training to inform development of a simulation-based mastery learning (SBML) curriculum to teach patients and caregivers VAD self-care skills and knowledge. METHODS: We conducted semistructured, in-person interviews with patients with a VAD, their caregivers, VAD coordinators, and physicians (cardiac surgeons, an infectious disease physician, and advanced heart failure cardiologists). We used a 2-cycle team-based iterative inductive approach to coding and analysis. RESULTS: We interviewed 16 patients, 12 caregivers, 7 VAD coordinators, and 11 physicians. Seven major themes were derived from the interviews including (1) identification of critical curricular content, (2) need for standardization and assessment, (3) training modalities, (4) benefits of repetition, (5) piercing it all together, (6) need for refresher training, and (7) provision of training before implant. CONCLUSIONS: Findings from this study suggest that SBML is a natural fit for the high-risk tasks needed to save VAD self-care. The 7 unique training-related themes derived from the qualitative data informed the design and development of a VAD SBML self-care curriculum.

A Mastery Learning Capstone Course to Teach and Assess Components of Three Entrustable Professional Activities to Graduating Medical Students.

PROBLEM: Thirteen measurable Entrustable Professional Activities (EPAs) have been proposed by the Association of American Medical Colleges as a means to operationalize medical school graduates' patient care qualifications. Mastery learning is an effective method for boosting clinical skills, but its applicability to the EPAs remains to be studied. The authors designed this study to evaluate a mastery learning intervention to teach and assess components of 3 of the 13 EPAs in a 4th-year capstone course. INTERVENTION: The course featured mastery learning principles and addressed three EPA-based skills: (a) obtain informed consent, (b) develop a differential diagnosis and write admission orders, and (c) write discharge prescriptions. All students underwent a baseline skills assessment, received feedback, engaged in deliberate practice with actionable feedback, and completed a similar skills-based posttest assessment. Students continued with practice and testing until the minimum passing standards (MPSs) were reached for each posttest. CONTEXT: All medical students at a single medical school (Northwestern University, Feinberg School of Medicine) who matriculated in 2012 and graduated with the class of 2016 participated in a required transition to residency course immediately prior to graduation. OUTCOME: There were 134 students eligible to participate, and 130 (97.0%) completed all curricular requirements and assessments. All 130 medical students who completed the course met or exceeded the MPS for each of the three EPA-based clinical skills. Reliability coefficients for outcome data were uniformly high. Measures for each of the three clinical skills showed statistically significant improvement. LESSONS LEARNED: The capstone course was an effective approach to teach and assess components of three EPA-based clinical skills to mastery learning standards in a 4th-year capstone course. We learned that this approach for implementation is feasible and results in significant improvement in components of EPA skill performance. Next steps will include developing assessments incorporating the mastery model into components of additional EPAs, identifying the best location within the curriculum to insert this content, and expanding the number of assessments as part of a larger assessment system.

Simulation-based education leads to decreased use of fluoroscopy in diagnostic coronary angiography.

OBJECTIVES: The aim of this study is to determine whether simulation-based education (SBE) translates into reduced procedure time, radiation, and contrast use in actual clinical care. BACKGROUND: As a high volume procedure often performed by novice cardiology fellows, diagnostic coronary angiography represents an excellent target for SBE. Reports of SBE in interventional cardiology are limited and there is little understanding of the potential downstream clinical impact of these interventions. METHODS: All diagnostic coronary angiograms performed at a single center between January 1, 2011 and June 30, 2015 were analyzed. Random effects linear regression models were used to compare outcomes between procedures performed by 12 cardiology fellows who underwent simulation-based training and those performed by 20 traditionally trained fellows. RESULTS: Thirty-two cardiology fellows performed 2,783 diagnostic coronary angiograms. Procedures performed by fellows trained with SBE were shorter (mean of 23.98 min vs. 24.94 min, P = 0.034) and were performed with decreased radiation (mean of 56,348 mGycm2 vs. 66,120 mGycm2 , P < 0.001). After controlling for year in training, procedure year, access site, and supervising attending physician, training on the simulator was independently associated with 117 fewer seconds of fluoroscopy time per procedure (P = 0.04). CONCLUSIONS: Diagnostic coronary angiography SBE is associated with decreased use of fluoroscopy in downstream clinical care. SBE may be a useful tool to reduce radiation exposure in the cardiac catheterization laboratory.

Attending Physician Adherence to a 29-Component Central Venous Catheter Bundle Checklist During Simulated Procedures.

OBJECTIVES: Central venous catheter insertions may lead to preventable adverse events. Attending physicians' central venous catheter insertion skills are not assessed routinely. We aimed to compare attending physicians' simulated central venous catheterinsertion performance to published competency standards. DESIGN: Prospective cohort study of attending physicians' simulated internal jugular and subclavian central venous catheter insertion skills versus a historical comparison group of residents who participated in simulation training. SETTING: Fifty-eight Veterans Affairs Medical Centers from February 2014 to December 2014 during a 2-day simulation-based education curriculum and two academic medical centers in Chicago. SUBJECTS: A total of 108 experienced attending physicians and 143 internal medicine and emergency medicine residents. INTERVENTION: None. MEASUREMENTS AND MAIN RESULTS: Using a previously published central venous catheter insertion skills checklist, we compared Veterans Affairs Medical Centers attending physicians' simulated central venous catheter insertion performance to the same simulated performance by internal medicine and emergency medicine residents from two academic centers. Attending physician performance was compared to residents' baseline and posttest (after simulation training) performance. Minimum passing scores were set previously by an expert panel. Attending physicians performed higher on the internal jugular (median, 75.86% items correct; interquartile range, 68.97-86.21) and subclavian (median, 83.00%; interquartile range, 59.00-86.21) assessments compared to residents' internal jugular (median, 37.04% items correct; interquartile range, 22.22-68.97) and subclavian (median, 33.33%; interquartile range, 0.00-70.37; both p < 0.001) baseline assessments. Overall simulated performance was poor because only 12 of 67 attending physicians (17.9%) met or exceeded the minimum passing score for internal jugular central venous catheter insertion and only 11 of 47 (23.4%) met or exceeded the minimum passing score for subclavian central venous catheter insertion. Resident posttest performance after simulation training was significantly higher than attending physician performance (internal jugular: median, 96%; interquartile range, 93.10-100.00; subclavian: median, 100%; interquartile range, 96.00-100.00; both p < 0.001). CONCLUSIONS: This study demonstrates highly variable simulated central venous catheter insertion performance among a national cohort of experienced attending physicians. Hospitals, healthcare systems, and governing bodies should recognize that even experienced physicians require periodic clinical skill assessment and retraining.

Targeting clinical outcomes: Endovascular simulation improves diagnostic coronary angiography skills.

OBJECTIVE: The purpose of this study is to determine the effects of simulation-based medical education (SBME) on the skills required to perform coronary angiography in the cardiac catheterization laboratory. BACKGROUND: Cardiovascular fellows commonly learn invasive procedures on patients. Because this approach is not standardized, it can result in inconsistent skill acquisition through exclusion of concepts and skills. Also, the learning curve varies between trainees yielding variability in skill acquisition. Therefore, coronary angiography skills are an excellent target for SBME in an environment in which direct patient care is not jeopardized. METHODS: From January 2013 to June 2013, 14 cardiovascular fellows entering the cardiac catheterization laboratory at a tertiary care teaching hospital were tested on an endovascular simulator to assess baseline skills. All fellows subsequently underwent didactic teaching and preceptor-lead training on the endovascular simulator. Topics included basic catheterization skills and a review of catheterization laboratory systems. Following training, all fellows underwent a post-training assessment on the endovascular simulator. Paired t tests were used to compare items on the skills checklist and simulator defined variables. RESULTS: Cardiovascular fellows scored significantly higher on a diagnostic coronary angiography skills checklist following SBME using an endovascular simulator. The mean pretest score was 66.6% (SD = 9.7%) compared to 86.0% (SD = 6.3%) following simulator training (P < 0.001). Additional findings include significant reduction in procedure time and use of cine-fluoroscopy at posttest. CONCLUSIONS: SBME significantly improved cardiovascular fellows' performance of simulated coronary angiography skills. Standardized simulation-based education is a valuable adjunct to traditional clinical education for cardiovascular fellows.

Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU Nurses.

OBJECTIVE: This study evaluated the impact of a simulation-based mastery learning (SBML) curriculum on central line maintenance and care among a group of ICU nurses. METHODS: The intervention included 5 tasks: (a) medication administration, (b) injection cap (needleless connector) changes, (c) tubing changes, (d) blood drawing, and (e) dressing changes. All participants underwent a pretest, engaged in deliberate practice with directed feedback, and completed a posttest. We compared pretest and posttest scores and assessed correlations between demographics, self-confidence, and pretest performance. RESULTS: The number of nurses passing each task at pretest varied from 24 of 49 (49%) for dressing changes to 44 of 49 (90%) for tubing changes. At pretest, scores ranged from a median of 0.0% to 73.1%. At posttest, all scores rose to a median of 100.0%. Total years in nursing and ICU nursing had significant, negative correlations with medication administration pretest performance (r = -0.42, P = .003; r = -0.42, P = .003, respectively). CONCLUSION: ICU nurses displayed large variability in their ability to perform central line maintenance tasks. After SBML, there was significant improvement, and all nurses reached a predetermined level of competency.

A comparative assessment and gap analysis of commonly used team rating scales.

BACKGROUND: The purpose of this article was to conduct a gap analysis of important team constructs that may be absent in widely used team assessments. METHODS AND MATERIALS: Two assessment tools with known validity evidence (1) Non-Technical Skills for Surgeons (NOTSS) and (2) the Cannon-Bowers Scale were used to evaluate 11 teams of surgical residents (n = 33) performing simulated laparoscopic hernia repairs. Faculty raters' scores were used to compare the surveys and assess validity and reliability. Raters' detailed observation notes were used to indicate important behavioral constructs that were missing from the team rating scales. RESULTS: When assessing inter-item correlations (reliability) four of five NOTSS' scale items had significant correlations (r = 0.9-1.0, P < 0.05) with the Cannon-Bowers items. While the correlations were only noted for three of six Cannon-Bowers items, in each instance the same four of five NOTSS items correlated with the three Cannon-Bowers items, thus providing further validity evidence for both scales. When evaluating the gap, key emerging themes included the need to focus on critical team errors, individual team member contributions, task performance, and overall team performance. These gaps, plus items from the NOTSS and Cannon-Bowers scales, were incorporated into a new rating scale. CONCLUSIONS: Despite continued evidence of validity and reliability, there were several behavioral constructs that were not represented when using the NOTSS and Cannon-Bowers scales. Critical team errors, individual team member contributions, task performance, and overall team performance appear important in our ability to understand teams and teamwork.

Development of a Simulation-Based Mastery Learning Curriculum for Late Goals of Care Discussions.

INTRODUCTION: Fellows in critical care medicine (CCM) routinely help patients and families navigate complex decisions near the end of life. These "late goals of care" (LGOC) discussions require rigorous skills training and impact patient care. Innovation is needed to ensure that fellow training in leading these discussions is centered on reproducible competency-based standards. The aims of this study were to (1) describe the development of a simulation-based mastery learning (SBML) curriculum for LGOC discussions and (2) set a defensible minimum passing standard (MPS) to ensure uniform skill acquisition among learners. INNOVATION: We developed an SBML curriculum for CCM fellows structured around REMAP, a mnemonic outlining foundational components of effective communication around serious illness. A multidisciplinary expert panel iteratively created an LGOC discussion assessment tool. Pilot testing was completed to refine the checklist, set the MPS, and assess skill acquisition. OUTCOMES: The LGOC discussion assessment tool included an 18-item checklist and 6 scaled items. The tool produced reliable data (k ≥ 0.7 and ICC of ≥ 0.7). Using the Mastery Angoff method, the panel set the MPS at 87%. Ten CCM fellows participated in the pilot study. Performance on the checklist significantly improved from a median score of 52% (IQR 44%-72%) at pretest to 96% (IQR 82%-97%) at post-test (P = 0.005). The number of learners who met the MPS similarly improved from 10% during pre-testing to 70% during post-testing (P = 0.02). COMMENT: We describe the development of a LGOC SBML curriculum for CCM fellows which includes a robust communication skills assessment and the delineation of a defensible MPS.

Effect of Polypectomy Simulation-Based Mastery Learning on Skill Retention Among Practicing Endoscopists.

PURPOSE: Practicing endoscopists frequently perform and teach screening colonoscopies and polypectomies, but there is no standardized method to train and assess physicians who perform polypectomy procedures. The authors created a polypectomy simulation-based mastery learning (SBML) curriculum and hypothesized that completion of the curriculum would lead to immediate improvement in polypectomy skills and skill retention at 6 and 12 months after training. METHOD: The authors performed a pretest-posttest cohort study with endoscopists who completed SBML and were randomized to follow-up at 6 or 12 months from May 2021 to August 2022. Participants underwent SBML training, including a pretest, a video lecture, deliberate practice, and a posttest. All learners were required to meet or exceed a minimum passing standard on a 17-item skills checklist before completing training and were randomized to follow-up at 6 or 12 months. The authors compared simulated polypectomy skills performance on the checklist from pretest to posttest and posttest to 6- or 12-month follow-up test. RESULTS: Twenty-four of 30 eligible participants (80.0%) completed the SBML intervention, and 20 of 24 (83.3%) completed follow-up testing. The minimum passing standard was set at 93% of checklist items correct. The pretest passing rate was 4 of 24 participants (16.7%) compared with 24 of 24 participants (100%) at posttest ( P < .001). There were no significant differences in passing rates from posttest to combined 6- and 12-month posttest in which 18 of 20 participants (90.0%) passed. CONCLUSIONS: Before training and despite years of clinical experience, practicing endoscopists demonstrated poor performance of polypectomy skills. SBML was an effective method for practicing endoscopists to acquire and maintain polypectomy skills during a 6- to 12-month period.

Raising the bar: reassessing standards for procedural competence.

BACKGROUND: How often passing standards for clinical skills examinations should be reexamined is unknown. PURPOSE: The goal is to determine if improved resident performance affects the passing standard imposed by expert judges. METHOD: In 2006, we set a passing standard for a central venous catheter insertion clinical skills examination using an expert panel. From 2007 to 2010, trainee scores improved steadily. In 2010, a new expert panel provided judgments for the examination using 4 consecutive years of resident performance data. Interrater reliabilities and test-retest reliability (stability) were calculated. Passing standards from 2006 and 2010 were compared. RESULTS: Judgments provided were reliable and stable. The new passing standard was 88% for internal jugular and 87% for subclavian central venous catheter insertion compared to 79% for both sites in 2006. CONCLUSIONS: Cumulative performance data influenced experts to set a more stringent minimum passing standard. Standards should be regularly reviewed to ensure they are fair and appropriately rigorous.