Systematic Development and Validity Evidence for a Checklist to Assess Bed Mobility Skills Among Physical Therapy Students.

INTRODUCTION: Assessments with strong validity evidence are necessary to accurately assess health professions students' performance of clinical skills. The aim of this study was to develop and validate a checklist assessment of physical therapy students' performance of bed mobility skills. METHODS: A checklist was developed using a 4-step process: 1) evidence review and preliminary checklist development, 2) Delphi review to reach consensus on content, 3) pilot testing and checklist editing, 4) final round of Delphi review. Consensus during Delphi review was defined as 100% of participants rating an item "keep as is" and zero comments in Round 1, and >50% of participants rating each item agree/strongly agree in subsequent Delphi rounds. Interrater reliability (IRR) was measured by two raters scoring 32 recorded exam simulations. RESULTS: All 48 items of the checklist reached consensus after three rounds of Delphi review (12 participants in Round 1, 11 participants in Rounds 2-3). IRR was substantial with 88.5% agreement, Cohen's kappa coefficient=0.61, p<0.001, 95% CI [0.56, 0.66]. DISCUSSION: This checklist has potential to be used to assess student readiness to evaluate and train patients in bed mobility tasks for first-time clinical experiences and to serve as a methodological template for future checklist development.

Twelve tips for developing simulation-based mastery learning clinical skills checklists.

Simulation-based mastery learning is a powerful educational paradigm that leads to high levels of performance through a combination of strict standards, deliberate practice, formative feedback, and rigorous assessment. Successful mastery learning curricula often require well-designed checklists that produce reliable data that contribute to valid decisions. The following twelve tips are intended to help educators create defensible and effective clinical skills checklists for use in mastery learning curricula. These tips focus on defining the scope of a checklist using established principles of curriculum development, crafting the checklist based on a literature review and expert input, revising and testing the checklist, and recruiting judges to set a minimum passing standard. While this article has a particular focus on mastery learning, with the exception of the tips related to standard setting, the general principles discussed apply to the development of any clinical skills checklist.

Powerful medical education improves health care quality and return on investment.

INTRODUCTION: Powerful medical education (PME) involves the use of new technologies informed by the science of expertise that are embedded in laboratories and organizations that value evidence-based education and support innovation. This contrasts with traditional medical education that relies on a dated apprenticeship model that yields uneven results. PME involves an amalgam of features, conditions and assumptions, and contextual variables that comprise an approach to developing clinical competence grounded in education impact metrics including efficiency and cost-effectiveness. METHODS: This article is a narrative review based on SANRA criteria and informed by realist review principles. The review addresses the PME model with an emphasis on mastery learning and deliberate practice principles drawn from the new science of expertise. Pub Med, Scopus, and Web of Science search terms include medical education, the science of expertise, mastery learning, translational outcomes, cost effectiveness, and return on investment. Literature coverage is comprehensive with selective citations. RESULTS: PME is described as an integrated set of twelve features embedded in a group of seven conditions and assumptions and four context variables. PME is illustrated via case examples that demonstrate improved ventilator patient management learning outcomes compared to traditional clinical education and mastery learning of breaking bad news communication skills. Evidence also shows that PME of physicians and other health care providers can have translational, downstream effects on patient care practices, patient outcomes, and return on investment. Several translational health care quality improvements that derive from PME include reduced infections; better communication among physicians, patients, and families; exceptional birth outcomes; more effective patient education; and return on investment. CONCLUSIONS: The article concludes with challenges to hospitals, health systems, and medical education organizations that are responsible for producing physicians who are expected to deliver safe, effective, and cost-conscious health care.

Tomographic Plane Visualization (ToPlaV): a Tool to Enhance Echocardiographic Training.

Visual learning is an important part of echocardiographic training. Our aim is to describe and evaluate a visual teaching tool, tomographic plane visualization (ToPlaV) as an adjunct to skills training in pediatric echocardiography image acquisition. This tool incorporates learning theory by applying psychomotor skills that closely emulate the skills used in echocardiography. ToPlaV was used as part of a transthoracic bootcamp for first year cardiology fellows. A qualitative survey was given to trainees to evaluate their perceptions of its usefulness. There was universal agreement among fellow trainees that ToPlaV is a useful training tool. ToPlaV is a simple, low cost, education tool which can complement a simulator and live models. We propose that ToPlaV should be incorporated into early training in echocardiography skills for pediatric cardiology fellows.

Simulation Based Mastery Learning of Transesophageal Echocardiography.

Transesophageal echocardiography (TEE) education is part of pediatric cardiology fellow training. Simulation-based mastery learning (SBML) is an efficient and valuable education experience. The aim of this project was to equip trainees with the basic knowledge and skill required to perform a pediatric TEE. The secondary aim was to assess the utility of using SBML for pediatric TEE training. The target group is trainees from pediatric cardiology and cardiac anesthesia who participated in a TEE bootcamp. A baseline knowledge pretest was obtained. The knowledge session consisted of preparation via reading material, viewing recorded lectures and completing an iterative multiple-choice examination, which was repeated until a minimum passing score of 90% was achieved. The skills session involved a review of TEE probe manipulation and image acquisition, followed by rapid cycle deliberate practice using simulation to acquire TEE skills at 3 levels, advancing in complexity from level 1 to level 3. Eight individuals (7 pediatric cardiology fellows at varying training levels and one anesthesia attending) participated in the TEE bootcamp. All reached a minimum knowledge post test score of at least 90% before the skills session. All subjects reached mastery in TEE probe manipulation. All reached mastery in image acquisition for the skill level that they attempted (level 1-8/8, level 2-8/8, level 3-4/4, with 4 participants not attempting level 3). A TEE bootcamp using SBML is a powerful medical education strategy. SBML is a rigorous approach that can be used to achieve high and uniform TEE learning outcomes among trainees of different training levels and backgrounds.

Communicating Diagnostic Uncertainty at Emergency Department Discharge: A Simulation-Based Mastery Learning Randomized Trial.

PURPOSE: There are no standardized approaches for communicating with patients discharged from the emergency department with diagnostic uncertainty. This trial tested efficacy of the Uncertainty Communication Education Module, a simulation-based mastery learning curriculum designed to establish competency in communicating diagnostic uncertainty. METHOD: Resident physicians at 2 sites participated in a 2-arm waitlist randomized controlled trial from September 2019 to June 2020. After baseline (T1) assessment of all participants via a standardized patient encounter using the Uncertainty Communication Checklist (UCC), immediate access physicians received training in the Uncertainty Communication Education Module, which included immediate feedback, online educational modules, a smartphone-based application, and telehealth deliberate practice with standardized patients. All physicians were retested 16-19 weeks later (T2) via in-person standardized patient encounters; delayed access physicians then received the intervention. A final test of all physicians occurred 11-15 weeks after T2 (T3). The primary outcome measured the percentage of physicians in the immediate versus delayed access groups meeting or exceeding the UCC minimum passing standard at T2. RESULTS: Overall, 109 physicians were randomized, with mean age 29 years (range 25-46). The majority were male (n = 69, 63%), non-Hispanic/Latino (n = 99, 91%), and White (n = 78, 72%). At T2, when only immediate access participants had received the curriculum, immediate access physicians demonstrated increased mastery (n = 29, 52.7%) compared with delayed access physicians (n = 2, 3.7%, P < .001; estimated adjusted odds ratio of mastery for the immediate access participants, 31.1 [95% CI, 6.8-143.1]). There were no significant differences when adjusting for training site or stage of training. CONCLUSIONS: The Uncertainty Communication Education Module significantly increased mastery in communicating diagnostic uncertainty at the first postintervention test among emergency physicians in standardized patient encounters. Further work should assess the impact of clinical implementation of these communication skills.

Simulation-based training improves polypectomy skills among practicing endoscopists.

Background and study aims Practicing endoscopists have variable polypectomy skills during colonoscopy and limited training opportunities for improvement. Simulation-based training enhances procedural skill, but its impact on polypectomy is unclear. We developed a simulation-based polypectomy intervention to improve polypectomy competency. Methods All faculty endoscopists at our tertiary care center who perform colonoscopy with polypectomy were recruited for a simulation-based intervention assessing sessile and stalked polypectomy. Endoscopists removed five polyps in a simulation environment at pretest followed by a training intervention including a video, practice, and one-on-one feedback. Within 1-4 weeks, endoscopists removed five new simulated polyps at post-test. We used the Direct Observation of Polypectomy Skills (DOPyS) checklist for assessment, evaluating individual polypectomy skills, and global competency (scale: 1-4). Competency was defined as an average global competency score of ≥ 3. Results 83 % (29/35) of eligible endoscopists participated and 95 % (276/290) of planned polypectomies were completed. Only 17 % (5/29) of endoscopists had average global competency scores that were competent at pretest compared with 52 % (15/29) at post-test ( P  = 0.01). Of all completed polypectomies, the competent polypectomy rate significantly improved from pretest to post-test (55 % vs. 71 %; P  < 0.01). This improvement was significant for sessile polypectomy (37 % vs. 65 %; P  < 0.01) but not for stalked polypectomy (82 % vs. 80 %; P  = 0.70). Conclusions Simulation-based training improved polypectomy skills among practicing endoscopists. Further studies are needed to assess the translation of simulation-based education to clinical practice.

Ethical imperative of psychological safety in healthcare: in response to the Manifesto for healthcare simulation practice.

Psychological safety is valued in other high-risk industries as an essential element to ensure safety. Yet, in healthcare, psychological safety is not mandatorily measured, quantified, or reported as an independent measure of safety. All members of the healthcare team's voice and safety are important. Calls for personal, physical or patient safety should never be disregarded or met with retaliation.

Psychometric Validation of Central Venous Catheter Insertion Mastery Learning Checklist Data and Decisions.

INTRODUCTION: Resident physicians are expected to acquire competence at central venous catheter (CVC) insertion to a mastery standard. Valid competence decisions about resident physicians' CVC performance rely on reliable data and rigorous achievement standards. This study used data from 3 CVC simulation-based mastery learning studies involving internal medicine (IM) and emergency medicine (EM) residents to address 2 questions: What is the effectiveness of a CVC mastery learning education intervention? Are minimum passing standards (MPSs) set by faculty supported by item response theory (IRT) analyses? METHODS: Pretraining and posttraining skills checklist data were drawn from 3 simulation-based mastery learning research reports about CVC internal jugular (IJ) and subclavian (SC) insertion skill acquisition. Residents were required to meet or exceed a posttest skills MPS. Generalized linear mixed effect models compared checklist performance from pre to postintervention. Minimum passing standards were determined by Angoff and Hofstee standard setting methods. Item response theory models were used for cut-score evaluation. RESULTS: Internal medicine and EM residents improved significantly on every IJ and SC checklist item after mastery learning. Item response theory analyses support the IJ and SC MPSs. CONCLUSIONS: Mastery learning is an effective education intervention to achieve clinical skill acquisition among IM and EM residents. Item response theory analyses reveal desirable measurement properties for the MPSs previously set by expert faculty panels. Item response theory analysis is useful for evaluating standards for mastery learning interventions. The CVC mastery learning curriculum, reliable outcome data, and high achievement standards together contribute to reaching valid decisions about the competence of resident physicians to perform the clinical procedure.

Improving cardiology fellow education of right heart catheterization using a simulation based curriculum.

BACKGROUND: Medical procedures are traditionally taught informally at patients' bedside through observation and practice using the adage "see one, do one, teach one." This lack of formalized training can cause trainees to be unprepared to perform procedures independently. Simulation based education (SBE) increases competence, reduces complications, and decreases costs. We developed, implemented, and evaluated the efficacy of a right heart catheterization (RHC) SBE curriculum. METHODS: The RHC curriculum consisted of a pretest, video didactics, deliberate practice, and a posttest. Pre-and posttest skills examinations consisted of a dichotomous 43-item checklist on RHC skills and a 14-item hemodynamic waveform quiz. We enrolled two groups of fellows: 6 first-year, novice cardiology fellows at Northwestern University in their first month of training, and 11 second- and third-year fellows who had completed traditional required, level I training in RHC. We trained the first-year fellows at the beginning of the 2018-2019 year using the SBE curriculum and compared them to the traditionally-trained cardiology fellows who did not complete SBE. RESULTS: The SBE-trained fellows significantly improved RHC skills, hemodynamic knowledge, and confidence from pre- to posttesting. SBE-trained fellows performed similarly to traditionally-trained fellows on simulated RHC skills checklists (88.4% correct vs. 89.2%, p = .84), hemodynamic quizzes (94.0% correct vs. 86.4%, p = .12), and confidence (79.4 vs. 85.9 out of 100, p = .15) despite less clinical experience. CONCLUSIONS: A SBE curriculum for RHC allowed novice cardiology fellows to achieve level I skills and knowledge at the beginning of fellowship and can train cardiology fellows before patient contact.

Are USMLE Scores Valid Measures for Chief Resident Selection?

BACKGROUND: The US Medical Licensing Examination (USMLE) Step 1 and Step 2 scores are often used to inform a variety of secondary medical career decisions, such as residency selection, despite the lack of validity evidence supporting their use in these contexts. OBJECTIVE: We compared USMLE scores between non-chief residents (non-CRs) and chief residents (CRs), selected based on performance during training, at a US academic medical center that sponsors a variety of graduate medical education programs. METHODS: This was a retrospective cohort study of residents' USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores from 2015 to 2020. The authors used archived data to compare USMLE Step 1 and Step 2 CK scores between non-CR residents in each of the eligible programs and their CRs during the 6-year study period. RESULTS: Thirteen programs enrolled a total of 1334 non-CRs and 211 CRs over the study period. There were no significant differences overall between non-CRs and CRs average USMLE Step 1 (239.81 ± 14.35 versus 240.86 ± 14.31; P = .32) or Step 2 scores (251.06 ± 13.80 versus 252.51 ± 14.21; P = .16). CONCLUSIONS: There was no link between USMLE Step 1 and Step 2 CK scores and CR selection across multiple clinical specialties over a 6-year period. Reliance on USMLE Step 1 and 2 scores to predict success in residency as measured by CR selection is not recommended.

Promoting Readiness for Residency: Embedding Simulation-Based Mastery Learning for Breaking Bad News Into the Medicine Subinternship.

PURPOSE: It is challenging to add rigorous, competency-based communication skills training to existing clerkship structures. The authors embedded a simulation-based mastery learning (SBML) curriculum into a medicine subinternship to demonstrate feasibility and determine the impact on the foundational skill of breaking bad news (BBN). METHOD: All fourth-year students enrolled in a medicine subinternship at Northwestern University Feinberg School of Medicine from September 2017 through August 2018 were expected to complete a BBN SBML curriculum. First, students completed a pretest with a standardized patient using a previously developed BBN assessment tool. Learners then participated in a 4-hour BBN skills workshop with didactic instruction, focused feedback, and deliberate practice with simulated patients. Students were required to meet or exceed a predetermined minimum passing standard (MPS) at posttest. The authors compared pretest and posttest scores to evaluate the effect of the intervention. Participant demographic characteristics and course evaluations were also collected. RESULTS: Eighty-five students were eligible for the study, and 79 (93%) completed all components. Although 55/79 (70%) reported having personally delivered serious news to actual patients, baseline performance was poor. Students' overall checklist performance significantly improved from a mean of 65.0% (SD = 16.2%) items correct to 94.2% (SD = 5.9%; P < .001) correct. There was also statistically significant improvement in scaled items assessing quality of communication, and all students achieved the MPS at mastery posttest. All students stated they would recommend the workshop to colleagues. CONCLUSIONS: It is feasible to embed SBML into a required clerkship. In the context of this study, rigorous SBML resulted in uniformly high levels of skill acquisition, documented competency, and was positively received by learners.

Setting a Minimum Passing Standard for the Uncertainty Communication Checklist Through Patient and Physician Engagement.

BACKGROUND: Historically, medically trained experts have served as judges to establish a minimum passing standard (MPS) for mastery learning. As mastery learning expands from procedure-based skills to patient-centered domains, such as communication, there is an opportunity to incorporate patients as judges in setting the MPS. OBJECTIVE: We described our process of incorporating patients as judges to set the MPS and compared the MPS set by patients and emergency medicine residency program directors (PDs). METHODS: Patient and physician panels were convened to determine an MPS for a 21-item Uncertainty Communication Checklist. The MPS for both panels were independently calculated using the Mastery Angoff method. Mean scores on individual checklist items with corresponding 95% confidence intervals were also calculated for both panels and differences analyzed using a t test. RESULTS: Of 240 eligible patients and 42 eligible PDs, 25 patients and 13 PDs (26% and 65% cooperation rates, respectively) completed MPS-setting procedures. The patient-generated MPS was 84.0% (range 45.2-96.2, SD 10.2) and the physician-generated MPS was 88.2% (range 79.7-98.1, SD 5.5). The overall MPS, calculated as an average of these 2 results, was 86.1% (range 45.2-98.1, SD 9.0), or 19 of 21 checklist items. CONCLUSIONS: Patients are able to serve as judges to establish an MPS using the Mastery Angoff method for a task performed by resident physicians. The patient-established MPS was nearly identical to that generated by a panel of residency PDs, indicating similar expectations of proficiency for residents to achieve skill "mastery."

Development of the Uncertainty Communication Checklist: A Patient-Centered Approach to Patient Discharge From the Emergency Department.

Clear communication with patients upon emergency department (ED) discharge is important for patient safety during the transition to outpatient care. Over one-third of patients are discharged from the ED with diagnostic uncertainty, yet there is no established approach for effective discharge communication in this scenario. From 2017 to 2019, the authors developed the Uncertainty Communication Checklist for use in simulation-based training and assessment of emergency physician communication skills when discharging patients with diagnostic uncertainty. This development process followed the established 12-step Checklist Development Checklist framework and integrated patient feedback into 6 of the 12 steps. Patient input was included as it has potential to improve patient-centeredness of checklists related to assessment of clinical performance. Focus group patient participants from 2 clinical sites were included: Thomas Jefferson University Hospital, Philadelphia, PA, and Northwestern University Hospital, Chicago, Illinois.The authors developed a preliminary instrument based on existing checklists, clinical experience, literature review, and input from an expert panel comprising health care professionals and patient advocates. They then refined the instrument based on feedback from 2 waves of patient focus groups, resulting in a final 21-item checklist. The checklist items assess if uncertainty was addressed in each step of the discharge communication, including the following major categories: introduction, test results/ED summary, no/uncertain diagnosis, next steps/follow-up, home care, reasons to return, and general communication skills. Patient input influenced both what items were included and the wording of items in the final checklist. This patient-centered, systematic approach to checklist development is built upon the rigor of the Checklist Development Checklist and provides an illustration of how to integrate patient feedback into the design of assessment tools when appropriate.

Simulation-based mastery learning compared to standard education for discussing diagnostic uncertainty with patients in the emergency department: a randomized controlled trial.

BACKGROUND: Diagnostic uncertainty occurs frequently in emergency medical care, with more than one-third of patients leaving the emergency department (ED) without a clear diagnosis. Despite this frequency, ED providers are not adequately trained on how to discuss diagnostic uncertainty with these patients, who often leave the ED confused and concerned. To address this training need, we developed the Uncertainty Communication Education Module (UCEM) to teach physicians how to discuss diagnostic uncertainty. The purpose of the study is to evaluate the effectiveness of the UCEM in improving physician communications. METHODS: The trial is a multicenter, two-arm randomized controlled trial designed to teach communication skills using simulation-based mastery learning (SBML). Resident emergency physicians from two training programs will be randomly assigned to immediate or delayed receipt of the two-part UCEM intervention after completing a baseline standardized patient encounter. The two UCEM components are: 1) a web-based interactive module, and 2) a smart-phone-based game. Both formats teach and reinforce communication skills for patient cases involving diagnostic uncertainty. Following baseline testing, participants in the immediate intervention arm will complete a remote deliberate practice session via a video platform and subsequently return for a second study visit to assess if they have achieved mastery. Participants in the delayed intervention arm will receive access to UCEM and remote deliberate practice after the second study visit. The primary outcome of interest is the proportion of residents in the immediate intervention arm who achieve mastery at the second study visit. DISCUSSION: Patients' understanding of the care they received has implications for care quality, safety, and patient satisfaction, especially when they are discharged without a definitive diagnosis. Developing a patient-centered diagnostic uncertainty communication strategy will improve safety of acute care discharges. Although use of SBML is a resource intensive educational approach, this trial has been deliberately designed to have a low-resource, scalable intervention that would allow for widespread dissemination and uptake. TRIAL REGISTRATION: The trial was registered at clinicaltrials.gov (NCT04021771). Registration date: July 16, 2019.