Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Pilot testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams.

OBJECTIVE: To test the feasibility of implementing a standardized teamwork training program with full operating room teams in multiple institutions, driven by malpractice insurer support and incentives. BACKGROUND: Failures in intraoperative teamwork are among the leading causes of preventable patient injury and death in surgical patients. Teamwork training, particularly using simulation, can be an effective intervention but is difficult to scale. METHODS: A malpractice insurer convened a collaborative with 4 Harvard-affiliated simulation programs to develop a standardized operating room teamwork training curriculum, including principles of communication, assertiveness, and use of the World Health Organization Surgical Safety Checklist. Participant teams were compensated for lost operative time via malpractice premium discounts, continuing education credits, and compensation for lost wages. The course was delivered through a simulation program involving the management of intraoperative emergency scenarios. Participants were surveyed for their perceptions of the program and of its impact on clinical practice. RESULTS: A total of 221 active operating room staff members participated in the program. Each team contained at least 1 attending surgeon, 1 attending anesthesiologist, and 1 operating room nurse (mean size per team: 7 ± 2 participants). No study dates were cancelled because of lack of attendance. The survey response rate was 99% (218/221). Overall, the vast majority of participants found the scenarios realistic [94% (95% confidence interval: 90.9%, 97.2%)], appropriately challenging [95.4% (92.6%, 98.2%)], relevant to their practice [96.3% (93.8%, 98.8%)], and found the training would help them provide safer patient care [92.6% (89.1%, 96.1%)]. Surgeons reported their greatest personal deficit as communication skills. Operating room nurses and anesthesiologists reported a greater need than surgeons to work on personal assertiveness. CONCLUSIONS: A standardized multicenter team training program involving full operative teams is feasible with high-fidelity simulation and modest compensation for lost time. The vast majority of the multidisciplinary participants believed the course to have had a meaningful impact on their approach to clinical practice.

Simulation-based assessment to identify critical gaps in safe anesthesia resident performance.

BACKGROUND: Valid methods are needed to identify anesthesia resident performance gaps early in training. However, many assessment tools in medicine have not been properly validated. The authors designed and tested use of a behaviorally anchored scale, as part of a multiscenario simulation-based assessment system, to identify high- and low-performing residents with regard to domains of greatest concern to expert anesthesiology faculty. METHODS: An expert faculty panel derived five key behavioral domains of interest by using a Delphi process (1) Synthesizes information to formulate a clear anesthetic plan; (2) Implements a plan based on changing conditions; (3) Demonstrates effective interpersonal and communication skills with patients and staff; (4) Identifies ways to improve performance; and (5) Recognizes own limits. Seven simulation scenarios spanning pre-to-postoperative encounters were used to assess performances of 22 first-year residents and 8 fellows from two institutions. Two of 10 trained faculty raters blinded to trainee program and training level scored each performance independently by using a behaviorally anchored rating scale. Residents, fellows, facilitators, and raters completed surveys. RESULTS: Evidence supporting the reliability and validity of the assessment scores was procured, including a high generalizability coefficient (ρ = 0.81) and expected performance differences between first-year resident and fellow participants. A majority of trainees, facilitators, and raters judged the assessment to be useful, realistic, and representative of critical skills required for safe practice. CONCLUSION: The study provides initial evidence to support the validity of a simulation-based performance assessment system for identifying critical gaps in safe anesthesia resident performance early in training.

Inspiring careers in STEM and healthcare fields through medical simulation embedded in high school science education.

The most effective ways to promote learning and inspire careers related to science, technology, engineering, and mathematics (STEM) remain elusive. To address this gap, we reviewed the literature and designed and implemented a high-fidelity, medical simulation-based Harvard Medical School MEDscience course, which was integrated into high school science classes through collaboration between medical school and K-12 faculty. The design was based largely on the literature on concepts and mechanisms of self-efficacy. A structured telephone survey was conducted with 30 program alumni from the inaugural school who were no longer in high school. Near-term effects, enduring effects, contextual considerations, and diffusion and dissemination were queried. Students reported high incoming attitudes toward STEM education and careers, and these attitudes showed before versus after gains (P < .05). Students in this modest sample overwhelmingly attributed elevated and enduring levels of impact on their interest and confidence in pursuing a science or healthcare-related career to the program. Additionally, 63% subsequently took additional science or health courses, 73% participated in a job or educational experience that was science related during high school, and 97% went on to college. Four of every five program graduates cited a health-related college major, and 83% offered their strongest recommendation of the program to others. Further study and evaluation of simulation-based experiences that capitalize on informal, naturalistic learning and promote self-efficacy are warranted.

Society for Simulation in Healthcare Guidelines for Simulation Training.

BACKGROUND: Simulation has become a staple in the training of healthcare professionals with accumulating evidence on its effectiveness. However, guidelines for optimal methods of simulation training do not currently exist. METHODS: Systematic reviews of the literature on 16 identified key questions were conducted and expert panel consensus recommendations determined using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) methodology. OBJECTIVE: These evidence-based guidelines from the Society for Simulation in Healthcare intend to support healthcare professionals in decisions on the most effective methods for simulation training in healthcare. RESULTS: Twenty recommendations on 16 questions were determined using GRADE. Four expert recommendations were also provided. CONCLUSIONS: The first evidence-based guidelines for simulation training are provided to guide instructors and learners on the most effective use of simulation in healthcare.

Close the GAPS: A Standardized Perioperative Protocol Reduces Breast Reconstruction Implant Infections.

BACKGROUND: Implant-based breast reconstruction (IBBR) is a complex process with significant practice variability. Infections after IBBR are associated with higher rates of readmission, reoperation, and reconstructive failure. To reduce process variability and postoperative infections, the authors implemented an evidence-based, standardized protocol for IBBR. METHODS: The protocol was applied to all patients undergoing IBBR at a single institution from December of 2019 to February of 2021. Intraoperative protocol adherence was recorded, and infection events were considered minor (managed with outpatient antibiotics) or major (managed with readmission or reoperation). A historic control group was retrospectively analyzed for comparison. RESULTS: Sixty-nine patients (120 breasts) in the protocol group were compared with 159 patients (269 breasts) in the retrospective group. No differences were found in demographic characteristics, comorbidities, or type of reconstruction (expander versus implant). Intraoperative protocol adherence was 80.5% (SD, 13.9%). Overall infection rate was significantly lower in the protocol group versus controls (8.7% versus 17.0%; P < 0.05). When dichotomized, protocol patients had a lower rate of both minor (2.9% versus 5.7%; P = 0.99) and major (5.8% versus 11.3%; P = 0.09) infections, although this was not statistically significant. Rate of reconstructive failure secondary to infection was significantly lower in the protocol group (4.4% versus 8.8%; P < 0.05). Among protocol patients, those without infection had higher protocol adherence (81.5% versus 72.2%; P < 0.06), which neared statistical significance. CONCLUSION: A standardized perioperative protocol for IBBR reduces process variability and significantly decreases rate of overall infections and reconstructive failure secondary to infection. CLINICAL QUESTION/LEVEL OF EVIDENCE: Therapeutic, III.

Using immersive healthcare simulation for physiology education: initial experience in high school, college, and graduate school curricula.

In the natural world, learning emerges from the joy of play, experimentation, and inquiry as part of everyday life. However, this kind of informal learning is often difficult to integrate within structured educational curricula. This report describes an educational program that embeds naturalistic learning into formal high school, college, and graduate school science class work. Our experience is based on work with hundreds of high school, college, and graduate students enrolled in traditional science classes in which mannequin simulators were used to teach physiological principles. Specific case scenarios were integrated into the curriculum as problem-solving exercises chosen to accentuate the basic science objectives of the course. This report also highlights the historic and theoretical basis for the use of mannequin simulators as an important physiology education tool and outlines how the authors' experience in healthcare education has been effectively translated to nonclinical student populations. Particular areas of focus include critical-thinking and problem-solving behaviors and student reflections on the impact of the teaching approach.

Primary care clinicians' knowledge and confidence about newborn screening for sickle cell disease: randomized assessment of educational strategies.

OBJECTIVE: In Massachusetts, primary care clinicians receive and act upon hemoglobinopathy newborn screening results. We assessed clinicians' knowledge, confidence, and practices regarding hemoglobinopathy newborn screening, and the effect of mailed educational materials vs interactive seminar on knowledge and confidence. METHODS: A randomized educational intervention trial was performed at 15 community health sites. Practices were randomized to determine the order in which the educational interventions were administered: mailed educational materials first or interactive seminars on the management of hemoglobinopathy newborn screening results first. Clinicians' demographics, knowledge, confidence, and practices were assessed by a survey. Posttests were administered soon after the intervention. RESULTS: Responses came from 85 of 170 eligible providers (50%). Twenty-nine percent of respondents provided both pretests and posttests. In respondents with paired data, knowledge on a 5-point scale improved by 1.4 +/- 0.4 (mean +/- standard error of the mean, p = .003), while self-efficacy on a 16-point scale increased by 1.3 +/- 0.3, p = .002. There were no significant differences between seminar and mailed-materials groups. CONCLUSIONS: Both educational strategies led to modest improvements in knowledge about newborn screening for hemoglobin disorders. Enhancing knowledge and confidence about newborn screening-related tasks may improve clinicians' capacity to act upon newborn screening results for hemoglobinopathies.

Guided Self-Assessment and Action Plans: What Do Residents Need to Succeed?

Resident feedback and program evaluation are essential to ACGME-accredited training programs. We sought to integrate these requirements into our program by creating a systematic process for program improvement focusing on personal learning plans (PLPs). Residents completed a PLP tool every 6 months, followed by an evaluation completed with the program director. Among respondents, 96% reported the PLP process provided useful feedback. A majority found the PLP process useful in developing learning strategies and modeling lifelong learning. The integrated PLP/program improvement process serves as an effective strategy for quickly identifying and capitalizing on both individual and program opportunities for improvement.

Increasing the Effectiveness of "Stop the Bleed" Training Through Stepwise Mastery Learning with Deliberate Practice.

OBJECTIVE: The Stop the Bleed (STB) Campaign supported by the American College of Surgeons Committee on Trauma (ACSCT) and numerous other national organizations aspires to translate lifesaving military successes into reductions in civilian hemorrhagic deaths. While a curricular framework has been described, precise approaches to hands-on training are not specified and training success rates are not yet optimized. Our aim was to test the feasibility and effectiveness of an STB program enhanced by stepwise mastery learning with deliberate practice. STUDY DESIGN: Learners participated in an STB program combining evidence-based training models: the Peyton 4-stage model and simulation-based mastery learning with deliberate practice. ASCTC-certified STB coaches used a 3-point, behaviorally explicit checklist to test 4 skills: apply direct pressure; apply standard and improvised tourniquets; pack a wound. An anonymous questionnaire was administered. SETTING: Simulation Center, Emory University School of Medicine. PARTICIPANTS: College students (N = 30) with no previous trauma training. RESULTS: 100% of participants reached mastery level for all 4 hemorrhage control skills within 4 tries. Additionally, 87% could state a definitive sign of life-threatening bleeding. 76% predicted comfort using a tourniquet in a real-life emergency; among 6 who would be very uncomfortable, 5 nonetheless would definitely recommend the course. CONCLUSIONS: We demonstrate feasibility and increased effectiveness of an STB course using evidence-based procedural training techniques. Adopting these techniques in current STB programs could close the current trainee performance gap and substantially increase the annual number of successfully trained laypersons over current reported levels with no increase required in enrollees, programs, or resources. Future studies should address the challenges of knowledge retention and skill decay, just-in-time innovations, implementation science methods to broaden access, and barriers to responding to real-life crisis events. Surgery education leaders can close performance gaps and make a unique contribution to the Hartford Consensus principle: No one should die from uncontrolled bleeding.

The Impact of a Resident Communication Skills Curriculum on Patients' Experiences of Care.

Despite the high impact of lapses in communication skills on patient care, these skills are often not explicitly taught in residency training programs. We implemented a simulation and web-based curriculum in communication for anesthesia residents and used a patient survey adapted from the Four Habits Coding Scheme to detect changes in patient feedback on residents' communication skills after the curricular intervention. Postintervention mean ratings of residents for the overall survey were higher than preintervention mean ratings. Future research will focus on assessing the curriculum's effectiveness and exploring the generalizability of the survey and curriculum.

Effect of a pediatric hospitalist system on housestaff education and experience.

OBJECTIVE: To determine the effect of a pediatric hospitalist system on housestaff education and experience. SETTING: Free-standing academic children's hospital. Hospitalists were introduced in June 24, 1998, to supervise teaching and care on the general pediatric wards. METHODS: On 2 surveys, housestaff rated their skills, knowledge, and experiences on the wards (experiences survey), and the quality of teaching and supervision by attending physicians (attending survey). Responses before and after the introduction of the hospitalist system were compared using Wilcoxon nonparametric tests. RESULTS: Seventy-six (70%) of the 109 interns and 54 (62%) of the 87 senior residents responded to the experiences survey. Following introduction of the hospitalist system, the interns' mean overall rating of the general pediatrics wards rose from 4.1 to 4.7 on a 5-point Likert scale (P =.01). Their ratings of comparison rotations did not change significantly. Interns' satisfaction with the educational experience (3.2 to 3.5 of a 4-point Likert scale, P<.05), supervision, and quality of life on the pediatrics wards also improved significantly, as did their self-assessments of skills and knowledge related to general pediatrics training. Senior residents' ratings were generally unchanged. Three hundred seventy-one (63%) of the 593 attending physicians' surveys were completed. Compared with nonhospitalist attendings, hospitalists were rated more effective role models (4.7 vs 4.5 points, P<.05) and teachers (4.7 vs 4.4 points, P<.01). They were rated more knowledgeable (4.8 vs 4.5 of 5, P<.001) and accessible (4.7 vs 4.5 points, P<.05), involved housestaff more in the learning process (4.7 vs 4.4 points, P<.05), and gave better feedback (4.5 vs 4.2 points, P<.01). Hospitalists' bedside teaching, however, was rated lower (3.7 vs 4.2 points, P<.001). CONCLUSIONS: Overall, hospitalists were rated significantly higher as educators than were traditional attending physicians. Introduction of a hospitalist system was associated with improved intern experience and quality of life on general pediatrics wards, and with improved self-reported skills and knowledge in general pediatrics.

Notifying emergency department patients of negative test results: pitfalls of passive communication.

OBJECTIVE: Many emergency departments that perform a high volume of group A Streptococcus throat cultures inform patients or parents that unless they are notified of a positive result they can assume that their throat culture result is negative. Thus, positive throat culture results are communicated actively and negative results are communicated passively. We sought to determine the effectiveness and safety of such a system. METHODS: Prospective cohort study of 301 consecutive patients who had group A Streptococcus throat cultures obtained between March 28, 2000, and May 10, 2000, in the emergency department of an urban, academic, tertiary-care children's hospital. Outcomes were determined by telephone surveys of patients or their parents or guardians. The primary outcome was whether respondents had accurate knowledge of the throat culture result. Secondary outcomes included receipt of appropriate treatment instructions, need for another telephone call to the emergency department or primary-care provider to clarify the result or treatment instructions, satisfaction with the communication process, and preference to receive both positive and negative results or just positive results. RESULTS: Fifty-three percent of respondents did not have accurate knowledge of the throat culture result, 5% received inappropriate treatment instructions, 13% sought further clarification, 29% were dissatisfied with the communication process, and 85% preferred to receive both positive and negative results. Parents of patients with positive results were more likely to have accurate knowledge of the result (odds ratio 6.9, 95% confidence interval 1.4-65.7) and to be satisfied with the communication process. CONCLUSIONS: Passive communication of negative throat culture results to patients seen in the emergency department can be unreliable. Active communication of both positive and negative results may improve patient knowledge and satisfaction and ensure appropriate therapy, but in the absence of automated notification systems, would be resource intensive.

Are we closing the gap in faculty development needs for feedback training?

STUDY OBJECTIVE: To determine needs, adequacy, types of resources, and challenges in teaching faculty how to provide feedback to residents. DESIGN: Survey instrument. SETTING: Academic medical center. PARTICIPANTS: Of the 115 anesthesia residency program directors surveyed, 69 responses were received (60% response rate). MEASUREMENTS: Percentages of respondents who indicated categories of need, adequacy, and types of resources for teaching faculty to give feedback to residents were recorded, as were narrative descriptions of challenges confronted by respondents. MAIN RESULTS: While the percentage of programs with faculty development resources has increased from 20.2% in 1999 to 48% today, an overwhelming majority of program directors (90%) feel that faculty require more training in providing feedback to residents. The majority of program directors also want more resources to train their faculty in providing feedback. CONCLUSIONS: While the perceived gap in providing training for faculty in giving feedback to anesthesia residents has narrowed, program director responses suggest a substantial unmet need remains. Innovative new approaches are in order.

Research during pediatric residency training: outcome of a senior resident block rotation.

BACKGROUND: The Pediatric Residency Review Committee requires programs to provide a curriculum that advances residents' knowledge of the basic principles of research. In July 2002, the Boston Combined Residency Program instituted a 3-month career-development block (CDB) rotation. During the rotation residents pursue an academic or clinical project under mentorship by a faculty member. OBJECTIVE: Our objective for this study was to evaluate the outcome of the CDB rotation since it was implemented. METHODS: A survey was administered to 165 residents who completed the CDB rotation. RESULTS: Of 165 residents, 136 (82%) responded to the survey. Of 122 residents who reported the type of project they conducted, 59 (48%) completed a clinical/health services project, 24 (20%) completed a project in education or curriculum development, and 7 (6%) worked in basic science. Thirty-five residents (27%) received funding to support their work. Thirty-five residents (26%) presented at national meetings such as the Pediatric Academic Societies Meeting. Fifteen (11%) residents have had manuscripts accepted for publication, and 22 (16%) additional residents have submitted manuscripts for publication. Factors associated with successful publication included having received funding (odds ratio: 3.37 [95% confidence interval: 1.34-8.42]) and the nature of the research project (odds ratio: 3.55 [95% confidence interval: 1.40-9.04]). The majority of residents (84%) stated that the CDB rotation enhanced residency training. CONCLUSIONS: A dedicated academic rotation that includes protected time, senior faculty mentorship, and program funding, can lead to productive research accomplishments by pediatric residents. Support of academic work during residency training may encourage engagement in a variety of academically oriented activities.

Families' health-related social problems and missed referral opportunities.

OBJECTIVES: The objectives of this study were to characterize (1) families' cumulative burden of health-related social problems regarding access to health care, housing, food security, income security, and intimate partner violence; (2) families' experiences regarding screening and referral for social problems; and (3) parental acceptability of screening and referral. METHODS: We surveyed 205 parents of children who were 0 to 6 years of age and attended 2 urban pediatric clinics for a well-child visit using a self-administered, computer-based questionnaire. The questionnaire included previously validated questions about health-related social problems and new questions about screening and referral in the past 12 months. RESULTS: A total of 205 (79%) of 260 eligible families participated. Eighty-two percent of families reported > or = 1 health-related social problem; 54% experienced problems in > or = 2 social domains. Families experienced similar types and frequencies of problems despite demographic differences between clinics. One third of families reported no screening in any domain in the previous 12 months. Of 205 families, 143 (70%) identified at least 1 need for a referral; 101 (49%) expressed > or = 1 unmet referral need. Of families who reported receiving referrals, 115 referrals were received by 79 families; of the referrals made, 63% (73 of 115) led to contact with the referral agency, and 82% (60 of 73) of the referral agencies were considered helpful. A computer-based system in a pediatrician's office for future screening and referral for health-related social problems was deemed acceptable by 92% of parents. CONCLUSIONS: Urban children and families reported a significant burden of health-related social problems yet infrequent pediatric screening or referral for these problems. Of families who reported receiving referrals, a majority contacted the recommended agencies and found them helpful. This study also demonstrates the feasibility of using a computer-based questionnaire to identify health-related social problems in a routine outpatient clinic setting.

Impact of a hospitalist system on length of stay and cost for children with common conditions.

OBJECTIVE: This study examined mechanisms of efficiency in a managed care hospitalist system on length of stay and total costs for common pediatric conditions. PATIENTS AND METHODS: We conducted a retrospective cohort study (October 1993 to July 1998) of patients in a not-for-profit staff model (HMO 1) and a non-staff-model (HMO 2) managed care organization at a freestanding children's hospital. HMO 1 introduced a hospitalist system for patients in October 1996. Patients were included if they had 1 of 3 common diagnoses: asthma, dehydration, or viral illness. Linear regression models examining length-of-stay-specific costs for prehospitalist and posthospitalist systems were built. Distribution of length of stay for each diagnosis before and after the system change in both study groups was calculated. Interrupted time series analysis tested whether changes in the trends of length of stay and total costs occurred after implementation of the hospitalist system by HMO1 (HMO 2 as comparison group) for all 3 diagnoses combined. RESULTS: A total of 1970 patients with 1 of the 3 study conditions were cared for in HMO 1, and 1001 in HMO 2. After the hospitalist system was introduced in HMO 1, length of stay was reduced by 0.23 days (13%) for asthma and 0.19 days (11%) for dehydration; there was no difference for patients with viral illness. The largest relative reduction in length of stay occurred in patients with a shorter length of stay whose hospitalizations were reduced from 2 days to 1 day. This shift resulted in an average cost-per-case reduction of $105.51 (9.3%) for patients with asthma and $86.22 (7.8%) for patients with dehydration. During the same period, length of stay and total cost rose in HMO 2. CONCLUSIONS: Introduction of a hospitalist system in one health maintenance organization resulted in earlier discharges and reduced costs for children with asthma and dehydration compared with another one, with the largest reductions occurring in reducing some 2-day hospitalizations to 1 day. These findings suggest that hospitalists can increase efficiency and reduce costs for children with common pediatric conditions.