Developing a generalizable pediatric ECMO emergency checklist for clinical specialist: Progress and challenges.

INTRODUCTION: Extracorporeal membrane oxygenation (ECMO) emergencies require skilled clinical specialist (CS) who manage ECMO circuits. While tools for assessing CS skills have been published, there is significant variation in protocols and circuit design. This study aims to further develop these checklists to produce a generalizable ECMO skill assessment with adequate validity evidence to support its use as a summative evaluation tool. METHODS: An initial survey determined variation in ECMO circuit components and configurations, and the original checklists and simulations were altered through a modified Delphi process. The finalized checklist and simulation were then assessed for validity and reliability. Three trained raters assessed ten simulations from five subjects at two different institutions using two circuit designs. Data analysis was conducted using a fully crossed subject x rater x circuit generalizability (G) and decision (D) study. RESULTS: The G-study coefficient was 0 with 0% variance across subject and circuit. The greatest variance was among raters (28.7%). Significant variance was also associated with the subject and pump type relationship (27%). CONCLUSION: Despite the rigorous process used to modify the assessment, generalizability was poor. Lack of familiarity with center-specific circuit design played a key role. Future endeavors in ECMO skill assessment should focus either on developing and validating site-specific tools or standardizing circuit designs.

Society for Simulation in Healthcare Guidelines for Simulation Training.

BACKGROUND: Simulation has become a staple in the training of healthcare professionals with accumulating evidence on its effectiveness. However, guidelines for optimal methods of simulation training do not currently exist. METHODS: Systematic reviews of the literature on 16 identified key questions were conducted and expert panel consensus recommendations determined using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) methodology. OBJECTIVE: These evidence-based guidelines from the Society for Simulation in Healthcare intend to support healthcare professionals in decisions on the most effective methods for simulation training in healthcare. RESULTS: Twenty recommendations on 16 questions were determined using GRADE. Four expert recommendations were also provided. CONCLUSIONS: The first evidence-based guidelines for simulation training are provided to guide instructors and learners on the most effective use of simulation in healthcare.

Educational and Patient Care Impacts of In Situ Simulation in Healthcare: A Systematic Review.

ABSTRACT: This systematic review was performed to assess the effectiveness of in situ simulation education. We searched databases including MEDLINE and Embase for studies comparing in situ simulation with other educational approaches. Two reviewers screened articles and extracted information. Sixty-two articles met inclusion criteria, of which 24 were synthesized quantitatively using random effects meta-analysis. When compared with current educational practices alone, the addition of in situ simulation to these practices was associated with small improvements in clinical outcomes, including mortality [odds ratio, 0.66; 95% confidence interval (CI), 0.55 to 0.78], care metrics (standardized mean difference, -0.34; 95% CI, -0.45 to -0.21), and nontechnical skills (standardized mean difference, -0.52; 95% CI, -0.99 to -0.05). Comparisons between in situ and traditional simulation showed mixed learner preference and knowledge improvement between groups, while technical skills showed improvement attributable to in situ simulation. In summary, available evidence suggests that adding in situ simulation to current educational practices may improve patient mortality and morbidity.

Setting an Agenda: Results of a Consensus Process on Research Directions in Distance Simulation.

BACKGROUND: The COVID-19 pandemic forced rapid implementation and refinement of distance simulation methodologies in which participants and/or facilitators are not physically colocated. A review of the distance simulation literature showed that heterogeneity in many areas (including nomenclature, methodology, and outcomes) limited the ability to identify best practice. In April 2020, the Healthcare Distance Simulation Collaboration was formed with the goal of addressing these issues. The aim of this study was to identify future research priorities in the field of distance simulation using data derived from this summit. METHODS: This study analyzed textual data gathered during the consensus process conducted at the inaugural Healthcare Distance Simulation Summit to explore participant perceptions of the most pressing research questions regarding distance simulation. Participants discussed education and patient safety standards, simulation facilitators and barriers, and research priorities. Data were qualitatively analyzed using an explicitly constructivist thematic analysis approach, resulting in the creation of a theoretical framework. RESULTS: Our sample included 302 participants who represented 29 countries. We identified 42 codes clustered within 4 themes concerning key areas in which further research into distance simulation is needed: (1) safety and acceptability, (2) educational/foundational considerations, (3) impact, and (4) areas of ongoing exploration. Within each theme, pertinent research questions were identified and categorized. CONCLUSIONS: Distance simulation presents several challenges and opportunities. Research around best practices, including educational foundation and psychological safety, are especially important as is the need to determine outcomes and long-term effects of this emerging field.

The Concise Assessment of Leader Management Tool: Evaluation of Healthcare Provider Leadership During Real-Life Pediatric Emergencies.

INTRODUCTION: Resuscitation events in pediatric critical and emergency care are high risk, and strong leadership is an important component of an effective response. The Concise Assessment of Leadership Management (CALM) tool, designed to assess the strength of leadership skills during pediatric crises, has shown promising validity and reliability in simulated settings. The objective of this study was to generate further validity and reliability evidence for the CALM by applying it to real-life emergency events. METHODS: A prospective, video-based study was conducted in an academic pediatric emergency department. Three reviewers independently applied the CALM tool to the assessment of pediatric emergency department physicians as they led both a cardiac arrest and a sepsis event. Time to critical event (epinephrine, fluid, and antibiotic administration) was collected via video review. Based on Kane's framework, we conducted fully crossed, person × event × rater generalizability (G) and decision (D) studies. Interrater reliability was calculated using Gwet AC 2 and intraclass correlation coefficients. Time to critical events was correlated with CALM scores using Spearman coefficient. RESULTS: Nine team leaders were assessed in their leadership of 2 resuscitations each. The G coefficient was 0.68, with 26% subject variance, 20% rater variance, and no case variance. Thirty-three percent of the variance (33%) was attributed to third-order interactions and unknown factors. Gwet AC 2 was 0.3 and intraclass correlation was 0.58. The CALM score and time to epinephrine correlated at -0.79 ( P = 0.01). The CALM score and time to fluid administration correlated at -0.181 ( P = 0.64). CONCLUSIONS: This study provides additional validity evidence for the CALM tool's use in this context if used with multiple raters, aligning with data from the previous simulation-based CALM validity study. Further development may improve reliability. It also serves as an exemplar of the rigors of conducting validity work within medical simulation.

Development and Validation of a Point-of-Care-Ultrasound Image Quality Assessment Tool: The POCUS IQ Scale.

OBJECTIVES: We aimed to develop a standardized scoring tool to measure point-of-care ultrasound (POCUS) image quality and to determine validity evidence for its use to assess lung ultrasound image quality. METHODS: The POCUS Image Quality (POCUS IQ) scale was developed by POCUS-trained physicians to assess sonographers' image acquisition skills by evaluating image quality for any POCUS application. The scale was piloted using lung images of healthy standardized patients acquired by three expert sonographers compared to three novices before and after training. All images (experts, novices pre-training, novices post-training) were scored on the POCUS IQ scale by three blinded POCUS-trained physicians. Reliability was assessed with fully-crossed generalizability and decision studies. Validity was assessed using Messick's framework. RESULTS: Content validity was supported by the tool's development process of literature review, expert consensus, and pilot testing. Response process was supported by reviewer training and the blinded scoring process. Relation to other variables was supported by scores relating to sonographer experience: median expert score = 10.5/14 (IQR: 4), median novice pre-training score = 6/14 (IQR: 2.25), and novices' improvement after training (median post-training score = 12/14, IQR: 3.25). Internal structure was supported by internal consistency data (coefficient alpha = 0.84, omega coefficient = 0.91) and the generalizability study showing the main contributor to score variability was the sonographer (51%). The G-coefficient was 0.89, suggesting very good internal structure, however, Gwet's AC2  was 0.5, indicating moderate interrater reliability. The D study projected a minimum of 1 reviewer and 2 patients are needed for good psychometric reliability. CONCLUSIONS: The POCUS scale has good preliminary validity evidence as an assessment tool for lung POCUS image acquisition skills. Further studies are needed to demonstrate its utility for other POCUS applications and as a feedback tool for POCUS learners.

Preparing and Presenting Validation Studies: A Guide for the Perplexed.

SUMMARY STATEMENT: Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

From Concept to Publication: Effectiveness of the International Network for Simulation-Based Pediatric Innovation, Research, and Education Project Development Process at Generating Simulation Scholarship.

BACKGROUND: As simulation matures, it is critical to develop pathways for researchers. A recent analysis, however, demonstrates a low conversion rate between abstract and peer-reviewed journal publication in our field. The International Network for Simulation-based Pediatric Innovation, Research, and Education has used the ALERT Presentation process for the past decade as a means of accelerating research. In this study, we analyze the scholarly products attributable to ALERT Presentations. METHODS: Surveys were distributed to all International Network for Simulation-based Pediatric Innovation, Research, and Education Advanced Look Exploratory Research Template (ALERT) Presentation first authors from January 2011 through January 2020. Presenters were asked to provide information on abstracts, grants, journal publications, and book chapters related to their ALERT Presentation, as well as basic demographic information. A structured literature search was conducted for those ALERT Presentations whose authors did not return a survey. The resulting database was descriptively analyzed, and statistical correlations between demographic variables and scholarship were examined. RESULTS: One hundred sixty-five new ALERT presentations were presented over 10 years. We identified 361 associated scholarly works (170 conference abstracts, 125 peer-reviewed journal publications, 65 grants, and 1 book chapter). Sixty-one percent (101 of 165) of ALERT Presentations produced at least 1 item of scholarship, and 59% (34 of 58) of ALERT Presentations that resulted in at least 1 abstract also led to at least 1 peer-reviewed journal article. Presenter gender was associated with likelihood of journal publication. CONCLUSIONS: The ALERT Presentation process is an effective approach for facilitating the development of projects that result in disseminated scholarship. Wider adoption may benefit other simulation and education research networks.

A Decade Later-Progress and Next Steps for Pediatric Simulation Research.

SUMMARY STATEMENT: A decade ago, at the time of formation of the International Network for Pediatric Simulation-based Innovation, Research, and Education, the group embarked on a consensus building exercise. The goal was to forecast the facilitators and barriers to growth and maturity of science in the field of pediatric simulation-based research. This exercise produced 6 domains critical to progress in the field: (1) prioritization, (2) research methodology and outcomes, (3) academic collaboration, (4) integration/implementation/sustainability, (5) technology, and (6) resources/support/advocacy. This article reflects on and summarizes a decade of progress in the field of pediatric simulation research and suggests next steps in each domain as we look forward, including lessons learned by our collaborative grass roots network that can be used to accelerate research efforts in other domains within healthcare simulation science.

Simulathon 2020: Integrating Simulation Period Prevalence Methodology Into the COVID-19 Disaster Management Cycle in India.

SUMMARY STATEMENT: The disaster management cycle is an accepted model that encompasses preparation for and recovery from large-scale disasters. Over the past decade, India's Pediatric Simulation Training and Research Society has developed a national-scale simulation delivery platform, termed the Simulathon , with a period prevalence methodology that integrates with core aspects of this model. As an exemplar of the effectiveness of this approach, we describe the development, implementation, and outcomes of the 2020 Simulathon, conducted from April 20 to May 20 in response to the nascent COVID-19 pandemic disaster. We conclude by discussing how aspects of the COVID-19 Simulathon enabled us to address key aspects of the disaster management cycle, as well as challenges that we encountered. We present a roadmap by which other simulation programs in low- and middle-income countries could enact a similar process.

Attention-Deficit/Hyperactivity Disorder Practice Patterns: A Survey of Kentucky Pediatric Providers.

OBJECTIVE: Kentucky ranks among the highest in the nation for attention-deficit/hyperactivity disorder (ADHD) prevalence in children aged 4 to 17 years. In 2011, the American Academy of Pediatrics (AAP) released a clinical practice guideline based on the DSM-IV. A guideline revision based on the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM-5) was released in October 2019. In this study, we assess and describe pediatric providers' ADHD practices using the 2011 guideline and DSM-5 diagnostic criteria. METHODS: This was a cross-sectional, survey-based descriptive study. Kentucky Chapter of the AAP (KY AAP) members were anonymously surveyed. The results were examined for trends in routine practice. RESULTS: Fifty-eight general pediatricians and pediatric residents responded to the survey, yielding a 38% (58/154) response rate. Among respondents performing routine diagnosis of ADHD (N = 51), 73% (37/51) used DSM-5 criteria. Most providers usually or always initially assessed for coexisting behavioral conditions (96%; 49/51), developmental conditions (78%; 39/51), and adverse childhood experiences (73%; 37/51). Among respondents performing routine management of ADHD (N = 55), only 11% (6/55) of respondents indicated that they titrated stimulant medications every 3 to 7 days. After initiation of medication, 78% of providers scheduled a follow-up visit within 2 to 4 weeks. During subsequent visits, only half indicated discussing behavioral interventions, screening for coexisting conditions, and reviewing follow-up teacher-rated ADHD scales. CONCLUSION: Pediatricians in the KY AAP adhere to the DSM-5 criteria for diagnosing ADHD. Pediatric providers' practices would benefit from education in improvements in pharmacotherapy titration, surveillance of coexisting conditions associated with ADHD, discussion of psychosocial interventions, and school support strategies.

Standardized Vaccine-Hesitant Patients in the Assessment of the Effectiveness of Vaccine Communication Training.

OBJECTIVES: To determine if training residents in a structured communication method elicits specific behaviors in a laboratory model of interaction with vaccine-hesitant parents. STUDY DESIGN: Standardized patients portraying vaccine-hesitant parents were used to assess the effectiveness of training in the Announce, Inquire, Mirror, Secure (AIMS) Method for Healthy Conversations. Blinded pediatric residents were pseudorandomized to receive AIMS or control training and underwent pre- and post-training encounters with blinded standardized patients. Encounters were assessed by blinded raters using a novel tool. Participant confidence and standardized patient evaluations of the participants' general communication skills were assessed. RESULTS: Ratings were available for 27 AIMS and 26 control participants. Statistically significant increases in post-training scores (maximum = 30) were detected in AIMS, but not in control, participants (median, 21.3 [IQR, 19.8-24.8] vs 18.8 [IQR, 16.9-20.9]; P < .001). Elements (maximum score = 6) with significant increases were Inquire (0.67 [IQR, 0-1.76] vs -0.33 [IQR, -0.67 to 0.33]; P < .001); Mirror (1.33 [IQR, 0 to 2] vs -0.33 [IQR, -0.92 to 0]; P < .001) and Secure (0.33 [IQR, 0 to 1.67] vs -0.17 [IQR, -0.67 to 0.33]; P = .017). Self-confidence increased equally in both groups. Standardized patients did not detect a difference in communication skills after training and between groups. Internal consistency and inter-rater reliability of the assessment tool were modest. CONCLUSIONS: Standardized patients proved useful in studying the effectiveness of structured communication training, but may have been limited in their ability to perceive a difference between groups owing to the predetermined encounter outcome of vaccine refusal. AIMS training should be studied in real-world scenarios to determine if it impacts vaccine acceptance.

Psychometric Testing of the Debriefing Assessment for Simulation in Healthcare (DASH) for Trainee-led, In Situ Simulations in the Pediatric Emergency Department Context.

OBJECTIVES: Effective trainee-led debriefing after critical events in the pediatric emergency department has potential to improve patient care, but debriefing assessments for this context have not been developed. This study gathers preliminary validity and reliability evidence for the Debriefing Assessment for Simulation in Healthcare (DASH) as an assessment of trainee-led post-critical event debriefing. METHODS: Eight fellows led teams in three simulated critical events, each followed by a video-recorded discussion of performance mimicking impromptu debriefings occurring after real clinical events. Three raters assessed the recorded debriefings using the DASH, and their feedback was collated. Data were analyzed using generalizability theory, Gwet's AC2, intraclass correlation coefficient (ICC), and coefficient alpha. Validity was examined using Messick's framework. RESULTS: The DASH instrument had relatively low traditional inter-rater reliability (Gwet's AC2 = 0.24, single-rater ICC range = 0.16-0.35), with 30% fellow, 19% rater, and 23% rater by fellow variance. DASH generalizability (G) coefficient was 0.72, confirming inadequate reliability for research purposes. Decision (D) study results suggest the DASH can attain a G coefficient of 0.8 with five or more raters. Coefficient alpha was 0.95 for the DASH. A total of 90 and 40% of items from Elements 1 and 4, respectively, were deemed "not applicable" or left blank. CONCLUSIONS: Our results suggest that the DASH does not have sufficient validity and reliability to rigorously assess debriefing in the post-critical event environment but may be amenable to modification. Further development of the tool will be needed for optimal use in this context.

Guidelines for the Responsible Use of Deception in Simulation: Ethical and Educational Considerations.

STATEMENT: Many techniques and modifications commonly used by the simulation community have been identified as deceptive. Deception is an important issue addressed by both the newly adopted Healthcare Simulationist Code of Ethics and the American Psychological Association Code of Conduct. Some view these approaches as essential whereas others question their necessity as well as their untoward psychological effects. In an attempt to offer guidance to simulation-based healthcare educators, we explore educational practices commonly identified as deceptive along with their potential benefits and detriments. We then address important decision points and high-risk situations that should be avoided to uphold ethical boundaries and psychological safety among learners. These are subsequently analyzed in light of the Code of Ethics and used to formulate guidelines for educators that are intended to ensure that deception, when necessary, is implemented in as psychologically safe a manner as possible.

Reflections on Career Pathways of Simulation-Focused Experts in the Field.

STATEMENT: This article presents reflections of career pathways of simulation researchers as well as a discussion of the themes found in the stories presented. It is the intent of the authors to present and foster a discussion around the ways in which we as a simulation community wish to promote recognition of scholarship among simulation researchers and help support newcomers find success as simulation researchers in academia. We also present recommendations for those considering entering the field based on tactics that were successful and not successful among the scholars who shared their stories.

The Second Society for Simulation in Healthcare Research Summit: Beyond our Boundaries.

STATEMENT: In this article, we report on the Second Society for Simulation in Healthcare Research Summit held in 2017. This meeting succeeded the First Research Summit from 2011 with the goal of advancing the scope of healthcare simulation research. During the one and a half day summit, some of the world's leading experts in simulation, healthcare, and simulation in healthcare convened to discuss ideas about what research goals would be most beneficial to the healthcare simulation community, and what could be done to achieve them. We describe the rationale for the meeting, the organization, the program, and the articles that emerged from the Summit, which are found in this supplemental issue of Simulation in Healthcare.

The Development and Validation of a Concise Instrument for Formative Assessment of Team Leader Performance During Simulated Pediatric Resuscitations.

AIM: The aim of this study was to assess the validity of a formative feedback instrument for leaders of simulated resuscitations. METHODS: This is a prospective validation study with a fully crossed (person × scenario × rater) study design. The Concise Assessment of Leader Management (CALM) instrument was designed by pediatric emergency medicine and graduate medical education experts to be used off the shelf to evaluate and provide formative feedback to resuscitation leaders. Four experts reviewed 16 videos of in situ simulated pediatric resuscitations and scored resuscitation leader performance using the CALM instrument. The videos consisted of 4 pediatric emergency department resuscitation teams each performing in 4 pediatric resuscitation scenarios (cardiac arrest, respiratory arrest, seizure, and sepsis). We report on content and internal structure (reliability) validity of the CALM instrument. RESULTS: Content validity was supported by the instrument development process that involved professional experience, expert consensus, focused literature review, and pilot testing. Internal structure validity (reliability) was supported by the generalizability analysis. The main component that contributed to score variability was the person (33%), meaning that individual leaders performed differently. The rater component had almost zero (0%) contribution to variance, which implies that raters were in agreement and argues for high interrater reliability. CONCLUSIONS: These results provide initial evidence to support the validity of the CALM instrument as a reliable assessment instrument that can facilitate formative feedback to leaders of pediatric simulated resuscitations.