Assessing the Ability of a Large Language Model to Score Free-Text Medical Student Clinical Notes: Quantitative Study.

Background: Teaching medical students the skills required to acquire, interpret, apply, and communicate clinical information is an integral part of medical education. A crucial aspect of this process involves providing students with feedback regarding the quality of their free-text clinical notes. Objective: The goal of this study was to assess the ability of ChatGPT 3.5, a large language model, to score medical students' free-text history and physical notes. Methods: This is a single-institution, retrospective study. Standardized patients learned a prespecified clinical case and, acting as the patient, interacted with medical students. Each student wrote a free-text history and physical note of their interaction. The students' notes were scored independently by the standardized patients and ChatGPT using a prespecified scoring rubric that consisted of 85 case elements. The measure of accuracy was percent correct. Results: The study population consisted of 168 first-year medical students. There was a total of 14,280 scores. The ChatGPT incorrect scoring rate was 1.0%, and the standardized patient incorrect scoring rate was 7.2%. The ChatGPT error rate was 86%, lower than the standardized patient error rate. The ChatGPT mean incorrect scoring rate of 12 (SD 11) was significantly lower than the standardized patient mean incorrect scoring rate of 85 (SD 74; P=.002). Conclusions: ChatGPT demonstrated a significantly lower error rate compared to standardized patients. This is the first study to assess the ability of a generative pretrained transformer (GPT) program to score medical students' standardized patient-based free-text clinical notes. It is expected that, in the near future, large language models will provide real-time feedback to practicing physicians regarding their free-text notes. GPT artificial intelligence programs represent an important advance in medical education and medical practice.

Survey of Current Simulation Based Training in the US Military Health System.

INTRODUCTION: Simulation-based medical training has been shown to be effective and is widely used in civilian hospitals; however, it is unclear how widely and how effectively simulation is utilized in the U.S. Military Health System (MHS). The current operational state of medical simulation in the MHS is unknown, and there remains a need for a system-wide assessment of whether and how the advances in simulation-based medical training are employed to meet the evolving needs of the present-day warfighter. Understanding the types of skills and methods used within simulation programs across the enterprise is important data for leaders as they plan for the future in terms of curriculum development and the investment of resources. The aim of the present study is to survey MHS simulation programs in order to determine the prevalence of skills taught, the types of learners served, and the most common methodologies employed in this worldwide health care system. MATERIALS AND METHODS: A cross-sectional survey of simulation activities was distributed to the medical directors of all 93 simulation programs in the MHS. The survey was developed by the authors based on lists of critical wartime skills published by the medical departments of the US Army, Navy, and Air Force. Respondents were asked to indicate the types of learners trained at their program, which of the 82 unique skills included in the survey are trained at their site, and for each skill the modalities of simulation used, i.e., mannequin, standardized patients, part task trainers, augmented/virtual reality tools, or cadaver/live tissue. RESULTS: Complete survey responses were obtained from 75 of the 93 (80%) MHS medical simulation training programs. Across all skills included in the survey, those most commonly taught belonged predominantly to the categories of medic skills and nursing skills. Across all sites, the most common category of learner was the medic/corpsman (95% of sites), followed by nurses (87%), physicians (83%), non-medical combat lifesavers (59%), and others (28%) that included on-base first responders, law enforcement, fire fighters, and civilians. The skills training offered by programs included most commonly the tasks associated with medics/corpsmen (97%) followed by nursing (81%), advanced provider (77%), and General Medical Officer (GMO) skills (47%). CONCLUSION: The survey demonstrated that the most common skills taught were all related to point of injury combat casualty care and addressed the most common causes of death on the battlefield. The availability of training in medic skills, nursing skills, and advanced provider skills were similar in small, medium, and large programs. However, medium and small programs were less likely to deliver training for advanced providers and GMOs compared to larger programs. Overall, this study found that simulation-based medical training in the MHS is focused on medic and nursing skills, and that large programs are more likely to offer training for advanced providers and GMOs. Potential gaps in the availability of existing training are identified as over 50% of skills included in the nursing, advanced provider, and GMO skill categories are not covered by at least 80% of sites serving those learners.

Using an Innovative Model to Improve Performance of the Infant Hip Examination.

OBJECTIVES: Determine if a new teaching bundle targeting developmental dysplasia of the hip screening improved interns' examination skills across multiple pediatric residency programs. METHODS: This multicenter prospective cohort study included interns across 6 pediatric residency sites within the Academic Pediatric Association's Better Outcomes through Research for Newborns Network in 2022. Participants underwent a baseline hip examination assessment on models using a checklist derived from textbook descriptions of Galeazzi, Ortolani, and Barlow maneuvers before receiving a teaching bundle. Repeat testing occurred after instruction. Data were analyzed using t-test for continuous and χ2 test for categorical variables. Semistructured focus groups provided qualitative feedback regarding the teaching bundle. RESULTS: We enrolled 117 of 155 interns across 6 sites (76%) for participation in the teaching bundle. Only 2% of participants (n = 2) identified a positive Galeazzi sign at baseline, whereas 88% (n = 103, P < .001) did so on the postinstructional assessment. Although 27% of participants (n = 32) correctly identified a positive Barlow sign at baseline, 69% (n = 81, P < .001) did so on the postinstructional assessment. The ability to correctly detect a positive Ortolani sign increased from 22% (n = 26) to 92% (n = 108, P < .001). Fifteen interns participated in the semistructured focus groups, with resultant themes reinforcing the limited experience of the infant hip examination before this intervention and the positive impact of the teaching bundle. CONCLUSIONS: Most participants in this study did not have strong infant hip examination skills at entry into residency. A standardized teaching bundle significantly improved skills in examination technique and identifying abnormalities.

Retention of En Route Cricothyroidotomy Skills in Novice Providers Following a Simulation-Based Mastery Learning Curriculum.

INTRODUCTION: Surgical cricothyroidotomy (SC) is a vital skill that combat first responders must master as airway obstruction is the third most preventable cause of death on the battlefield. Degradation of skills over time is a known problem, and there is inadequate knowledge regarding the rate of SC skill retention. Our prior study showed that simulation-based mastery learning was effective in training 89 novices how to reliably perform an en route SC to mastery performance standards. This study aims to assess the durability of this skill by bringing participants back in 3 separate cohorts at 6, 12, or 24 months following the initial training to perform SC in the same test environment. MATERIALS AND METHODS: This was a randomized prospective trial. Random cohorts of equal subjects who previously underwent SC simulation-based mastery learning training were selected to return at 6, 12, and 24 months to retest in the same en route medical evacuation (MEDEVAC) helicopter scenario. A total of 22, 14, and 10 subjects returned at 6, 12, and 24 months, respectively, due to Coronavirus-19 impacts and travel limitations. Participants in the 24-month cohort received a refresher training prior to retesting. All attempts were recorded and blindly graded using the same 10 item standardized SC checklist used in initial training. Our previous work found that mastery criteria for performing a SC were ≤40 seconds and completion of 9/10 items on the checklist. Outcome measures in this study were time to complete the procedure and percent of subjects who completed at least 9/10 items on the SC checklist. RESULTS: There was an increase in time required to complete the procedure compared to initial training in all three retesting cohorts (initial: median 27.50, interquartile range 25.38-31.07 seconds; 6 months: median 36.33, interquartile range 31.59-55.22 seconds; 12 months: median 49.50, interquartile range 41.75-60.75 seconds; 24 months: median 38.79, interquartile range 30.20-53.08 seconds; P < .0001, P < .0001, P = .0039). There was a decline in median value checklist scores compared to initial training in the 6- and 12-month retesting cohorts (initial: median 10.00/10, interquartile range 9.50-10.00; 6 months: median 8.00/10, interquartile range 6.75-9.00; 12 months: median 8.00/10, interquartile range 6.75-9.25; P < .0001, P < .001). There was no difference in median checklist scores between the initial and 24-month retesting scenario (initial: median 10.00/10, interquartile range 9.50-10.00; 24 months: 10.00/10, interquartile range 9.00-10.00; P= .125). There was a decrease in retention of skills as only 31.82% of subjects at 6 months and 14.29% at 12 months met the defined passing criteria of time to completion of ≤40 seconds and checklist score of ≥9/10. A brief refresher course several months prior to the 24-month cohort retesting greatly increased the retention of SC procedural skills, with 60% of subjects meeting the time and checklist criteria. CONCLUSIONS: This study showed that the skill required to perform a SC after initial mastery training does decay significantly. A brief refresher course can help increase retention of skills. Based on our findings SC skills should be refreshed at a minimum of every 6 months to assure optimal proficiency.

A Review of Verbal and Written Patient Handoffs Applicable to the U.S. Military's Expeditionary Care System.

INTRODUCTION: Long considered a danger point in patient care, handoffs and patient care transitions contribute to medical errors and adverse events. Without standardization of patient handoffs, communication breakdowns arise and critical patient information is lost. Minimal training and informal learning have led to a lack of understanding the process involved in this vital aspect of patient care. In 2017, the U.S. Army commissioned a report to study the process of patient handoffs and identify training gaps. Our report summarizes that process and makes recommendations for implementation. MATERIALS AND METHODS: Scoping literature review of 139 articles published between 1999 and 2017 using PubMed, CINAHL, Cochrane, and Medline databases. Verbal tools for handoffs were evaluated against 12 criteria including patient ID, history, current situation, contingency planning, ability to ask questions, ownership, and read back. Written tools were evaluated against a matrix of 126 casualty/treatment attributes. RESULTS: Among verbal communication protocols, the highest scoring handoff mnemonics were HAND ME AN ISOBAR, IPASS the BATON, and I-SBARQ. Among written handoff tools, the highest scoring documents were the Special Operations Forces (SOF) Mechanism, Injuries, Signs, and Treatment (MIST) Casualty Treatment Card and the Department of Defense (DD) Form 1380 Tactical Combat Casualty Care (TCCC) Card. Four critical process elements for patient handoffs and transfers were identified: (1) interactive communications, (2) limited interruptions, (3) a process for verification, and (4) an opportunity to review any relevant historical data. CONCLUSIONS: The findings in this review highlight the need for standardized tools and techniques for patient handoffs in the U.S. Military's expeditionary care system. Future research is needed to trial verbal and nonverbal handoffs under field conditions to gather observational data to assess effectiveness. The results of our gap analyses may provide researchers insight for determining which handoffs to study. If standardized handoffs are utilized, training programs should incorporate the four critical elements into their curricula.

Innovative Curriculum to Train Medical Students on Lateral Canthotomy and Cantholysis.

PURPOSE: Orbital compartment syndrome (OCS) is an ocular emergency requiring prompt decompression with a lateral canthotomy and cantholysis (LCC) within 2 hours. This study evaluates the feasibility and effectiveness of a standardized LCC curriculum to train medical students to identify and treat OCS. METHODS: This was a prospective, non-randomized, non-comparative cohort study of 39 novice first-year medical students with no prior LCC training who underwent a standardized LCC curriculum incorporating both didactic and hands-on procedural training. Didactic knowledge of orbital anatomy and OCS was evaluated with written pre- and post-knowledge testing. Expert performance criteria were determined by expert consensus based on the performance of three oculoplastic surgeons and were defined as correctly performing all 12 critical checklist steps of an LCC within 3 minutes twice consecutively on a Sonalyst LCC training system eye model. Utilizing the principles of mastery learning, participants learned how to perform an LCC in a classroom environment and were evaluated on a final test of proficiency in a training lane designed to simulate an austere military environment. RESULTS: Participants required a median of 3.0 practice iterations to achieve expert performance in the classroom environment. During the testing phase, all participants correctly identified the eye with OCS, and 77% (n = 30) of learners successfully performed an LCC at the expert level within their first attempt. The median completion time of those who passed on their first testing was 130 seconds. The mean LCC knowledge test scores significantly improved from 48.7% to 71.2% (P < .001). CONCLUSION: This study successfully developed a standardized LCC curriculum utilizing the principles of hands-on mastery learning to train novice learners to perform an LCC efficiently and effectively.

The SOAP Feedback Training Program.

BACKGROUND: Accessible and efficient opportunities for health professional faculty to hone feedback skills are limited. In addition, feedback models to apply to the objective structured clinical examination (OSCE) setting are lacking. APPROACH: Annually, paediatric interns from Children's National Hospital and Walter Reed National Military Medical Center participate in an OSCE, which includes faculty observation and immediate feedback to trainees. In 2018, we incorporated the subjective, objective, assessment, plan (SOAP) Feedback Training Program during 20 min of the pre-OSCE faculty orientation. The SOAP Feedback Training Program introduced the SOAP feedback model (subjective, objective, assessment, plan), facilitated practice in pairs and distributed a cognitive aid referencing the model. We evaluated the quality of faculty feedback exchanges during the 2018 OSCE via retrospective video review using the Direct Observation of Clinical Skills Feedback Scale (DOCS-FBS). We compared the results to the 2015 initial evaluation and used focus groups to understand how and why faculty feedback changed. EVALUATION: Comparison of the initial evaluation to the post-SOAP Feedback Training Program intervention data using a Wilcoxon signed rank test showed statistically significant improvement in six of eight feedback items on the DOCS-FBS. Causal coding of focus group transcripts revealed that the SOAP Feedback Training Program evoked affective responses, reinforced prior practice in feedback delivery, improved feedback organisation and increased feedback delivery preparation. IMPLICATIONS: The SOAP Feedback Training Program is an effective intervention to teach the SOAP feedback model and improve faculty feedback quality in an OSCE setting. It is efficient and low resource, facilitating its potential use in settings beyond the OSCE.

Development of a Simulation Surgical Cricothyroidotomy Curriculum for Novice Providers: A Learning Curve Study.

INTRODUCTION: Airway obstruction is the third most common cause of preventable death on the battlefield, accounting for 1%-2% of total combat fatalities. No previous surgical cricothyroidotomy (SC) studies have analyzed the learning curve required to obtain proficiency despite being studied in numerous other surgical technique training experiments. The aims of this study were to establish expert SC performance criteria, develop a novel standardized SC curriculum, and determine the necessary number of practice iterations required by a novice to reach this pre-determined performance goal. MATERIALS AND METHODS: A standardized checklist and SC performance standards were established based on the performance of 12 board certified Military Health System surgeons with prior experience on performing a SC using a simulated trauma mannequin. Expert-level criteria were defined as a SC time to completion of 40 s or less and checklist score of at least 9/10, including all critical steps. Study subjects included 89 novice providers (54 active-duty first- and second-year medical students and 35 Navy corpsmen). Subjects received instruction on performing a SC using the principles of mastery learning and performed a final test of SC proficiency on a trauma mannequin within a realistic simulated MEDEVAC helicopter. The total number of subject practice attempts, checklist scores, and time to completion were measured and/or blindly scored. Learning curve and exponential plateau equations were used to characterize their improvement in mean time to SC completion and checklist scores. RESULTS: Mean pre-test knowledge scores for the entire group were 11.8 ± 3.1 out of 24 points. Total mean practice learning plateaued at checklist scores of 9.9/10 after 7 iterations and at a mean completion time of 30.4 s after 10 iterations. During the final test performance in the helicopter, 67.4% of subjects achieved expert-level performance on the first attempt. All subjects achieved expert-level performance by the end of two additional attempts. While a significantly larger proportion of medical students (79.9%) successfully completed the helicopter test on the first attempt compared to corpsmen (54.3%), there were no statistically significant differences in mean SC completion times and checklist scores between both groups (P > 0.05). Medical students performed a SC only 1.3 s faster and scored only 0.16 points higher than corpsmen. The effect size for differences were small to negligible (Cohen's d range 0.18-0.33 for SC completion time; Cohen's d range 0.45-0.46 for checklist scores). CONCLUSION: This study successfully defined SC checklist scores and completion times based on the performance of experienced surgeons on a simulator. Using these criteria and the principles of mastery learning, novices with little knowledge and experience in SC were successfully trained to the level of experienced providers. All subjects met performance targets after training and overall performance plateaued after approximately seven iterations. Over two-thirds of subjects achieved the performance target on the first testing attempt in a simulated helicopter environment. Performance was comparable between medical student and corpsmen subgroups. Further research will assess the durability of maintaining SC skills and the timing for introducing refresher courses after initial skill acquisition.

Validation of a Modified Jefferson Scale of Empathy for Observers to Assess Trainees.

OBJECTIVE: "Demonstrate insight and understanding into emotion" is a competency amenable to simulation-based assessment. The Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE) has validity evidence for patients to assess provider empathy. A version adapted for a third-party observers does not exist. Our aim was to modify the JSPPPE and use recorded standardized encounters to obtain validity evidence. METHODS: This cross-sectional study used video and data collected from 2 pediatric residencies. In 2018, 4 raters reviewed 24 videos of 12 interns communicating with standardized patients (SP) in 2 encounters and completed a modified JSPPE for observers (JSEO). Reliability between raters was established using Intraclass Correlations (ICC). JSEO mean scores were correlated to Essential Elements of Communication (EEC), JSPPPE, and faculty composite interpersonal communication (IC) scores using Spearman Rank. RESULTS: The mean ICC for all 4 raters was 0.573 (0.376-0.755). When ICC was calculated for pairs of raters, Rater 1 was an outlier. ICCs for mean scores for pairs among the 3 remaining raters was 0.81 to 0.84. Mean JSEO scores from the four raters correlated with the JSPPPE (rho = 0.45, P = .03) and IC (rho = 0.68, P < .001), but not the EEC (rho = 0.345, P = .1). CONCLUSIONS: We found validity evidence for the use of a modified JSPPPE for an observer to assess empathy in a recorded encounter with a SP. This may be useful as medical educators shift toward competency-based tracking. The brevity of this tool and potential assessment using video are also appealing.

I-PASS Mentored Implementation Handoff Curriculum: Frontline Provider Training Materials.

Introduction: The I-PASS Handoff Program is a comprehensive handoff curriculum that has been shown to decrease rates of medical errors and adverse events during patient handoffs. Frontline providers are the key individuals participating in handoffs of patient care. It is important they receive robust handoff training. Methods: The I-PASS Mentored Implementation Handoff Curriculum frontline provider training materials were created as part of the original I-PASS Study and adapted for the Society of Hospital Medicine (SHM) I-PASS Mentored Implementation Program. The adapted materials embrace a flipped classroom approach with an emphasis on adult learning theory principles. The training includes an overview of I-PASS handoff techniques, TeamSTEPPS team communication strategies, verbal handoff simulation scenarios, and a printed handoff document exercise. Results: As part of the SHM I-PASS Mentored Implementation Program, 2,735 frontline providers were trained at 32 study sites (16 adult and 16 pediatric) across North America. At the end of their training, 1,762 frontline providers completed the workshop evaluation form (64% response rate). After receiving the training, over 90% agreed/strongly agreed that they were able to distinguish a good- from a poor-quality handoff, articulate the elements of the I-PASS mnemonic, construct a high-quality patient summary, advocate for an appropriate environment for handoffs, and participate in handoff simulations. Universally, the training provided them with knowledge and skills relevant to their patient care activities. Discussion: The I-PASS frontline training materials were rated highly by those trained and are an integral part of a successful I-PASS Handoff Program implementation.

Rapid Application Temporary Tattoos for Medical Moulage: From Development to Testing and Commercialization.

Military medical education and training often utilize theatrical makeup, or moulage, to simulate injuries or pathologies. Traditional methods of moulage application are incredibly realistic when expertly applied. However, moulage can be expensive in terms of supplies, manpower, and time. We proposed that by creating a library of illustrations for use with temporary tattoos, the overall cost of moulage could decrease with little to no impact on training goals and objectives. The development, detailed testing and plans for commercialization are outlined herein.

A Focus on Feedback: Improving Learner Engagement and Faculty Delivery of Feedback in Hospital Medicine.

Feedback is an integral part of medical education. However, there is great variation of training and effectiveness of feedback delivery, especially in the inpatient setting. The unique learning environment provided in hospital medicine allows teachers the opportunity to provide feedback on learner performance under several longitudinal observations in areas such as direct patient care, procedural tasks, and interdisciplinary team leadership skills. Most important, feedback should occur on more than one occasion to truly empower change in knowledge, attitude, and skills. This article aims to provide the reader with foundational theories on feedback and strategies to use best practices for delivery.

I-PASS Mentored Implementation Handoff Curriculum: Champion Training Materials.

Introduction: The I-PASS Handoff Program is a comprehensive handoff curriculum that has been shown to decrease rates of medical errors and adverse events during patient handoffs. I-PASS champions are a critical part of the implementation and sustainment of this curriculum, and therefore, a rigorous program to support their training is necessary. Methods: The I-PASS Handoff champion training materials were created for the original I-PASS Study and adapted for the Society of Hospital Medicine (SHM) I-PASS Mentored Implementation Program. The adapted materials embrace a flipped classroom approach and adult learning theory. The training includes an overview of I-PASS handoff techniques, an opportunity to practice evaluating handoffs with the I-PASS observation tools using a handoff video vignette, and other key implementation principles. Results: As part of the SHM I-PASS Mentored Implementation Program, 366 champions were trained at 32 sites across North America and participated in a total of 3,491 handoff observations. A total of 346 champions completed the I-PASS Champion Workshop evaluation form at the end of their training (response rate: 94.5%). After receiving the training, over 90% agreed/strongly agreed that it provided them with knowledge or skills critical to their patient care activities and that they were able to distinguish the difference between high- and poor-quality handoffs, competently use the I-PASS handoff assessment tools, and articulate the importance of handoff observations. Conclusion: The I-PASS champion training materials were rated highly by those trained and are an integral part of a successful I-PASS Handoff Program implementation.

Patient safety after implementation of a coproduced family centered communication programme: multicenter before and after intervention study

OBJECTIVE: To determine whether medical errors, family experience, and communication processes improved after implementation of an intervention to standardize the structure of healthcare provider-family communication on family centered rounds. DESIGN: Prospective, multicenter before and after intervention study. SETTING: Pediatric inpatient units in seven North American hospitals, 17 December 2014 to 3 January 2017. PARTICIPANTS: All patients admitted to study units (3106 admissions, 13171 patient days); 2148 parents or caregivers, 435 nurses, 203 medical students, and 586 residents. INTERVENTION: Families, nurses, and physicians coproduced an intervention to standardize healthcare provider-family communication on ward rounds ("family centered rounds"), which included structured, high reliability communication on bedside rounds emphasizing health literacy, family engagement, and bidirectional communication; structured, written real-time summaries of rounds; a formal training programme for healthcare providers; and strategies to support teamwork, implementation, and process improvement. MAIN OUTCOME MEASURES: Medical errors (primary outcome), including harmful errors (preventable adverse events) and non-harmful errors, modeled using Poisson regression and generalized estimating equations clustered by site; family experience; and communication processes (eg, family engagement on rounds). Errors were measured via an established systematic surveillance methodology including family safety reporting. RESULTS: The overall rate of medical errors (per 1000 patient days) was unchanged (41.2 (95% confidence interval 31.2 to 54.5) pre-intervention v 35.8 (26.9 to 47.7) post-intervention, P=0.21), but harmful errors (preventable adverse events) decreased by 37.9% (20.7 (15.3 to 28.1) v 12.9 (8.9 to 18.6), P=0.01) post-intervention. Non-preventable adverse events also decreased (12.6 (8.9 to 17.9) v 5.2 (3.1 to 8.8), P=0.003). Top box (eg, "excellent") ratings for six of 25 components of family reported experience improved; none worsened. Family centered rounds occurred more frequently (72.2% (53.5% to 85.4%) v 82.8% (64.9% to 92.6%), P=0.02). Family engagement 55.6% (32.9% to 76.2%) v 66.7% (43.0% to 84.1%), P=0.04) and nurse engagement (20.4% (7.0% to 46.6%) v 35.5% (17.0% to 59.6%), P=0.03) on rounds improved. Families expressing concerns at the start of rounds (18.2% (5.6% to 45.3%) v 37.7% (17.6% to 63.3%), P=0.03) and reading back plans (4.7% (0.7% to 25.2%) v 26.5% (12.7% to 7.3%), P=0.02) increased. Trainee teaching and the duration of rounds did not change significantly. CONCLUSIONS: Although overall errors were unchanged, harmful medical errors decreased and family experience and communication processes improved after implementation of a structured communication intervention for family centered rounds coproduced by families, nurses, and physicians. Family centered care processes may improve safety and quality of care without negatively impacting teaching or duration of rounds. TRIAL REGISTRATION: ClinicalTrials.gov NCT02320175.

Multiple Patient Casualty Scenarios: A Measurement Tool for Teamwork.

INTRODUCTION: Teamwork is a critical aspect of patient care and is especially salient in response to multiple patient casualties. Effective training and measurement improve team performance. However, the literature currently lacks a scientifically developed measure of team performance within multiple causality scenarios, making training and feedback efforts difficult. The present effort addresses this gap by integrating the input of subject matter experts and the science of multicasualty teams and training to (1) identify overarching teamwork processes and corresponding behaviors necessary for team performance and (2) develop a behavioral observation tool to optimize teamwork in multicasualty training efforts. METHOD: A search of articles including team performance frameworks associated with team training was conducted, leading to the identification of a total of 14 articles. Trained coders extracted teamwork processes and the corresponding team behaviors indicative of effective performance from these articles. Five subject matter experts were interviewed using the critical incident technique to identify additional behaviors. RESULTS: Team situation awareness, team leadership, coordination, and information exchange emerged as the four core team processes required for team performance in scenarios with multiple patient casualties. Relevant behaviors and subbehaviors within these overarching processes were identified to inform a pilot behavioral framework of team performance. CONCLUSIONS: The processes and associated behaviors identified within this effort serve as scientifically grounded behaviors of team performance in the case of multiple patient casualties simulated training scenarios. Future work can use and further refine these results to ensure that measures of team performance are grounded in specific, observable, and scientifically delineated behaviors.

Validity Evidence for a Serious Game to Assess Performance on Critical Pediatric Emergency Medicine Scenarios.

INTRODUCTION: We developed a first-person serious game, PediatricSim, to teach and assess performances on seven critical pediatric scenarios (anaphylaxis, bronchiolitis, diabetic ketoacidosis, respiratory failure, seizure, septic shock, and supraventricular tachycardia). In the game, players are placed in the role of a code leader and direct patient management by selecting from various assessment and treatment options. The objective of this study was to obtain supportive validity evidence for the PediatricSim game scores. METHODS: Game content was developed by 11 subject matter experts and followed the American Heart Association's 2011 Pediatric Advanced Life Support Provider Manual and other authoritative references. Sixty subjects with three different levels of experience were enrolled to play the game. Before game play, subjects completed a 40-item written pretest of knowledge. Game scores were compared between subject groups using scoring rubrics developed for the scenarios. Validity evidence was established and interpreted according to Messick's framework. RESULTS: Content validity was supported by a game development process that involved expert experience, focused literature review, and pilot testing. Subjects rated the game favorably for engagement, realism, and educational value. Interrater agreement on game scoring was excellent (intraclass correlation coefficient = 0.91, 95% confidence interval = 0.89-0.9). Game scores were higher for attendings followed by residents then medical students (Pc < 0.01) with large effect sizes (1.6-4.4) for each comparison. There was a very strong, positive correlation between game and written test scores (r = 0.84, P < 0.01). CONCLUSIONS: These findings contribute validity evidence for PediatricSim game scores to assess knowledge of pediatric emergency medicine resuscitation.

Asymptomatic Kawasaki Disease in a 3-Month-Old Infant.

Kawasaki disease (KD) is the leading cause of acquired heart disease in children in the United States. It is a systemic vasculitis characterized by diffuse inflammation of medium and small blood vessels. If untreated it can lead to myocardial infarction, ischemic heart disease, or sudden death. Early recognition and treatment decrease the incidence of coronary consequences, resulting in improved clinical outcomes. Incomplete KD is much less likely to fulfill major clinical diagnostic criteria. Infants <12 months of age are more likely to have an incomplete presentation, and children <6 months of age are more likely to develop cardiac complications. We present a case of a 3-month-old, previously healthy white boy who was noted to have a new transient cardiac murmur during a routine health assessment. He was completely asymptomatic, and physical examination was otherwise within normal limits. An echocardiogram was performed and showed abnormal dilation of several coronary arteries, consistent with the coronary ectasia associated with KD. Laboratory evaluation was significant for values suggestive of systemic inflammation. Based on these results, a presumed diagnosis of incomplete KD was made and treatment administered. Close surveillance was undertaken, and serial laboratory studies and imaging showed gradual resolution of inflammatory markers and cardiac ectasia. This unique case of incomplete KD without any of the physical signs normally associated with the disease emphasizes the spectrum of presentation and the possibility of missing a diagnosis of incomplete disease, reinforcing the need to remain vigilant.

Do Standardized Patients Have Concerns About Students Not Captured by Traditional Assessment Forms?

Construct: Traditionally, standardized patients (SPs) assess students' clinical skills principally through numerical rating forms-an approach that may not fully capture SPs' concerns. SPs are students' closest approximation to real patients. To maximally benefit students' clinical training and evaluation it is important to find ways to give voice to the totality of SPs' perspectives. BACKGROUND: SPs have been shown to be a reliable and valid means to assess medical students' clinical skills in clinical skills examinations. We noticed, however, that SPs often express "off the record" concerns about students, which they do not include on traditional assessment forms. APPROACH: To explore these "off the record" concerns, we designed a Concerns item and added it to the traditional assessment form for an end-of-3rd-year clinical skills examination shared by three medical schools. We asked SPs to use this Concerns item to identify students about whom they had any "gut-level" concerns and provided them with a narrative opportunity to explain why. SPs were informed that the purpose of the item was to help students with difficulties and was not part of the student's grade. RESULTS: We analyzed the concerns data using quantitative and qualitative methods. Of 551 students at three schools, 223 (∼40%) had concerns recorded. Seventy students received two or more concerns. Qualitative analysis of SPs' comments revealed 3 major categories of concern: communication and interpersonal skills, history taking, and physical exam. Grouped under each were several subcategories. More than half of the written comments from the SPs related to the communication/interpersonal skills category and included subcategories commonly addressed in communications courses: lack of empathy, good listening skills, and lack of connection to the patient. They also included subcategories that in our experience are less commonly addressed: odd or off-putting mannerisms, lack of confidence, unprofessional behavior, domineering behavior, and biased behavior. Another 47% of concerns identified deficiencies in history taking and physical examination. Of the students with concerns noted by two or more SPs, SPs' narrative comments on 84%, 42%, and 48% of the students in the domains of communications, history, and physical exam respectively indicated potential problems not identified by scores on the traditional assessment form. CONCLUSION: The Concerns item is a narrative assessment method that may add value to traditional quantitative scoring by identifying and characterizing problematic student performance not captured by the traditional assessment form. It may thus contribute to giving fuller voice to the totality of SPs' perspective.

Acute Otitis Media Curriculum.

INTRODUCTION: Studies have shown that clinicians do a poor job of diagnosing middle ear infections, and an inaccurate diagnosis may result in inappropriate treatment, such as overuse of antibiotics. Antibiotic overuse is known to lead to increased bacterial resistance and puts patients at risk of deleterious side effects, including allergic reactions, vomiting, diarrhea, and rash. In this curriculum, we teach three important topics for achieving a successful middle ear exam and diagnosis of pathology. METHODS: The topics covered during the session are middle ear examination techniques, diagnosis of middle ear pathology and criteria for diagnosing acute otitis media, and treatment of acute otitis media. Each topic has a 30-minute lesson plan with broad goals, specific and measurable learning objectives, interactive learning activities, and assessment measures. Teaching activities include a mix of knowledge, skill, and attitude activities. Assessment measures aim at the highest level of Miller's pyramid of assessment when possible. Also included is a scholarship map that relates how this learning activity can fit into the assessment of learners at the program level by linking learning objectives with pediatric developmental milestones. RESULTS: The interactive nature of these lessons has been very well received by learners, and pediatric residents have had a self-perceived increase in skills. DISCUSSION: This curriculum provides a standardized approach to teaching the middle ear exam structured by educational rubrics and allows for accurate assessment of learners.

Optimizing Your Mentoring Relationship: A Toolkit for Mentors and Mentees.

INTRODUCTION: Mentorship is a vital component of academic and professional development. Mentees report positive impacts from mentorship programs, yet institutions and societies may struggle to meet their mentees' needs due to factors such as mentor fatigue and lack of mentor training. To address this in our own professional society, the Association of Pediatric Program Directors, we developed a mentor toolkit in order to utilize a variety of mentoring models, provide faculty development for midlevel mentors, and offer guidance to mentees. METHODS: Most of these tools were designed to be administered in an interactive format such as a workshop or seminar with think-pair-share opportunities. The toolkit begins by providing a definition of mentoring and reinforcing the benefits and the characteristics of effective mentoring relationships. Next, we discuss the important role that mentees have in creating and maintaining effective mentoring relationships (i.e., mentee-driven mentoring). We then introduce a mentoring mosaic activity designed to help mentees examine their professional network and think about how they might expand it to fulfill the spectrum of their mentoring needs. Next, we present guidelines for the implementation of four mentoring models that can be used within one's institution: traditional dyadic mentoring, peer group mentoring, meet the professor mentoring, and speed mentoring. We then provide tools that can be used to help facilitate effective mentoring development. RESULTS: This toolkit has successfully served as a self-guided resource at national meetings for many years, garnering positive feedback from mentors and mentees alike. DISCUSSION: The principles and methods are easily generalizable and may be used to guide mentorship programs within institutional and professional societies, as well as to assist mentors and mentees in optimizing their individual mentoring relationships.