Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Pilot testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams.

OBJECTIVE: To test the feasibility of implementing a standardized teamwork training program with full operating room teams in multiple institutions, driven by malpractice insurer support and incentives. BACKGROUND: Failures in intraoperative teamwork are among the leading causes of preventable patient injury and death in surgical patients. Teamwork training, particularly using simulation, can be an effective intervention but is difficult to scale. METHODS: A malpractice insurer convened a collaborative with 4 Harvard-affiliated simulation programs to develop a standardized operating room teamwork training curriculum, including principles of communication, assertiveness, and use of the World Health Organization Surgical Safety Checklist. Participant teams were compensated for lost operative time via malpractice premium discounts, continuing education credits, and compensation for lost wages. The course was delivered through a simulation program involving the management of intraoperative emergency scenarios. Participants were surveyed for their perceptions of the program and of its impact on clinical practice. RESULTS: A total of 221 active operating room staff members participated in the program. Each team contained at least 1 attending surgeon, 1 attending anesthesiologist, and 1 operating room nurse (mean size per team: 7 ± 2 participants). No study dates were cancelled because of lack of attendance. The survey response rate was 99% (218/221). Overall, the vast majority of participants found the scenarios realistic [94% (95% confidence interval: 90.9%, 97.2%)], appropriately challenging [95.4% (92.6%, 98.2%)], relevant to their practice [96.3% (93.8%, 98.8%)], and found the training would help them provide safer patient care [92.6% (89.1%, 96.1%)]. Surgeons reported their greatest personal deficit as communication skills. Operating room nurses and anesthesiologists reported a greater need than surgeons to work on personal assertiveness. CONCLUSIONS: A standardized multicenter team training program involving full operative teams is feasible with high-fidelity simulation and modest compensation for lost time. The vast majority of the multidisciplinary participants believed the course to have had a meaningful impact on their approach to clinical practice.

Simulated laparoscopic operating room crisis: An approach to enhance the surgical team performance.

OBJECTIVE: Diminishing human error and improving patient outcomes is the goal of task training and simulation experience. The fundamentals of laparoscopic surgery (FLS) is a validated tool to assess technical laparoscopic skills. We hypothesize that performance in a crisis depends on technical skills and team performance. The aim of this study was to develop and validate a high-fidelity simulation model of a laparoscopic crisis scenario in a mock endosuite environment. METHODS: To establish the feasibility of the model as well as its face and construct validity, the scenario evaluated the performances of FLS-certified surgeon experts (n = 5) and non-FLS certified novices (n = 5) during a laparoscopic crisis scenario, in a mock endosuite, on a simulated abdomen. Likert scale questionnaires were used for validity assessments. Groups were compared using previously validated rating scales on technical and nontechnical performance. Objective outcome measures assessed were: time to diagnose bleeding (TD), time to inform the team to convert (TT), and time to conversion to open (TC). SAS software was used for statistical analysis. RESULTS: Median scores for face validity were 4.29, 4.43, 4.71 (maximum 5) for the FLS, non-FLS, and nursing groups, respectively, with an inter-rater reliability of 93%. Although no difference was observed in Veress needle safety and laparoscopic equipment set up, there was a significant difference between the two groups in their overall technical and nontechnical abilities (p < 0.05), specifically in identifying bleeding, controlling bleeding, team communication, and team skills. There was a trend towards a difference between the two groups for TD, TT, and TC. While experts controlled bleeding in a shorter time, they persisted longer laparoscopically. CONCLUSIONS: Our evidence suggests that face and construct validity are established for a laparoscopic crisis simulation in a mock endosuite. Technical and nontechnical performance discrimination is observed between novices and experts. This innovative multidisciplinary simulation aims at improving error/problem recognition and timely initiation of appropriate and safe responses by surgical teams.

Developments in the Transition From Animal Use to Simulation-Based Biomedical Education.

STATEMENT: There has been a significant shift from the use of animals in biomedical training exercises toward simulation-based education methods. The transition has been driven by technological advances, empirical evidence of improved student outcomes, cost-effectiveness, and a growing concern for the welfare of animals. These factors have spurred policy changes worldwide in how medical and science curricula are delivered. We detail how some of these policy changes evolved and comment on the future direction of simulation-based education and its implications for healthcare providers, instructors, and the general public.

The use of in-situ simulation to improve safety in the plastic surgery office: a feasibility study.

OBJECTIVE: Simulation-based interventions and education can potentially contribute to safer and more effective systems of care. We utilized in-situ simulation to highlight safety issues, regulatory requirements, and assess perceptions of safety processes by the plastic surgery office staff. METHODS: A high-fidelity human patient simulator was brought to an office-based plastic surgery setting to enact a half-day full-scale, multidisciplinary medical emergency. Facilitated group debriefings were conducted after each scenario with special consideration of the principles of team training, communication, crisis management, and adherence to evidence-based protocols and regulatory standards. Abbreviated AHRQ Medical Office Safety Culture Survey was completed by the participants before and after the session. RESULTS: The in-situ simulations had a high degree of acceptance and face validity according to the participants. Areas highlighted by the simulation sessions included rapid communication, delegation of tasks, location of emergency materials, scope of practice, and logistics of transport. The participant survey indicated greater awareness of patient safety issues following participation in simulation and debriefing exercises in 3 areas (P < 0.05): the need to change processes if there is a recognized patient safety issue (100% vs 75%), openness to ideas about improving office processes (100% vs 88%), and the need to discuss ways to prevent errors from recurring (88% vs 62%). CONCLUSIONS: Issues of safety and regulatory compliance can be assessed in an office-based setting through the short-term (half-day) use of in-situ simulation with facilitated debriefing and the review of audiovisual recordings by trained facilities inspectors.

Early bedside care during preclinical medical education: can technology-enhanced patient simulation advance the Flexnerian ideal?

Flexner wanted medical students to study at the patient bedside-a remarkable innovation in his time-so that they could apply science to clinical care under the watchful eye of senior physicians. Ever since his report, medical schools have reserved the latter years of their curricula for such an "advanced" apprenticeship, providing clinical clerkship experiences only after an initial period of instruction in basic medical sciences. Although Flexner codified the segregation of preclinical and clinical instruction, he was committed to ensuring that both domains were integrated into a modern medical education. The aspiration to fully integrate preclinical and clinical instruction continues to drive medical education reform even to this day. In this article, the authors revisit the original justification for sequential preclinical-clinical instruction and argue that modern, technology-enhanced patient simulation platforms are uniquely powerful for fostering simultaneous integration of preclinical-clinical content in a way that Flexner would have applauded. To date, medical educators tend to focus on using technology-enhanced medical simulation in clinical and postgraduate medical education; few have devoted significant attention to using immersive clinical simulation among preclinical students. The authors present an argument for the use of dynamic robot-mannequins in teaching basic medical science, and describe their experience with simulator-based preclinical instruction at Harvard Medical School. They discuss common misconceptions and barriers to the approach, describe their curricular responses to the technique, and articulate a unifying theory of cognitive and emotional learning that broadens the view of what is possible, feasible, and desirable with simulator-based medical education.

How do clinical clerkship students experience simulator-based teaching? A qualitative analysis.

OBJECTIVES: To critically analyze the experience of clinical clerkship students exposed to simulator-based teaching, in order to better understand student perspectives on its utility. METHODS: A convenience sample of clinical students (n = 95) rotating through an emergency medicine, surgery, or longitudinal patient-doctor clerkship voluntarily participated in a 2-hour simulator-based teaching session. Groups of 3-5 students managed acute scenarios including respiratory failure, myocardial infarction, or multisystem trauma. After the session, students completed a brief written evaluation asking for free text commentary on the strengths and weaknesses of the experience; they also provided simple satisfaction ratings. Using a qualitative research approach, the textual commentary was transcribed and parsed into fragments, coded for emergent themes, and tested for inter-rater agreement. RESULTS: Six major thematic categories emerged from the qualitative analysis: The "Knowledge & Curriculum" domain was described by 35% of respondents, who commented on the opportunity for self-assessment, recall and memory, basic and clinical science learning, and motivation. "Applied Cognition and Critical Thought" was highlighted by 53% of respondents, who commented on the value of decision-making, active thought, clinical integration, and the uniqueness of learning-by-doing. "Teamwork and Communication" and "Procedural/Hands-On Skills" were each mentioned by 12% of subjects. Observations on the "Teaching/Learning Environment" were offered by 80% of students, who commented on the realism, interactivity, safety, and emotionality of the experience; here they also offered feedback on format, logistics, and instructors. Finally, "Suggestions for Use/Place in Undergraduate Medical Education" were provided by 22% of subjects, who primarily recommended more exposure. On a simple rating scale, 94% of students rated the quality of the simulator session as "excellent," whereas 91% felt the exercises should be "mandatory." CONCLUSION: Full-body simulation promises to address a wide range of pedagogical objectives using a unified educational platform. Students value experiential "practice without risk" and want more exposure to simulation. In this study, students thought that that an integrated simulation exercise could help solidify knowledge across domains, foster critical thought and action, enhance technical-procedural skills, and promote effective teamwork and communication.