Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Association of Simulation Participation With Diagnostic Reasoning Scores in Preclinical Students.

PURPOSE: The aim of the study was to evaluate for an association between the number of voluntary mannequin simulation sessions completed during the school year with scores on a year-end diagnostic reasoning assessment among second-year medical students. METHOD: This is retrospective analysis of participation in 0 to 8 extracurricular mannequin simulation sessions on diagnostic reasoning assessed among 129 second-year medical students in an end-of-year evaluation. For the final skills assessment, 2 physicians measured students' ability to reason through a standardized case encounter using the Diagnostic Justification (DXJ) instrument (4 categories each scored 0-3 by raters reviewing students' postencounter written summaries). Rater scores were averaged for a total DXJ score (0-12). To provide additional baseline comparison, zero participation students were divided into 2 groups based on intent to participate: those who signed up for extracurricular sessions but never attended versus those who never expressed interest. Scores across the attendance groups were compared with an analysis of variance and trend analysis. RESULTS: The class DXJ mean equaled 7.56, with a standard deviation of 2.78 and range of 0 to 12. Post hoc analysis after a significant analysis of variance (F = 4.91, df = 8, 128, P < 0.001) showed those participating in 1 or more extracurricular sessions had significantly higher DXJ scores than those not participating. Students doing 7 extracurricular sessions had significantly higher DXJ scores than those doing 0 and 2 (P < 0.05). Zero attendance groups were not different. A significant linear trend (R = 0.48, F = 38.0, df = 1, 127, P < 0.001) was found with 9 groups. A significant quadratic effect, like a dose-response pattern, was found (F = 18.1, df = 2, 125, P < 0.001) in an analysis including both zero attendance groups, a low (1-4 extracurricular sessions) group and a high (5-8) group. CONCLUSIONS: Higher year-end diagnostic reasoning scores were associated with increased voluntary participation in extracurricular mannequin-based simulation exercises in an approximate dose-response pattern.

Differences in safety climate among hospital anesthesia departments and the effect of a realistic simulation-based training program.

BACKGROUND: Safety climate is often measured via surveys to identify appropriate patient safety interventions. The introduction of an insurance premium incentive for simulation-based anesthesia crisis resource management (CRM) training motivated our naturalistic experiment to compare the safety climates of several departments and to assess the impact of the training. METHODS: We administered a 59-item survey to anesthesia providers in six academic anesthesia programs (Phase 1). Faculty in four of the programs subsequently participated in a CRM program using simulation. The survey was readministered 3 yr later (Phase 2). Factor analysis was used to create scales regarding common safety themes. Positive safety climate (% of respondents with positive safety attitudes) was computed for the scales to indicate the safety climate levels. RESULTS: The usable response rate was 44% (309/708) and 38% (293/772) in Phases 1 and 2 respectively. There was wide variation in response rates among hospitals and providers. Eight scales were identified. There were significantly different climate scores among hospitals but no difference between the trained and untrained cohorts. The positive safety climate scores varied from 6% to 94% on specific survey questions. Faculty and residents had significantly different perceptions of the degree to which residents are debriefed about their difficult clinical situations. CONCLUSIONS: Safety climate indicators can vary substantially among anesthesia practice groups. Scale scores and responses to specific questions can suggest practices for improvement. Overall safety climate is probably not a good criterion for assessing the impact of simulation-based CRM training. Training alone was insufficient to alter engrained behaviors in the absence of further reinforcing actions.

Developments in the Transition From Animal Use to Simulation-Based Biomedical Education.

STATEMENT: There has been a significant shift from the use of animals in biomedical training exercises toward simulation-based education methods. The transition has been driven by technological advances, empirical evidence of improved student outcomes, cost-effectiveness, and a growing concern for the welfare of animals. These factors have spurred policy changes worldwide in how medical and science curricula are delivered. We detail how some of these policy changes evolved and comment on the future direction of simulation-based education and its implications for healthcare providers, instructors, and the general public.

Anesthesia information management systems: a survey of current implementation policies and practices.

BACKGROUND: Anesthesia information management systems (AIMS) implementation is increasing, but there are no published recommendations from anesthesia professional societies to guide configuration and policy decisions that affect billing, security, medical-legal, and compliance issues. METHODS: A 45-question structured survey was developed by a committee of the Society for Technology in Anesthesia and was sent to the clinical administrator at 18 separate institutions, comprising six different installed AIMS systems. The primary goal of the survey was to establish a baseline of current policies and practices. RESULTS: There was more than two-third agreement among respondents for only 25% of questions. A number of configurations reported may increase exposure to billing denial, Medicare and Medicaid noncompliance, security breeches, and medical-legal defense difficulties. CONCLUSIONS: Developing guidelines by anesthesia professional organizations such as Society for Technology in Anesthesia to assist in the configuration of AIMS is recommended to help anesthesia departments avoid problems that may result in significant financial and legal risk.

The use of in-situ simulation to improve safety in the plastic surgery office: a feasibility study.

OBJECTIVE: Simulation-based interventions and education can potentially contribute to safer and more effective systems of care. We utilized in-situ simulation to highlight safety issues, regulatory requirements, and assess perceptions of safety processes by the plastic surgery office staff. METHODS: A high-fidelity human patient simulator was brought to an office-based plastic surgery setting to enact a half-day full-scale, multidisciplinary medical emergency. Facilitated group debriefings were conducted after each scenario with special consideration of the principles of team training, communication, crisis management, and adherence to evidence-based protocols and regulatory standards. Abbreviated AHRQ Medical Office Safety Culture Survey was completed by the participants before and after the session. RESULTS: The in-situ simulations had a high degree of acceptance and face validity according to the participants. Areas highlighted by the simulation sessions included rapid communication, delegation of tasks, location of emergency materials, scope of practice, and logistics of transport. The participant survey indicated greater awareness of patient safety issues following participation in simulation and debriefing exercises in 3 areas (P < 0.05): the need to change processes if there is a recognized patient safety issue (100% vs 75%), openness to ideas about improving office processes (100% vs 88%), and the need to discuss ways to prevent errors from recurring (88% vs 62%). CONCLUSIONS: Issues of safety and regulatory compliance can be assessed in an office-based setting through the short-term (half-day) use of in-situ simulation with facilitated debriefing and the review of audiovisual recordings by trained facilities inspectors.

Research regarding methods of assessing learning outcomes.

As the use of simulation-based assessment expands for healthcare workers, there is a growing need for research to quantify the psychometric properties of the associated process and outcome measures.