Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Operative team communication during simulated emergencies: Too busy to respond?

BACKGROUND: Ineffective communication among members of a multidisciplinary team is associated with operative error and failure to rescue. We sought to measure operative team communication in a simulated emergency using an established communication framework called "closed loop communication." We hypothesized that communication directed at a specific recipient would be more likely to elicit a check back or closed loop response and that this relationship would vary with changes in patients' clinical status. METHODS: We used the closed loop communication framework to code retrospectively the communication behavior of 7 operative teams (each comprising 2 surgeons, anesthesiologists, and nurses) during response to a simulated, postanesthesia care unit "code blue." We identified call outs, check backs, and closed loop episodes and applied descriptive statistics and a mixed-effects negative binomial regression to describe characteristics of communication in individuals and in different specialties. RESULTS: We coded a total of 662 call outs. The frequency and type of initiation and receipt of communication events varied between clinical specialties (P < .001). Surgeons and nurses initiated fewer and received more communication events than anesthesiologists. For the average participant, directed communication increased the likelihood of check back by at least 50% (P = .021) in periods preceding acute changes in the clinical setting, and exerted no significant effect in periods after acute changes in the clinical situation. CONCLUSION: Communication patterns vary by specialty during a simulated operative emergency, and the effect of directed communication in eliciting a response depends on the clinical status of the patient. Operative training programs should emphasize the importance of quality communication in the period immediately after an acute change in the clinical setting of a patient and recognize that communication patterns and needs vary between members of multidisciplinary operative teams.