Competence in Decision Making: Setting Performance Standards for Critical Care.

BACKGROUND: Health care professionals must be able to make frequent and timely decisions that can alter the illness trajectory of intensive care patients. A competence standard for this ability is difficult to establish yet assuring practitioners can make appropriate judgments is an important step in advancing patient safety. We hypothesized that simulation can be used effectively to assess decision-making competence. To test our hypothesis, we used a "standard-setting" method to derive cut scores (standards) for 16 simulated ICU scenarios targeted at decision-making skills and applied them to a cohort of critical care trainees. METHODS: Panelists (critical care experts) reviewed digital audio-video performances of critical care trainees managing simulated critical care scenarios. Based on their collectively agreed-upon definition of "readiness" to make decisions in an ICU setting, each panelist made an independent judgment (ready, not ready) for a large number of recorded performances. The association between the panelists' judgments and the assessment scores was used to derive scenario-specific performance standards. RESULTS: For all 16 scenarios, the aggregate panelists' ratings (ready/not ready for independent decision making) were positively associated with the performance scores, permitting derivation of performance standards for each scenario. CONCLUSIONS: Minimum competence standards for high-stakes decision making can be established through standard-setting techniques. We effectively identified "front-line" providers who are, or are not, ready to make independent decisions in an ICU setting. Our approach may be used to assure stakeholders that clinicians are competent to make appropriate judgments. Further work is needed to determine whether our approach is effective in simulation-based assessments in other domains.

Simulation-Based Assessment of Critical Care "Front-Line" Providers.

OBJECTIVES: Develop a standardized simulation method to assess clinical skills of ICU providers. DESIGN: Simulation assessment. SETTING: Simulation laboratory. SUBJECTS: Residents, Critical Care Medicine fellows, acute care nurse practitioner students. INTERVENTIONS: Performance scoring in scenarios from multiple Critical Care Medicine competency domains. MEASUREMENTS AND MAIN RESULTS: Three-hundred eighty-four performances by 48 participants were scored using checklists (% correct) and holistic "global" ratings (1 [unprepared] to 9 [expert]). One-hundred eighty were scored by two raters. Mean checklist and global scores (± SD) ranged from 65.0% (± 16.3%) to 84.5% (± 17.3%) and 4.7 (± 1.4) to 7.2 (± 1.2). Checklist and global scores for Critical Care Medicine fellows and senior acute care nurse practitioner students (Experienced group, n = 26) were significantly higher than those for the Novice acute care nurse practitioner students (Novice group, n = 14) (75.6% ± 15.6% vs 68.8% ± 21.0% and 6.1 ± 1.6 vs 5.4 ± 1.5, respectively; p < 0.05). Residents (Intermediate group, n = 8) scored between the two (75.4% ± 18.3% and 5.7 ± 1.7). 38.5% of the Experienced group scored in the top quartile for mean global score, compared with 12.5% of the Intermediate and 7.1% of the Novice groups. Conversely, 50% of the Novice group scored in the lower quartile (< 5.3), compared with 37.5% of the Intermediate and 11.5% of the Experienced groups. Psychometric analyses yielded discrimination values greater than 0.3 for most scenarios and reliability for the eight-scenario assessments of 0.51 and 0.60, with interrater reliability of 0.71 and 0.75, for checklist and global scoring, respectively. CONCLUSIONS: The simulation assessments yielded reasonably reliable measures of Critical Care Medicine decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.

A Capstone Course for Senior Medical Students: from Innovative Elective to Required Core Curriculum.

We describe our institution's development and implementation of our Capstone course from a small elective course to the only required fourth-year course. The course's structure evolved from mostly didactic to one including various workshops and simulation sessions. Course content has become increasingly specialty-specific. Implementation requires high faculty and resident involvement. Evaluations indicate a positive impact of the course on participants' self-reported confidence and residency preparedness. Assessment remains pass/fail with more specialty-specific questions. As steadily increasing numbers of medical schools are developing transition to residency courses, we share our Capstone course's evolution and lessons learned over the past nine years. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-023-01880-2.

Using human simulation in the oncology clinical practice setting.

Patients are referred to the tertiary care teaching institution with diverse and sometimes rare diagnoses requiring complex therapy. Assuring that nurses in the oncology setting are delivering quality nursing care requires educational support that integrates evidence-based teaching methods. Clinical simulation holds promise for novice and experienced nurses to see the "big picture" associated with acutely ill oncology patients. Simulation scenarios focusing on commonly experienced oncologic emergencies are described, and practical information for the development and implementation of a simulation program is provided.

Decision-making skills improve with critical care training: Using simulation to measure progress.

PURPOSE: Health care professionals are expected to acquire decision-making skills during their training, but few methods are available to assess progress in acquiring these essential skills. The purpose of this study was to determine whether a simulation methodology could be used to assess whether decision-making skills improve during critical care training. MATERIALS AND METHODS: Sixteen simulated scenarios were designed to assess a critical care provider's ability to make decisions in the care of a critical ill patient. Seventeen (17) critical care providers managed 8 of the scenarios early during their training and then managed a second set of 8 scenarios (T2) at the conclusion of their training. RESULTS: Provider's mean global scenario scores (0-9) increased significantly fromT1 and T2 (5.64 ±â€¯0.74) and (6.54 ±â€¯0.64) with a large effect size (1.3). Acute care nurse practitioners and fellows achieved similar overall scores at the conclusion of their training (ACNP 6.43 ±â€¯0.57; Fellows 6.64 ±â€¯0.72). CONCLUSIONS: These findings provide evidence to support the validity of a simulation-based method to assess progress in decision-making skills. A simulation methodology could be used to establish a performance standard that determined a provider's ability to make independent decisions.

Outcomes of a proficiency-based skills curriculum at the beginning of the fourth year for senior medical students entering surgery.

INTRODUCTION: We hypothesized that a proficiency-based curriculum administered early in the fourth year to senior medical students (MS4) would achieve outcomes comparable to a similar program administered during surgical internship. METHODS: MS4 (n = 18) entering any surgical specialty enrolled in a proficiency-based skills curriculum at the beginning of the fourth year that included suturing/knot-tying, on-call problems, laparoscopic, and other skills (urinary catheter, sterile prep/drape, IV placement, informed consent, electrosurgical use). Assessment was at 4-12 weeks after training by a modified Objective Structured Assessment of Technical Skills (OSATS). Suturing and knot tying tasks were assessed by time and OSATS technical proficiency (TP) scores (1 [novice], 3 [proficient], 5 [expert]). Outcomes were compared with PGY-1 residents who received similar training at the beginning of internship and assessment 4-12 weeks later. Data are presented as mean values ± standard deviation; statistical significance was assessed by Student's t test. RESULTS: Fifteen of 18 MS4 (83%) reached proficiency on all 15 tasks, and 2 others were proficient on all but 1 laparoscopic task. Compared with PGY-1s, MS4 were significantly faster for 3 of 5 suturing and tying tasks and total task time (547 ± 63 vs 637 ± 127 s; P < .05). Mean TP scores were similar for both groups (MS4, 3.4 ± 0.5 vs PGY-1, 3.1 ± .57; P = NS). MS4 OSATS scores were higher for IV placement, informed consent, and urinary catheter placement, but lower for prep and drape and for management of on-call problems. CONCLUSION: MS4 who participate in a proficiency-based curriculum taught early in the fourth year are able to meet proficiency targets in a high percentage of cases. This approach should better prepare MS4 for surgical internship.

Decision making in trauma settings: simulation to improve diagnostic skills.

INTRODUCTION: In the setting of acute injury, a wrong, missed, or delayed diagnosis can impact survival. Clinicians rely on pattern recognition and heuristics to rapidly assess injuries, but an overreliance on these approaches can result in a diagnostic error. Simulation has been advocated as a method for practitioners to learn how to recognize the limitations of heuristics and develop better diagnostic skills. The objective of this study was to determine whether simulation could be used to provide teams the experiences in managing scenarios that require the use of heuristic as well as analytic diagnostic skills to effectively recognize and treat potentially life-threatening injuries. METHODS: Ten scenarios were developed to assess the ability of trauma teams to provide initial care to a severely injured patient. Seven standard scenarios simulated severe injuries that once diagnosed could be effectively treated using standard Advanced Trauma Life Support algorithms. Because diagnostic error occurs more commonly in complex clinical settings, 3 complex scenarios required teams to use more advanced diagnostic skills to uncover a coexisting condition and treat the patient. Teams composed of 3 to 5 practitioners were evaluated in the performance of 7 (of 10) randomly selected scenarios (5 standard, 2 complex). Expert rates scored teams using standardized checklists and global scores. RESULTS: Eighty-three surgery, emergency medicine, and anesthesia residents constituted 21 teams. Expert raters were able to reliably score the scenarios. Teams accomplished fewer checklist actions and received lower global scores on the 3 analytic scenarios (73.8% [12.3%] and 5.9 [1.6], respectively) compared with the 7 heuristic scenarios (83.2% [11.7%] and 6.6 [1.3], respectively; P < 0.05 for both). Teams led by more junior residents received higher global scores on the analytic scenarios (6.4 [1.3]) than the more senior team leaders (5.3 [1.7]). CONCLUSIONS: This preliminary study indicates that teams led by more senior residents received higher scores when managing heuristic scenarios but were less effective when managing the scenarios that require a more analytic approach. Simulation can be used to provide teams with decision-making experiences in trauma settings and could be used to improve diagnostic skills as well as study the decision-making process.

Retention of suturing and knot-tying skills in senior medical students after proficiency-based training: Results of a prospective, randomized trial.

PURPOSE: We evaluated suturing skills performance and retention in senior medical students (MS4) at the beginning of their fourth year and 7 months later. METHODS: MS4 students entering a surgery specialty were randomized to a proficiency-based suturing/knot-tying curriculum at the beginning of fourth year (Intervention, n = 11) versus no training (Control, n = 10). Time and technical proficiency (TP, proficiency ≥3) were assessed at baseline and 7 months. Performance was compared with past "Boot-Camp" MS4, categorical PGY-1 interns and PGY-2 residents. Data are mean ± SD. RESULTS: At baseline, Intervention and Control MS4 had similar total task times (848 ± 199 vs 845 ± 209 seconds) and TP scores (1.8 ± 0.15 vs 1.8 ± 0.3). At 7 months, Intervention MS4 total task times were faster (549 ± 80 vs 719 ± 151 seconds, P < .01) and mean TP scores greater (3.3 ± 0.6 vs 2.1 vs 0.4, P < .001) than Control MS4. Intervention MS4 also performed better at 7 months than Boot-Camp MS4 (662 ± 171 seconds and 2.6 ± 0.5, P < .04) and were similar to PGY-1 interns (601 ± 74 seconds, TP 2.7 ± 0.7 seconds) and end of PGY-2 residents (475 ± 81 seconds and 3.6 ± 0.3 seconds). CONCLUSION: A proficiency-based suturing and knot-tying curriculum taught early in the fourth year results in improved MS4 performance compared with no training or a traditional "boot camp" program.

Video review using a reliable evaluation metric improves team function in high-fidelity simulated trauma resuscitation.

OBJECTIVE: To demonstrate that instruction of proper team function can occur using high-fidelity simulated trauma resuscitation with video-assisted debriefing and that this process can be integrated rapidly into a standard general surgery curriculum. DESIGN: The rater reliability of our team metric was assessed by having physicians and nonphysicians rate the same video-recorded trauma simulations at intervals in time. To assess the effectiveness of video debriefing, subjects participated in a 3-week trauma team training course that consisted of 2 video-recorded simulation sessions, each approximately 2 hours in length separated by a 90-minute debriefing session. To assess the impact of the debriefing session, video recordings of participants performing resuscitations before and after the debriefing were reviewed by a panel of blinded traumatologists and graded using our team evaluation instrument. SETTING: The study took place at the high-fidelity simulation center at a large, urban academic training hospital. PARTICIPANTS: All 11 PGY-2 general surgery and combined general surgery and plastic surgery residents at our institution. RESULTS: Our instrument was found to have high interrater correlation (interclass correlation coefficient [ICC], 0.926; 95% confidence interval, 0.893-0.953). Initially, residents were either unsure as to their competency to serve as team leader (70%) or felt they were not competent to serve as team leader (30%). Ninety percent of residents found the video debriefing very to extremely helpful in improving team function and clinical competency. All participants felt more competent as both team leaders and team members because of the video debriefing. The mean team function score improved significantly after video debriefing (4.39 [±0.3] vs 5.45 [±0.4] prevideo vs postvideo review, p < 0.05). CONCLUSIONS: Video review with debriefing is an effective means of teaching team competencies and improving team function in simulated trauma resuscitation. This strategy can be integrated readily into the surgical curriculum analogous to other applications of simulation technology.

Do preclinical background and clerkship experiences impact skills performance in an accelerated internship preparation course for senior medical students?

BACKGROUND: Dedicated skills courses may help to prepare 4th-year medical students for surgical internships. The purpose of this study was to analyze the factors that influence the preparedness of 4th-year medical students planning a surgical career, and the role that our skills course plays in that preparedness. METHODS: A comprehensive skills course for senior medical students matching in a surgical specialty was conducted each spring from 2006 through 2009. Students were surveyed for background skills, clerkship experience, and skills confidence levels (1-5 Likert scale). Assessment included 5 suturing and knot-tying tasks pre- and postcourse and a written examination. Data are presented as mean values ± standard deviations; statistical analyses were by 2-tailed t test, linear regression, and analysis of variance. RESULTS: Sixty-five 4th-year students were enrolled; most common specialties were general surgery (n = 22) and orthopedics (n = 16). Thirty-five students were elite musicians (n = 16) or athletes (n = 19) and 8 regular videogamers. Suturing task times improved significantly from pre- to postcourse for all 5 tasks (total task times pre, 805 ± 202 versus post, 627 ± 168 seconds [P < .0001]) as did confidence levels for 8 skills categories, including management of on-call problems (P < .05). Written final examination proficiency (score ≥70%) was achieved by 81% of students. Total night call experience 3rd year was 23.3 ± 10.7 nights (7.3 ± 4.3 surgical call) and 4th year 10.5 ± 7.4 nights (7.2 ± 6.8 surgical call). Precourse background variables significantly associated with outcome measures were athletics with precourse suturing and 1-handed knot tying (P < .05); general surgery specialty and instrument tying (P = .012); suturing confidence levels and precourse suturing and total task times (P = .024); and number of nonsurgical call nights with confidence in managing acute on-call problems (P = .028). No significant correlation was found between these variables and postcourse performance. CONCLUSION: Completion of an accelerated skills course results in comparable levels of student performance postcourse across a variety of preclinical backgrounds and clerkship experiences.

Team behavior during trauma resuscitation: a simulation-based performance assessment.

INTRODUCTION: Trauma resuscitations require a coordinated response from a diverse group of health care providers. Currently, there are no widely accepted methods of assessing team effectiveness in this setting. Simulation affords a method to assess team effectiveness. The purpose of this study was to use a simulation setting to develop a specialized assessment instrument for team response in trauma resuscitation. METHODS: We developed our assessment instrument using clinical simulation. Four teams of 3 postgraduate year-2 surgical trainees in conjunction with scripted confederates were videotaped enacting 6 separate trauma resuscitation scenarios that mirrored clinical conditions encountered at our level 1 trauma center. Ten of the resulting videotaped scenarios represented a spectrum of team behavior (ineffective to effective) and were scored by 8 experienced clinicians using the Mayo High Performance Teamwork Scale. RESULTS: Based in part on the Mayo High Performance Teamwork Scale, we created a prototype trauma team assessment tool consisting of 7 attributes that we scored in binary fashion (present/absent). We validated this prototype by assigning a normalized ranking score to each of the 10 scenarios based on the score supplied by each rater. The presence/absence of the 7 attributes varied significantly among scenarios (52.5% to 93.8%; P < .001). Median scores differed significantly comparing the 5 lowest-ranking scenarios with the 5 highest-ranking scenarios (P < .001). CONCLUSION: Our prototype instrument may be effective at assessing team effectiveness during trauma resuscitations. This instrument may prove useful for assessing team competency skills, providing timely feedback to teams, and examining the relationship between effective team function and clinically important outcomes. Further, it may be applicable to other high-acuity, time-sensitive clinical situations.

Setting performance standards for mannequin-based acute-care scenarios: an examinee-centered approach.

BACKGROUND: In medicine, standard setting methodologies have been developed for both selected-response and performance-based assessments. For simulation-based tasks, research efforts have been directed primarily at assessments that incorporate standardized patients. Mannequin-based evaluations often demand complex, time-sensitive, hierarchically ordered, sequential actions that are difficult to evaluate and score. Moreover, collecting reliable proficiency judgments, necessary to estimate meaningful cut points, can be challenging. The purpose of this investigation was to explore whether expert judgments obtained using an examinee-centered standard setting method that was previously validated for standardized patient-based assessments could be used to set defensible standards for acute-care, mannequin-based scenarios. METHODS: Nineteen physicians were recruited to serve as panelists. For each of 12 simulation scenarios, between 8 and 10 performance samples (audio-video recordings), covering the expected ability continuum, were chosen for review. The performance samples were selected from a previously administered evaluation of postgraduate trainees. Based on a consensus definition of readiness to enter unsupervised practice, the panelists made independent judgments of each performance. For each scenario, the association between the panelists' judgments and the assessment scores was summarized and used to estimate a scenario-specific cut score. RESULTS: For 9 of the scenarios, there was at least a moderately strong relationship between the aggregate panelists' rating and the performance scores, thus allowing for estimation of meaningful numeric standards. For the other 3 scenarios, the aggregate decision rules used by the panelists did not correspond with the achievement measures. For scenarios independently rated by split panels, the estimated cut scores were similar. CONCLUSIONS: An examinee-centered approach, using aggregate expert judgments of audio-video performances, was suitable for setting standards on most acute-care, mannequin-based scenarios. It is necessary, however, to have valid scores for the chosen scenarios and to sample performances across the ability spectrum.

Performance of residents and anesthesiologists in a simulation-based skill assessment.

BACKGROUND: Anesthesiologists and anesthesia residents are expected to acquire and maintain skills to manage a wide range of acute intraoperative anesthetic events. The purpose of this study was to determine whether an inventory of simulated intraoperative scenarios provided a reliable and valid measure of anesthesia residents' and anesthesiologists' skill. METHODS: Twelve simulated acute intraoperative scenarios were designed to assess the performance of 64 residents and 35 anesthesiologists. The participants were divided into four groups based on their training and experience. There were 31 new CA-1, 12 advanced CA-1, and 22 CA-2/CA-3 residents as well as a group of 35 experienced anesthesiologists who participated in the assessment. Each participant managed a set of simulated events. The advanced CA-1 residents, CA-2/CA-3 residents, and 35 anesthesiologists managed 8 of 12 intraoperative simulation exercises. The 31 CA-1 residents each managed 3 intraoperative scenarios. RESULTS: The new CA-1 residents received lower scores on the simulated intraoperative events than the other groups of participants. The advanced CA-1 residents, CA-2/CA-3 residents, and anesthesiologists performed similarly on the overall assessment. There was a wide range of scores obtained by individuals in each group. A number of the exercises were difficult for the majority of participants to recognize and treat, but most events effectively discriminated among participants who achieved higher and lower overall scores. CONCLUSION: This simulation-based assessment provided a valid method to distinguish the skills of more experienced anesthesia residents and anesthesiologists from residents in early training. The overall score provided a reliable measure of a participant's ability to recognize and manage simulated acute intraoperative events. Additional studies are needed to determine whether these simulation-based assessments are valid measures of clinical performance.

Reliability and validity of a simulation-based acute care skills assessment for medical students and residents.

BACKGROUND: Medical students and residents are expected to be able to manage a variety of critical events after training, but many of these individuals have limited clinical experiences in the diagnosis and treatment of these conditions. Life-sized mannequins that model critical events can be used to evaluate the skills required to manage and treat acute medical conditions. The purpose of this study was to develop and test simulation exercises and associated scoring methods that could be used to evaluate the acute care skills of final-year medical students and first-year residents. METHODS: The authors developed and tested 10 simulated acute care situations that clinical faculty at a major medical school expects graduating physicians to be able to recognize and treat at the conclusion of training. Forty medical students and residents participated in the evaluation of the exercises. Four faculty members scored the students/residents. RESULTS: The reliability of the simulation scores was moderate and was most strongly influenced by the choice and number of simulated encounters. The validity of the simulation scores was supported through comparisons of students'/residents' performances in relation to their clinical backgrounds and experience. CONCLUSION: Acute care skills can be validly and reliably measured using a simulation technology. However, multiple simulated encounters, covering a broad domain, are needed to effectively and accurately estimate student/resident abilities in acute care settings.

An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation.

PURPOSE: This investigation aimed to explore the measurement properties of scores from a patient simulator exercise. METHODS: Analytic and holistic scores were obtained for groups of medical students and residents. Item analysis techniques were used to explore the nature of specific examinee actions. Interrater reliability was calculated. Scores were contrasted for third year medical students, fourth year medical students and emergency department residents. RESULTS: Interrater reliabilities for analytic and holistic scores were 0.92 and 0.81, respectively. Based on item analysis, proper timing and sequencing of actions discriminated between low- and high-ability examinees. In general, examinees with more advanced training obtained higher scores on the simulation exercise. CONCLUSION: Reliable and valid measures of clinical performance can be obtained from a trauma simulation provided that care is taken in the development and scoring of the scenario.

Acute care skills in anesthesia practice: a simulation-based resident performance assessment.

BACKGROUND: A recurring initiative in graduate education is to find more effective methods to assess specialists' skills. Life-sized simulators could be used to assess the more complex skills expected in specialty practice if a curriculum of relevant exercises were developed that could be simply and reliably scored. The purpose of this study was to develop simulation exercises and associated scoring methods and determine whether these scenarios could be used to evaluate acute anesthesia care skills. METHODS: Twenty-eight residents (12 junior and 16 senior) managed three intraoperative and three postoperative simulation exercises. Trainees were required to make a diagnosis and intervention in a simulation encounter designed to recreate an acute perioperative complication. The videotaped performances were scored by six raters. Three raters used a checklist scoring system. Three faculty raters measured when trainees performed three key diagnostic or therapeutic actions during each 5-min scenario. These faculty also provided a global score using a 10-cm line with scores from 0 (unsatisfactory) to 10 (outstanding). The scenarios included (1) intraoperative myocardial ischemia, (2) postoperative anaphylaxis, (3) intraoperative pneumothorax, (4) postoperative cerebral hemorrhage with intracranial hypertension, (5) intraoperative ventricular tachycardia, and (6) postoperative respiratory failure. RESULTS: The high correlation among all of the scoring systems and small variance among raters' scores indicated that all of the scoring systems measured similar performance domains. Scenarios varied in their overall difficulty. Even though trainees who performed well on one exercise were likely to perform well in subsequent scenarios, the authors found that there were considerable differences in case difficulty. CONCLUSION: This study suggests that simulation can be used to measure more complex skills expected in specialty training. Similar to other studies that assess a broad content domain, multiple encounters are needed to estimate skill effectively and accurately.