Validating Parallel-Forms Tests for Assessing Anesthesia Resident Knowledge.

We created a serious game to teach first year anesthesiology (CA-1) residents to perform general anesthesia for cesarean delivery. We aimed to investigate resident knowledge gains after playing the game and having received one of 2 modalities of debriefing. We report on the development and validation of scores from parallel test forms for criterion-referenced interpretations of resident knowledge. The test forms were intended for use as pre- and posttests for the experiment. Validation of instruments measuring the study's primary outcome was considered essential for adding rigor to the planned experiment, to be able to trust the study's results. Parallel, multiple-choice test forms development steps included: (1) assessment purpose and population specification; (2) content domain specification and writing/selection of items; (3) content validation by experts of paired items by topic and cognitive level; and (4) empirical validation of scores from the parallel test forms using Classical Test Theory (CTT) techniques. Field testing involved online administration of 52 shuffled items from both test forms to 24 CA-1's, 21 second-year anesthesiology (CA-2) residents, 2 fellows, 1 attending anesthesiologist, and 1 of unknown rank at 3 US institutions. Items from each form yielded near-normal score distributions, with similar medians, ranges, and standard deviations. Evaluations of CTT item difficulty (item p values) and discrimination (D) indices indicated that most items met assumptions of criterion-referenced test design, separating experienced from novice residents. Experienced residents performed better on overall domain scores than novices (P < .05). Kuder-Richardson Formula 20 (KR-20) reliability estimates of both test forms were above the acceptability cut of .70, and parallel forms reliability estimate was high at .86, indicating results were consistent with theoretical expectations. Total scores of parallel test forms demonstrated item-level validity, strong internal consistency and parallel forms reliability, suggesting sufficient robustness for knowledge outcomes assessments of CA-1 residents.

Development and Validation of an Automated Tool to Retrieve and Curate Faculty Publications of Academic Departments.

Introduction Academic departments need to monitor their faculty's academic productivity for various purposes, such as reporting to the medical school dean, assessing the allocation of non-clinical research time, evaluating for rank promotion, and reporting to the Accreditation Council for Graduate Medical Education (ACGME). Our objective was to develop and validate a simple method that automatically generates query strings to identify and process distinct department faculty publications listed in PubMed and Scopus. Methods We created a macro-enabled Excel workbook (Microsoft, Redmond, WA) to automate the retrieval of faculty publications from the PubMed and Scopus bibliometric databases (available at https://bit.ly/get-pubs). Where the returned reference includes the digital object identifier (doi), a link is provided in the workbook. Duplicate publications are removed automatically, and false attributions are managed. Results At the University of Miami, between 2020 and 2021, there were 143 anesthesiology faculty-authored publications with a PubMed identifier (PMID), 95.8% identified by the query and 4.2% missed. At Vanderbilt University Medical Center, between 2019 and 2021, there were 760 anesthesiology faculty-authored publications with a PMID, 94.3% identified by the query and 5.7% missed. Recall, precision, and the F1 score were all above 93% at both medical centers. Conclusions We developed a highly accurate, simple, transportable, scalable method to identify publications in PubMed and Scopus authored by anesthesiology faculty. Manual checking and faculty feedback are required because not all names can be disambiguated, and some references are missed. This process can greatly reduce the burden of curating a list of faculty publications. The methodology applies to other academic departments that track faculty publications.

Natural Language Mapping of Electrocardiogram Interpretations to a Standardized Ontology.

BACKGROUND: Interpretations of the electrocardiogram (ECG) are often prepared using software outside the electronic health record (EHR) and imported via an interface as a narrative note. Thus, natural language processing is required to create a computable representation of the findings. Challenges include misspellings, nonstandard abbreviations, jargon, and equivocation in diagnostic interpretations. OBJECTIVES: Our objective was to develop an algorithm to reliably and efficiently extract such information and map it to the standardized ECG ontology developed jointly by the American Heart Association, the American College of Cardiology Foundation, and the Heart Rhythm Society. The algorithm was to be designed to be easily modifiable for use with EHRs and ECG reporting systems other than the ones studied. METHODS: An algorithm using natural language processing techniques was developed in structured query language to extract and map quantitative and diagnostic information from ECG narrative reports to the cardiology societies' standardized ECG ontology. The algorithm was developed using a training dataset of 43,861 ECG reports and applied to a test dataset of 46,873 reports. RESULTS: Accuracy, precision, recall, and the F1-measure were all 100% in the test dataset for the extraction of quantitative data (e.g., PR and QTc interval, atrial and ventricular heart rate). Performances for matches in each diagnostic category in the standardized ECG ontology were all above 99% in the test dataset. The processing speed was approximately 20,000 reports per minute. We externally validated the algorithm from another institution that used a different ECG reporting system and found similar performance. CONCLUSION: The developed algorithm had high performance for creating a computable representation of ECG interpretations. Software and lookup tables are provided that can easily be modified for local customization and for use with other EHR and ECG reporting systems. This algorithm has utility for research and in clinical decision-support where incorporation of ECG findings is desired.

Drug Calculation Errors in Anesthesiology Residents and Faculty: An Analysis of Contributing Factors.

BACKGROUND: Limited data exist regarding computational drug error rates in anesthesia residents and faculty. We investigated the frequency and magnitude of computational errors in a sample of anesthesia residents and faculty. METHODS: With institutional review board approval from 7 academic institutions in the United States, a 15-question computational test was distributed during rounds. Error rates and the magnitude of the errors were analyzed according to resident versus faculty, years of practice (or residency training), duration of sleep, type of question, and institution. RESULTS: A total of 371 completed the test: 209 residents and 162 faculty. Both groups committed 2 errors (median value) per test, for a mean error rate of 17.0%. Twenty percent of residents and 25% of faculty scored 100% correct answers. The error rate for postgraduate year 2 residents was less than for postgraduate year 1 (P = .012). The error rate for faculty increased with years of experience, with a weak correlation (R = 0.22; P = .007). The error rates were independent of the number of hours of sleep. The error rate for percentage-type questions was greater than for rate, dose, and ratio questions (P = .001). The error rates varied with the number of operations needed to calculate the answer (P < .001). The frequency of large errors (100-fold greater or less than the correct answer) by residents was twice that of faculty. Error rates varied among institutions ranged from 12% to 22% (P = .021). CONCLUSIONS: Anesthesiology residents and faculty erred frequently on a computational test, with junior residents and faculty with more experience committing errors more frequently. Residents committed more serious errors twice as frequently as faculty.

Development of a Multiple-Choice Test for Novice Anesthesia Residents to Evaluate Knowledge Related to Management of General Anesthesia for Urgent Cesarean Delivery.

BACKGROUND: Teaching trainees the knowledge and skills to perform general anesthesia (GA) for cesarean delivery (CD) requires innovative strategies, as they may never manage such cases in training. We used a multistage design process to create a criterion-referenced multiple-choice test as an assessment tool to evaluate CA1's knowledge related to this scenario. METHODS: Three faculty created 33 questions, categorized as: (1) physiologic changes of pregnancy (PCP), (2) pharmacology (PHA), (3) anesthetic implications of pregnancy (AIP), and (4) crisis resource management principles (CRM). A Delphi process (3 rounds) provided content validation. In round 1, experts (n = 15) ranked questions on a 7-point Likert scale. Questions ranked ≥ 5 in importance by ≥ 70% of experts were retained. Five questions were eliminated, several were revised, and 1 added. In round 2, consensus (N = 14) was reached in all except 7 questions. In round 3 (N = 14), all questions stabilized. A pilot test of the 29-question instrument evaluating internal consistency, reliability, convergent validity, and item analysis was conducted with the July CA1 classes at our institution after a lecture on GA for CD (n = 26, "instructed group") and another institution with no lecture (n = 26, "uninstructed group"), CA2s (N = 17), and attendings (N = 10). RESULTS: Acceptable internal consistency and reliability was demonstrated (ρ = 0.67). Convergent validity coefficients between the CA1 uninstructed and instructed group suggested theoretical meaningfulness of the 4 sub-scales: PCP correlated at 0.29 with PHA, 0.35 with CRM, and 0.25 with AIP. PHA correlated with CRM and AIP at 0.23 and 0.28, respectively. The correlation between CRM and AIP was 0.29. CONCLUSION: The test produces moderately reliable scores to assess CA1s' knowledge related to GA for urgent CD.

Development and Validation of an Assessment of Regional Anesthesia Ultrasound Interpretation Skills.

BACKGROUND: Interpretation of ultrasound images and knowledge of anatomy are essential skills for ultrasound-guided peripheral nerve blocks. Competency-based educational models promoted by the Accreditation Council for Graduate Medical Education require the development of assessment tools for the achievement of different competency milestones to demonstrate the longitudinal development of skills that occur during training. METHODS: A rigorous study guided by psychometric principles was undertaken to identify and validate the domains and items in an assessment of ultrasound interpretation skills for regional anesthesia. A survey of residents, academic faculty, and community anesthesiologists, as well as video recordings of experts teaching ultrasound-guided peripheral nerve blocks, was used to develop short video clips with accompanying multiple choice-style questions. Four rounds of pilot testing produced a 50-question assessment that was subsequently administered online to residents, fellows, and faculty from multiple institutions. RESULTS: Test results from 90 participants were analyzed with Item Response Theory model fitting indicating that a 47-item subset of the test fits the model well (P = 0.11). There was a significant linear relation between expected and predicted item difficulty (P < 0.001). Overall test scores increased linearly with higher levels of formal anesthesia training, regional anesthesia training, number of ultrasound-guided blocks performed per year, and a self-rating of regional anesthesia skill (all P < 0.001). CONCLUSIONS: This study provides evidence for the reliability, content validity, and construct validity of a 47-item multiple choice-style online test of ultrasound interpretation skills for regional anesthesia, which can be used as an assessment of competency milestone achievement in anesthesiology training.

Considerations for anesthesia staffing in a trauma center: new standards, education, and safety.

PURPOSE OF REVIEW: A review of recent updates to trauma anesthesia service requirements in the USA and UK, the evolving role of the trauma anesthesiologist, and opportunities for education and training. Considerations of cost and safety for staffing arrangements are discussed. RECENT FINDINGS: Certifying and specialty organizations have recently escalated the availability requirements and training recommendations for anesthesiology services in trauma centers. SUMMARY: There is a growing recognition that trauma anesthesiology represents a distinct area of subspecialty knowledge. Anesthesiology specialty organizations advocate for trauma-specific knowledge and training for trauma anesthesia providers. Requiring the in-house presence of anesthesia providers in level I and level II trauma centers may impose significant costs on medical centers that do not currently provide those services.

Frequency of interactions and hand disinfections among anesthesiologists while providing anesthesia care in the operating room: induction versus maintenance.

We evaluated the behaviors of anesthesiologists during induction and maintenance of anesthesia. Contacts with surfaces occurred a mean (±standard error) of 154.8 ± 7.7 and 60 ± 3.1 times per hour during induction and maintenance, respectively (P < .0001). Hand hygiene events were 1.8 ± 0.27 per hour during induction versus 1.19 ± 0.27 during maintenance (P = .018).

Randomized crossover study evaluating the effect of a hand sanitizer dispenser on the frequency of hand hygiene among anesthesiology staff in the operating room.

Forty anesthesia providers were evaluated with and without hand sanitizer dispensers present on the anesthesia machine. Having a dispenser increased the frequency of hand hygiene only from 0.5 to 0.8 events per hour (P = .01). Other concomitant interventions are needed to further increase hand hygiene frequency among anesthesia providers.

Emotional intelligence and the relationship to resident performance: a multi-institutional study.

STUDY OBJECTIVE: To test the hypothesis that emotional intelligence, as measured by a BarOn Emotional Quotient Inventory (EQ-i), the 125-item version personal inventory (EQ-i:125), correlates with resident performance. DESIGN: Survey (personal inventory) instrument. SETTING: Five U.S. academic anesthesiology residency programs. PARTICIPANTS: Postgraduate year (PGY) 2, 3, and 4 residents enrolled in university-based anesthesiology residency programs. MEASUREMENTS: Residents confidentially completed the BarOn EQ-i:125 personal inventory. The deidentified resident evaluations were sent to the principal investigator of a separate data collection study for data analysis. Data collected from the inventory were correlated with daily evaluations of the residents by residency program faculty. Results of the individual BarOn EQ-i:125 and daily faculty evaluations of the residents were compiled and analyzed. MAIN RESULTS: Univariate correlation analysis and multivariate canonical analysis showed that some aspects of the BarOn EQ-i:125 were significantly correlated with, and likely to be predictors of, resident performance. CONCLUSIONS: Emotional intelligence, as measured by the BarOn EQ-i personal inventory, has considerable promise as an independent indicator of performance as an anesthesiology resident.

Trauma in the elderly: considerations for anesthetic management.

The volume of geriatric trauma patients is expected to increase significantly in coming years. Recognition of severe injuries may be delayed because they are less likely to mount classic symptoms of hemodynamic instability. Head injuries of any severity may place geriatric patients at increased risk of mortality, but there are currently no geriatric-specific treatment recommendations that differ from usual adult guidelines. Our understanding of best practices in geriatric trauma and anesthesia care continues to expand, as it does in all other areas of medicine.