Critical Appraisal of Anesthesiology Educational Research for 2020.

Background: This study reviews and appraises the articles published about anesthesiology education in 2020. The objective is to highlight high-quality evidence while showcasing articles with innovative ideas and high relevance to the practices of the anesthesiology education community. Methods: Three Ovid MEDLINE databases, Embase.com, ERIC, PsycINFO, and PubMed were searched, followed by a manual review of articles published in the highest impact factor journals in both the fields of anesthesiology and medical education. Abstracts were double screened, and quantitative articles were subsequently scored by 3 randomly assigned raters. Qualitative studies were scored by 2 raters. Two different rubrics were used for scoring quantitative and qualitative studies. In addition, reviewers rated each article on its overall quality to create an additional list of top articles based solely on the opinion of the reviewers. Results: A total of 2,491 citations were identified through the search criteria and the manual review. Of those, 61 articles met the inclusion criteria (57 quantitative and 4 qualitative). The top 12 quantitative papers and the top qualitative papers with the highest scores are reported and summarized. Conclusions: We found that teaching clinical procedures continues to be a topic of interest, with more studies of improved rigor identified. New trends in wellness studies and increasing attention to distance learning and technology-assisted instructional methods were additional topics covered over the year.

Critical Appraisal of Anesthesiology Educational Research for 2019.

Background: This study reviews and appraises the articles published about anesthesiology education in 2019. Through this critical appraisal, those interested in anesthesiology education are able to quickly review literature published during this year and explore innovative ways to improve education for all those involved in the practice of anesthesiology. Methods: Three Ovid MEDLINE databases, Embase.com, ERIC, and PsycINFO were searched followed by a manual review of articles published in the highest impact factor journals in both the fields of anesthesiology and medical education. Abstracts were double-screened and quantitative articles were subsequently scored by 3 randomly assigned raters. Qualitative studies were scored by 2 raters. Two different rubrics were used for scoring quantitative and qualitative studies; both allowed for scores ranging from 1 to 25. In addition, reviewers rated each article on its overall quality to create an additional list of top articles based solely on the opinion of the reviewers. Results: A total of 2374 unique citations were identified through the search criteria and the manual review. Of those, 70 articles met the inclusion criteria (62 quantitative and 8 qualitative). The top 12 quantitative papers and the top 2 qualitative papers with the highest scores were reported and summarized.Conclusions: This critical appraisal continues to be a useful tool for those working in anesthesiology education by highlighting the best research articles published over the year. Highlighting trends in medical education research in anesthesiology can help those in the field to think critically about the direction of this type of research.

Using Machine Learning to Evaluate Attending Feedback on Resident Performance.

BACKGROUND: High-quality and high-utility feedback allows for the development of improvement plans for trainees. The current manual assessment of the quality of this feedback is time consuming and subjective. We propose the use of machine learning to rapidly distinguish the quality of attending feedback on resident performance. METHODS: Using a preexisting databank of 1925 manually reviewed feedback comments from 4 anesthesiology residency programs, we trained machine learning models to predict whether comments contained 6 predefined feedback traits (actionable, behavior focused, detailed, negative feedback, professionalism/communication, and specific) and predict the utility score of the comment on a scale of 1-5. Comments with ≥4 feedback traits were classified as high-quality and comments with ≥4 utility scores were classified as high-utility; otherwise comments were considered low-quality or low-utility, respectively. We used RapidMiner Studio (RapidMiner, Inc, Boston, MA), a data science platform, to train, validate, and score performance of models. RESULTS: Models for predicting the presence of feedback traits had accuracies of 74.4%-82.2%. Predictions on utility category were 82.1% accurate, with 89.2% sensitivity, and 89.8% class precision for low-utility predictions. Predictions on quality category were 78.5% accurate, with 86.1% sensitivity, and 85.0% class precision for low-quality predictions. Fifteen to 20 hours were spent by a research assistant with no prior experience in machine learning to become familiar with software, create models, and review performance on predictions made. The program read data, applied models, and generated predictions within minutes. In contrast, a recent manual feedback scoring effort by an author took 15 hours to manually collate and score 200 comments during the course of 2 weeks. CONCLUSIONS: Harnessing the potential of machine learning allows for rapid assessment of attending feedback on resident performance. Using predictive models to rapidly screen for low-quality and low-utility feedback can aid programs in improving feedback provision, both globally and by individual faculty.

Critical Appraisal of Anesthesiology Educational Research for 2018.

BACKGROUND: Through a critical appraisal of the education research in anesthesiology, this article enables those interested in the field to read the high-quality articles for the past year and possibly implement these evidence-based interventions and concepts into practice. This study reviews and appraises all medical education studies published in 2018 in anesthesiology and summarizes the highest-rated articles evaluated. METHODS: Three Ovid MEDLINE databases, Embase.com, ERIC, PsycINFO, and PubMed were searched followed by a manual review of articles published in the highest impact factor journals in both the fields of anesthesiology and medical education. Abstracts were double-screened and quantitative articles subsequently scored by 3 randomly assigned raters. Qualitative studies were scored by 2 raters. Two different rubrics were used for scoring quantitative and qualitative studies, both allowed for scores ranging from 1 to 25. RESULTS: A total of 888 unique citations were identified through the search criteria. Of those, 39 articles met the inclusion criteria (36 quantitative and 3 qualitative). The top 11 quantitative papers and the top qualitative paper with the highest scores were reported and summarized. CONCLUSIONS: As the second article to critically review the literature available for education in anesthesiology, we are able to add to this annual series to help further disseminate the articles of the highest quality in anesthesiology education. Because this is only the second year, we can only report on initial suggestions of trends that we hope will help guide future research.

Critical Appraisal of Anesthesiology Educational Research for 2017.

Background Critical appraisals provide a method for establishing the status of an area of study or evaluating the effectiveness of literature within it. The purpose of this study was to review and appraise studies published in 2017 on medical education in anesthesiology and to provide summaries of the highest-quality medical education research articles in the field. Methods Three Ovid MEDLINE databases, Embase.com, Education Resources Information Center (ERIC), and PsycINFO, were searched followed by a manual review of articles published in the highest impact factor journals in both the fields of anesthesiology and medical education. Abstracts were double-screened and quantitative articles subsequently scored by three randomly assigned raters. Qualitative studies were scored by two raters. Two different rubrics were used for scoring quantitative and qualitative studies, both allowed for scores ranging from 1-25. Results A total of 864 unique citations were identified through the search criteria. Of those, 62 articles met the inclusion criteria, with 59 quantitative and three qualitative. The top 10 papers with the highest scores were reported and summarized. Discussion As the first article to critically review the literature available for education in anesthesiology, we hope that this study will serve as the first manuscript in an annual series that will help individuals involved in anesthesiology education gain an understanding of the highest-quality research in the field. Once this process is repeated, trends can be tracked and serve as a resource to educators and researchers in anesthesiology for years to come.

Results of a Flipped Classroom Teaching Approach in Anesthesiology Residents.

BACKGROUND: In a flipped classroom approach, learners view educational content prior to class and engage in active learning during didactic sessions. OBJECTIVE: We hypothesized that a flipped classroom improves knowledge acquisition and retention for residents compared to traditional lecture, and that residents prefer this approach. METHODS: We completed 2 iterations of a study in 2014 and 2015. Institutions were assigned to either flipped classroom or traditional lecture for 4 weekly sessions. The flipped classroom consisted of reviewing a 15-minute video, followed by 45-minute in-class interactive sessions with audience response questions, think-pair-share questions, and case discussions. The traditional lecture approach consisted of a 55-minute lecture given by faculty with 5 minutes for questions. Residents completed 3 knowledge tests (pretest, posttest, and 4-month retention) and surveys of their perceptions of the didactic sessions. A linear mixed model was used to compare the effect of both formats on knowledge acquisition and retention. RESULTS: Of 182 eligible postgraduate year 2 anesthesiology residents, 155 (85%) participated in the entire intervention, and 142 (78%) completed all tests. The flipped classroom approach improved knowledge retention after 4 months (adjusted mean = 6%; P = .014; d = 0.56), and residents preferred the flipped classroom (pre = 46%; post = 82%; P < .001). CONCLUSIONS: The flipped classroom approach to didactic education resulted in a small improvement in knowledge retention and was preferred by anesthesiology residents.

A Competitive Objective Structured Clinical Examination Event to Generate an Objective Assessment of Anesthesiology Resident Skills Development.

Residency programs are charged with teaching, assessing, and documenting resident competency for a multitude of skills throughout the course of residency training. An innovative, competition-based objective structured clinical examination event was designed in our department to objectively assess the skill level of anesthesiology residents. After conducting the identical event for 2 years in postgraduate year 1 (PGY1) and postgraduate year 2 (PGY2) residents, we tested the hypothesis that the event can provide adequate standardization to appropriately document progression in technical and nontechnical skills. Twenty-one residents participated in both events during their PGY1 and PGY2 years: n = 10, 2012/2013, n = 11, 2013/2014. The PGY1 participants in 2012 were retested in 2013 (as PGY2 residents) during an identical event, and their performance was compared as a group and on an individual level. The PGY1 residents in 2013 did the same in 2014. Four workstations were analyzed to determine whether improvement in performance occurred between the PGY1 and the PGY2 years: (1) preoperative assessment, (2) operating room anesthesia station checkout, (3) peripheral IV and endotracheal tube placement, and (4) transfer of care in the postanesthesia care unit. The performances of PGY1 and PGY2 residents were compared. The assessments were performed by anesthesiology faculty using checklists, time to complete task, and Likert scale ratings. Data analysis showed improved technical anesthesia skills (operating room setup, peripheral IV, and endotracheal tube placement) and more complete anesthesia-related information management in the preoperative assessment and postoperative transition of care in the postanesthesia care unit in PGY2 residents compared with the PGY1 performance of the same residents. The described event is a valuable tool for objective assessment of multiple anesthesia skills and possible milestones during residency.

A Smartphone-based Decision Support Tool Improves Test Performance Concerning Application of the Guidelines for Managing Regional Anesthesia in the Patient Receiving Antithrombotic or Thrombolytic Therapy.

BACKGROUND: The American Society of Regional Anesthesia and Pain Medicine (ASRA) consensus statement on regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy is the standard for evaluation and management of these patients. The authors hypothesized that an electronic decision support tool (eDST) would improve test performance compared with native physician behavior concerning the application of this guideline. METHODS: Anesthesiology trainees and faculty at 8 institutions participated in a prospective, randomized trial in which they completed a 20-question test involving clinical scenarios related to the ASRA guidelines. The eDST group completed the test using an iOS app programmed to contain decision logic and content of the ASRA guidelines. The control group completed the test by using any resource in addition to the app. A generalized linear mixed-effects model was used to examine the effect of the intervention. RESULTS: After obtaining institutional review board's approval and informed consent, 259 participants were enrolled and randomized (eDST = 122; control = 137). The mean score was 92.4 ± 6.6% in the eDST group and 68.0 ± 15.8% in the control group (P < 0.001). eDST use increased the odds of selecting correct answers (7.8; 95% CI, 5.7 to 10.7). Most control group participants (63%) used some cognitive aid during the test, and they scored higher than those who tested from memory alone (76 ± 15% vs. 57 ± 18%, P < 0.001). There was no difference in time to completion of the test (P = 0.15) and no effect of training level (P = 0.56). CONCLUSIONS: eDST use improved application of the ASRA guidelines compared with the native clinician behavior in a testing environment.

Effect of a cognitive aid on adherence to perioperative assessment and management guidelines for the cardiac evaluation of noncardiac surgical patients.

BACKGROUND: The 2007 American College of Cardiologists/American Heart Association Guidelines on Perioperative Cardiac Evaluation and Care for Noncardiac Surgery is the standard for perioperative cardiac evaluation. Recent work has shown that residents and anesthesiologists do not apply these guidelines when tested. This research hypothesized that a decision support tool would improve adherence to this consensus guideline. METHODS: Anesthesiology residents at four training programs participated in an unblinded, prospective, randomized, cross-over trial in which they completed two tests covering clinical scenarios. One quiz was completed from memory and one with the aid of an electronic decision support tool. Performance was evaluated by overall score (% correct), number of incorrect answers with possibly increased cost or risk of care, and the amount of time required to complete the quizzes both with and without the cognitive aid. The primary outcome was the proportion of correct responses attributable to the use of the decision support tool. RESULTS: All anesthesiology residents at four institutions were recruited and 111 residents participated. Use of the decision support tool resulted in a 25% improvement in adherence to guidelines compared with memory alone (P < 0.0001), and participants made 77% fewer incorrect responses that would have resulted in increased costs. Use of the tool was associated with a 3.4-min increase in time to complete the test (P < 0.001). CONCLUSIONS: Use of an electronic decision support tool significantly improved adherence to the guidelines as compared with memory alone. The decision support tool also prevented inappropriate management steps possibly associated with increased healthcare costs.

Anesthesiology resident personality type correlates with faculty assessment of resident performance.

STUDY OBJECTIVE: To study the association between anesthesiology residents' personality preference types, faculty evaluations of residents' performance, and knowledge. DESIGN: Convenience sample and prospective study. SETTING: Academic department of anesthesiology. SUBJECTS: Consenting anesthesiology residents (n = 36). INTERVENTIONS: All participants completed the Myers Briggs Type Indicator® (MBTI®). MEASUREMENTS: All residents' 6-month summation of daily focal evaluations completed by faculty [daily performance score (DPS); 1 = unsatisfactory, 2 = needs improvement, 3 = meets expectations, 4 = exceeds expectations], as well as a global assessment of performance (GAP) score based on placement of each resident into perceived quartile compared with their peers (ie,1 = first, or top, quartile) by senior faculty (n = 7) who also completed the MBTI, were obtained. The resident MBTI personality preferences were compared with the DPS and GAP scores, the United States Medical Licensing Examination (USMLE) I and II scores, and faculty MBTI personality type. MAIN RESULTS: There was no association between personality preference type and performance on standardized examinations (USMLE I, II). The mean GAP score was better (higher quartile score) for Extraverts than Introverts (median 2.0 vs 2.6, P = 0.0047) and for Sensing versus Intuition (median 2.0 vs 2.6, P = 0.0206) preference. Faculty evaluator MBTI preference type did not influence the GAP scores they assigned residents. Like GAP, the DPS was better for residents with Sensing versus Intuition preference (median 3.5 vs 3.3, P = 0.0111). No difference in DPS was noted between Extraverts and Introverts. CONCLUSIONS: Personality preference type was not associated with resident performance on standardized examinations, but it was associated with faculty evaluations of resident performance. Residents with Sensing personality preference were evaluated more favorably on global and focal faculty evaluations than those residents who chose the Intuition preference. Extraverted residents were evaluated more favorably on global but not focal assessment of performance.