Using Machine Learning to Evaluate Attending Feedback on Resident Performance.

BACKGROUND: High-quality and high-utility feedback allows for the development of improvement plans for trainees. The current manual assessment of the quality of this feedback is time consuming and subjective. We propose the use of machine learning to rapidly distinguish the quality of attending feedback on resident performance. METHODS: Using a preexisting databank of 1925 manually reviewed feedback comments from 4 anesthesiology residency programs, we trained machine learning models to predict whether comments contained 6 predefined feedback traits (actionable, behavior focused, detailed, negative feedback, professionalism/communication, and specific) and predict the utility score of the comment on a scale of 1-5. Comments with ≥4 feedback traits were classified as high-quality and comments with ≥4 utility scores were classified as high-utility; otherwise comments were considered low-quality or low-utility, respectively. We used RapidMiner Studio (RapidMiner, Inc, Boston, MA), a data science platform, to train, validate, and score performance of models. RESULTS: Models for predicting the presence of feedback traits had accuracies of 74.4%-82.2%. Predictions on utility category were 82.1% accurate, with 89.2% sensitivity, and 89.8% class precision for low-utility predictions. Predictions on quality category were 78.5% accurate, with 86.1% sensitivity, and 85.0% class precision for low-quality predictions. Fifteen to 20 hours were spent by a research assistant with no prior experience in machine learning to become familiar with software, create models, and review performance on predictions made. The program read data, applied models, and generated predictions within minutes. In contrast, a recent manual feedback scoring effort by an author took 15 hours to manually collate and score 200 comments during the course of 2 weeks. CONCLUSIONS: Harnessing the potential of machine learning allows for rapid assessment of attending feedback on resident performance. Using predictive models to rapidly screen for low-quality and low-utility feedback can aid programs in improving feedback provision, both globally and by individual faculty.

Learners and Luddites in the Twenty-first Century: Bringing Evidence-based Education to Anesthesiology.

Anesthesiologists are both teachers and learners and alternate between these roles throughout their careers. However, few anesthesiologists have formal training in the methodologies and theories of education. Many anesthesiology educators often teach as they were taught and may not be taking advantage of current evidence in education to guide and optimize the way they teach and learn. This review describes the most up-to-date evidence in education for teaching knowledge, procedural skills, and professionalism. Methods such as active learning, spaced learning, interleaving, retrieval practice, e-learning, experiential learning, and the use of cognitive aids will be described. We made an effort to illustrate the best available evidence supporting educational practices while recognizing the inherent challenges in medical education research. Similar to implementing evidence in clinical practice in an attempt to improve patient outcomes, implementing an evidence-based approach to anesthesiology education may improve learning outcomes.

Do You Really Mean It? Assessing the Strength, Frequency, and Reliability of Applicant Commitment Statements During the Anesthesiology Residency Match.

BACKGROUND: Despite the critical nature of the residency interview process, few metrics have been shown to adequately predict applicant success in matching to a given program. While evaluating and ranking potential candidates, bias can occur when applicants make commitment statements to a program. Survey data show that pressure to demonstrate commitment leads applicants to express commitment to multiple institutions including telling >1 program that they will rank them #1. The primary purpose of this cross-sectional observational study is to evaluate the frequency of commitment statements from applicants to 5 anesthesiology departments during a single interview season, report how often each statement is associated with a successful match, and identify how frequently candidates incorrectly represented commitments to rank a program #1. METHODS: During the 2014 interview season, 5 participating anesthesiology programs collected written and verbal communications from applicants. Three residency program directors independently reviewed the statements to classify them into 1 of 3 categories; guaranteed commitment, high rank commitment, or strong interest. Each institution provided a deidentified rank list with associated commitment statements, biographical data, whether candidates were ranked-to-match, and if they successfully matched. RESULTS: Program directors consistently differentiated among strong interest, high rank, and guaranteed commitment statements with κ coefficients of 0.9 (95% CI, 0.8-0.9) or greater between any pair of reviewers. Overall, 35.8% of applicants (226/632) provided a statement demonstrating at least strong interest and 5.4% (34/632) gave guaranteed commitment statements. Guaranteed commitment statements resulted in a 95.7% match rate to that program in comparison to statements of high rank (25.6%), strong interest (14.6%), and those who provided no statement (5.9%). For those providing guaranteed commitment statements, it can be assumed that the 1 candidate (4.3%) who did not match incorrectly represented himself. Variables such as couples match, "R" positions, and not being ranked-to-match on both advanced and categorical rank lists were eliminated because they can result in a nonmatch despite truthfully ranking a program #1. CONCLUSIONS: Each level of commitment statement resulted in a progressively increased frequency of a successful match to the recipient program. Only 5.4% of applicants committed to rank a program #1, but these statements were very reliable. These data can help program directors interpret commitment statements and assist accurate evaluation of the interest of candidates throughout the match process.

Should Objective Structured Clinical Examinations Assist the Clinical Competency Committee in Assigning Anesthesiology Milestones Competency?

BACKGROUND: With the integration of Objective Structured Clinical Examinations into the Anesthesiology primary board certification process, residency programs may choose to implement Objective Structured Clinical Examinations for resident skill assessment. The aim of this study was to evaluate Objective Structured Clinical Examination-based milestone assessment and compare with Clinical Competency Committee milestone assessment that is based purely on clinical evaluations. METHODS: An annual Objective Structured Clinical Examination event was used to obtain milestone assessment of clinical anesthesia year 0-clinical anesthesia year 3 residents for selected milestones in patient care, professionalism, and interpersonal/communication skills. The Objective Structured Clinical Examination scenarios were different for each training level. The Clinical Competency Committee evaluated each resident semiannually based on clinical evaluations of resident performance. The Clinical Competency Committee milestone assessments from 2014 to 2016 that were recorded closest to the Objective Structured Clinical Examination event (±3 months) were compared to the Objective Structured Clinical Examination milestone assessments. A total of 35 residents were included in this analysis in 3 different training cohorts: A (graduates 2016, n = 12); B (graduates 2017, n = 10); and C (graduates 2018, n = 13). All residents participated in Objective Structured Clinical Examinations because their clinical anesthesia year 0 year and Clinical Competency Committee milestone data had been reported since December 2014. RESULTS: Both assessment techniques indicated a competency growth proportional to the length in training. Despite limited cumulative statistics in this study, average trends in the Objective Structured Clinical Examination-Clinical Competency Committee relationship indicated: (1) a good proportionality in reflecting competency growth; (2) a grade enhancement associated with Clinical Competency Committee assessment, dominated by evaluations of junior residents (clinical anesthesia year 0-clinical anesthesia year 1); and (3) an expectation bias in Clinical Competency Committee assessment, dominated by evaluation of senior residents (clinical anesthesia year 2-clinical anesthesia year 3). CONCLUSIONS: Our analysis confirms the compatibility of the 2 evaluation methods in reflecting longitudinal growth. The deviation of Objective Structured Clinical Examination assessments versus Clinical Competency Committee assessments suggests that Objective Structured Clinical Examinations may be providing additional or different information on resident performance. Educators might consider using both assessment methods to provide the most reliable and valid competency assessments during residency.

Enhancing Feedback on Professionalism and Communication Skills in Anesthesia Residency Programs.

BACKGROUND: Despite its importance, training faculty to provide feedback to residents remains challenging. We hypothesized that, overall, at 4 institutions, a faculty development program on providing feedback on professionalism and communication skills would lead to (1) an improvement in the quantity, quality, and utility of feedback and (2) an increase in feedback containing negative/constructive feedback and pertaining to professionalism/communication. As secondary analyses, we explored these outcomes at the individual institutions. METHODS: In this prospective cohort study (October 2013 to July 2014), we implemented a video-based educational program on feedback at 4 institutions. Feedback records from 3 months before to 3 months after the intervention were rated for quality (0-5), utility (0-5), and whether they had negative/constructive feedback and/or were related to professionalism/communication. Feedback records during the preintervention, intervention, and postintervention periods were compared using the Kruskal-Wallis and χ tests. Data are reported as median (interquartile range) or proportion/percentage. RESULTS: A total of 1926 feedback records were rated. The institutions overall did not have a significant difference in feedback quantity (preintervention: 855/3046 [28.1%]; postintervention: 896/3327 [26.9%]; odds ratio: 1.06; 95% confidence interval, 0.95-1.18; P = .31), feedback quality (preintervention: 2 [1-4]; intervention: 2 [1-4]; postintervention: 2 [1-4]; P = .90), feedback utility (preintervention: 1 [1-3]; intervention: 2 [1-3]; postintervention: 1 [1-2]; P = .61), or percentage of feedback records containing negative/constructive feedback (preintervention: 27%; intervention: 32%; postintervention: 25%; P = .12) or related to professionalism/communication (preintervention: 23%; intervention: 33%; postintervention: 24%; P = .03). Institution 1 had a significant difference in feedback quality (preintervention: 2 [1-3]; intervention: 3 [2-4]; postintervention: 3 [2-4]; P = .001) and utility (preintervention: 1 [1-3]; intervention: 2 [1-3]; postintervention: 2 [1-4]; P = .008). Institution 3 had a significant difference in the percentage of feedback records containing negative/constructive feedback (preintervention: 16%; intervention: 28%; postintervention: 17%; P = .02). Institution 2 had a significant difference in the percentage of feedback records related to professionalism/communication (preintervention: 26%; intervention: 57%; postintervention: 31%; P < .001). CONCLUSIONS: We detected no overall changes but did detect different changes at each institution despite the identical intervention. The intervention may be more effective with new faculty and/or smaller discussion sessions. Future steps include refining the rating system, exploring ways to sustain changes, and investigating other factors contributing to feedback quality and utility.

A Smartphone-based Decision Support Tool Improves Test Performance Concerning Application of the Guidelines for Managing Regional Anesthesia in the Patient Receiving Antithrombotic or Thrombolytic Therapy.

BACKGROUND: The American Society of Regional Anesthesia and Pain Medicine (ASRA) consensus statement on regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy is the standard for evaluation and management of these patients. The authors hypothesized that an electronic decision support tool (eDST) would improve test performance compared with native physician behavior concerning the application of this guideline. METHODS: Anesthesiology trainees and faculty at 8 institutions participated in a prospective, randomized trial in which they completed a 20-question test involving clinical scenarios related to the ASRA guidelines. The eDST group completed the test using an iOS app programmed to contain decision logic and content of the ASRA guidelines. The control group completed the test by using any resource in addition to the app. A generalized linear mixed-effects model was used to examine the effect of the intervention. RESULTS: After obtaining institutional review board's approval and informed consent, 259 participants were enrolled and randomized (eDST = 122; control = 137). The mean score was 92.4 ± 6.6% in the eDST group and 68.0 ± 15.8% in the control group (P < 0.001). eDST use increased the odds of selecting correct answers (7.8; 95% CI, 5.7 to 10.7). Most control group participants (63%) used some cognitive aid during the test, and they scored higher than those who tested from memory alone (76 ± 15% vs. 57 ± 18%, P < 0.001). There was no difference in time to completion of the test (P = 0.15) and no effect of training level (P = 0.56). CONCLUSIONS: eDST use improved application of the ASRA guidelines compared with the native clinician behavior in a testing environment.

Effect of a cognitive aid on adherence to perioperative assessment and management guidelines for the cardiac evaluation of noncardiac surgical patients.

BACKGROUND: The 2007 American College of Cardiologists/American Heart Association Guidelines on Perioperative Cardiac Evaluation and Care for Noncardiac Surgery is the standard for perioperative cardiac evaluation. Recent work has shown that residents and anesthesiologists do not apply these guidelines when tested. This research hypothesized that a decision support tool would improve adherence to this consensus guideline. METHODS: Anesthesiology residents at four training programs participated in an unblinded, prospective, randomized, cross-over trial in which they completed two tests covering clinical scenarios. One quiz was completed from memory and one with the aid of an electronic decision support tool. Performance was evaluated by overall score (% correct), number of incorrect answers with possibly increased cost or risk of care, and the amount of time required to complete the quizzes both with and without the cognitive aid. The primary outcome was the proportion of correct responses attributable to the use of the decision support tool. RESULTS: All anesthesiology residents at four institutions were recruited and 111 residents participated. Use of the decision support tool resulted in a 25% improvement in adherence to guidelines compared with memory alone (P < 0.0001), and participants made 77% fewer incorrect responses that would have resulted in increased costs. Use of the tool was associated with a 3.4-min increase in time to complete the test (P < 0.001). CONCLUSIONS: Use of an electronic decision support tool significantly improved adherence to the guidelines as compared with memory alone. The decision support tool also prevented inappropriate management steps possibly associated with increased healthcare costs.

A Novel Tool to Guide Reintegration of Anesthesiologists Into Clinical and Academic Work After Concussion.

Concussion is a common form of mild traumatic brain injury that can cause somatic, cognitive, and behavioral impairments lasting days to weeks. There are no published guidelines or recommendations to facilitate the safe and successful reintegration of anesthesiologist clinicians and trainees into clinical and academic work after concussion. We developed a simple 4-phase postconcussion recovery protocol for anesthesiologists who have suffered concussion and describe the successful use of this postconcussion recovery protocol to support reintegration of an Anesthesiology Critical Care Medicine fellow who developed mild concussion during vacation leave.

Utilization of the Flipped Classroom in Anesthesiology Graduate Medical Education: An Initial Survey of Faculty Beliefs and Practices About Active Learning.

BACKGROUND: Flipped classroom (FC) is an active learning (AL) technique thought to have potential benefits in anesthesiology resident education. This survey aimed to determine the frequency of FC utilization, barriers of utilization, and means to overcome these barriers. METHOD: A web-based questionnaire was developed to survey anesthesiology faculty on their knowledge of and experience with FC. The Society of Academic Associations of Anesthesiology and Perioperative Medicine sent the survey to all United States core program directors (PD) via their list serve with a request for the PDs to forward the survey invitation to their clinical faculty. Descriptive statistics were summarized. RESULTS: A total of 244 anesthesiology faculty completed the survey. Reported faculty understanding of AL and FC were 57%. Of these faculty, 87% utilized AL and 57% utilized FC in their personal teaching practice during the past year (spring 2015-spring 2016). The most prevalent barriers to utilization of FC were faculty concern that learners would not come to class prepared or participate in class, faculty comfort with delivering traditional lectures, lack of faculty knowledge of necessary technological tools, and faculty concern about perceived increase in time needed to create a FC session. Eighty-nine percent of all faculty desired education on FC with preference for institutional workshops or grand rounds. CONCLUSIONS: Our survey found a discrepancy between faculty knowledge of FC and usage of this method in anesthesiology resident teaching. More educational resources are warranted to address barriers and familiarize faculty with FC applications in anesthesiology resident education.

Replicating an Educational OSCE Project for Skill Assessment of Junior Anesthesiology Residents at Multiple Institutions: A Qualitative Description.

BACKGROUND: Educational research projects are often developed and implemented at a single institution. However, the research project methods and results may not be generalizable and able to be replicated successfully at other institutions. The aim of this study was to investigate the process of replicating an effective educational Objective Structured Clinical Examination (OSCE) event at multiple other institutions. METHODS: An OSCE event was initially designed and implemented at the primary institution to assess the skill level of junior residents on the performance of basic anesthesia tasks. After the initial implementation, additional institutions were recruited to participate in a replication of this OSCE event at their own institutions. The primary institution provided the OSCE scenarios, assessment tools, rater training, and resident participant instructions. The participating secondary institutions' (n = 4) event managers obtained Institutional Review Board [IRB] approval, developed the event schedule, assigned faculty evaluators, and organized the simulation space at their own medical centers. The events were assessed by the secondary institutions' resident and faculty participants via an anonymous survey regarding the event's content and their perception of its educational value. RESULTS: We replicated a complex educational OSCE event, developed and implemented at 1 institution, at 4 other institutions. Resident participants (n = 60), participating faculty (n = 24), and event directors (n = 4) indicated a high level of appreciation for the OSCE event. CONCLUSION: Using a structured approach, educational OSCE events can be successfully replicated at multiple institutions. Organization of multi-institutional studies and collaborative efforts is complex. This study illustrates 1 example of how to successfully approach multi-institutional educational projects.

Results of a Flipped Classroom Teaching Approach in Anesthesiology Residents.

BACKGROUND: In a flipped classroom approach, learners view educational content prior to class and engage in active learning during didactic sessions. OBJECTIVE: We hypothesized that a flipped classroom improves knowledge acquisition and retention for residents compared to traditional lecture, and that residents prefer this approach. METHODS: We completed 2 iterations of a study in 2014 and 2015. Institutions were assigned to either flipped classroom or traditional lecture for 4 weekly sessions. The flipped classroom consisted of reviewing a 15-minute video, followed by 45-minute in-class interactive sessions with audience response questions, think-pair-share questions, and case discussions. The traditional lecture approach consisted of a 55-minute lecture given by faculty with 5 minutes for questions. Residents completed 3 knowledge tests (pretest, posttest, and 4-month retention) and surveys of their perceptions of the didactic sessions. A linear mixed model was used to compare the effect of both formats on knowledge acquisition and retention. RESULTS: Of 182 eligible postgraduate year 2 anesthesiology residents, 155 (85%) participated in the entire intervention, and 142 (78%) completed all tests. The flipped classroom approach improved knowledge retention after 4 months (adjusted mean = 6%; P = .014; d = 0.56), and residents preferred the flipped classroom (pre = 46%; post = 82%; P < .001). CONCLUSIONS: The flipped classroom approach to didactic education resulted in a small improvement in knowledge retention and was preferred by anesthesiology residents.

Multiple failed external defibrillation attempts during robot-assisted internal mammary harvest for myocardial revascularization.

We describe multiple failed external defibrillation attempts via rescue defibrillation pads for ventricular fibrillation that occurred secondary to direct electrical current transmission through the pericardial sac from electrocautery during robot-assisted internal mammary harvest. Only after resumption of two-lung ventilation and decompression of the iatrogenic pneumothorax was the patient successfully defibrillated. Conditions necessary for robotic intrathoracic surgery may make defibrillation and resuscitation difficult if they become necessary.

Cognitive aid use improves transition of care by graduating medical students during a simulated crisis.

BACKGROUND: Residents are expected to have transition of care (ToC) skills upon entering graduate medical education. It is unclear whether experience and training during medical school is adequate. OBJECTIVE: The aim of the project was to assess: 1) graduating medical students' ability to perform ToC in a crisis situation, and 2) whether using a cognitive aid improves the ToC quality. METHODS: The authors developed simulation scenarios for rapid response teams and a cognitive aid to assist in the ToC during crisis situations. Graduating medical students were enrolled and randomly divided into teams of three students, randomly assigned into one of two groups: teams using a cognitive aid for ToC (CA), or not using a cognitive aid (nCA). In the scenario, teams respond to a deteriorating patient and then transfer care to the next provider after stabilization. Three faculty reviewed the recording to assess completeness of the ToC and the overall quality. A completeness score was expressed as a fraction of the maximum score. Statistical analysis was performed using a t-test and Mann-Whitney U test. RESULTS: A total of 112 senior medical students participated: CA n=19, nCA n=17. The completeness score of the ToC and overall quality improved when using the cognitive aid (completeness score: CA 0.80±0.06 vs. nCA 0.52±0.07, p<0.01; ToC quality: CA 3.16±0.65 vs. nCA 1.92±0.56, p<0.01). Participants' rating of knowledge and comfort with the ToC process increased after the simulation. CONCLUSION: The completeness of information transfer during the ToC process by graduating medical students improved by using a cognitive aid in a simulated patient crisis.

Ultrasound Skill and Application of Knowledge Assessment using an Innovative OSCE Competition-Based Simulation Approach.

BACKGROUND: Despite Point-of Care Ultrasound (PoC US) rapidly becoming an important tool in perioperative medicine structured education, PoC US is currently rarely integrated into the anesthesiology residency curriculum. The aim of this project was to assess the current ultrasound skills of anesthesiology residents at one institution and evaluate the needs for development of a formal ultrasound curriculum. METHODS: A event containing 6 different OSCE PoC US stations was developed with following stations: vascular, peripheral nerve block, lung ultrasound, transthoracic echocardiography (TTE) human model, pathologic TTE (simulator), and inferior vena cava (IVC) evaluation (simulator). The ability to obtain an US image or to interpret the US information was evaluated using a checklist and global rating scale. After IRB approval, anesthesiology residents participated in this event (n=30; PGY 2-4). RESULTS: All residents were able to identify vascular structures and demonstrated sufficient ultrasound skill for lung anatomy IVC assessment. The lowest scores were observed for performing and interpreting TTE. There were no differences in resident ultrasound skills for all OSCE stations except minor differences between PGY 2 and PGY 4 in TTE pathology station. While more advanced residents had more clinical exposure to ultrasound for procedures and point-of-care diagnosis, we did not find growth in ultrasound skill level. Despite performing sufficient ultrasound guided peripheral nerve blocks, PGY 4 residents were not able to consistently identify common nerve block targets. CONCLUSIONS: Our findings indicate that exposure and clinical use of ultrasound for procedures and point-of-care diagnosis is not sufficient for developing competency in PoC US and that a formal curriculum throughout the entire anesthesiology residency is needed to ensure PoC US competency.

A Competitive Objective Structured Clinical Examination Event to Generate an Objective Assessment of Anesthesiology Resident Skills Development.

Residency programs are charged with teaching, assessing, and documenting resident competency for a multitude of skills throughout the course of residency training. An innovative, competition-based objective structured clinical examination event was designed in our department to objectively assess the skill level of anesthesiology residents. After conducting the identical event for 2 years in postgraduate year 1 (PGY1) and postgraduate year 2 (PGY2) residents, we tested the hypothesis that the event can provide adequate standardization to appropriately document progression in technical and nontechnical skills. Twenty-one residents participated in both events during their PGY1 and PGY2 years: n = 10, 2012/2013, n = 11, 2013/2014. The PGY1 participants in 2012 were retested in 2013 (as PGY2 residents) during an identical event, and their performance was compared as a group and on an individual level. The PGY1 residents in 2013 did the same in 2014. Four workstations were analyzed to determine whether improvement in performance occurred between the PGY1 and the PGY2 years: (1) preoperative assessment, (2) operating room anesthesia station checkout, (3) peripheral IV and endotracheal tube placement, and (4) transfer of care in the postanesthesia care unit. The performances of PGY1 and PGY2 residents were compared. The assessments were performed by anesthesiology faculty using checklists, time to complete task, and Likert scale ratings. Data analysis showed improved technical anesthesia skills (operating room setup, peripheral IV, and endotracheal tube placement) and more complete anesthesia-related information management in the preoperative assessment and postoperative transition of care in the postanesthesia care unit in PGY2 residents compared with the PGY1 performance of the same residents. The described event is a valuable tool for objective assessment of multiple anesthesia skills and possible milestones during residency.

Objective Assessment of Anesthesiology Resident Skills Using an Innovative Competition-Based Simulation Approach.

Residency programs are charged with teaching, assessing, and documenting resident competency for a multitude of skills. Documentation of competency requires demonstrating specific milestones mandated by the Accreditation Council for Graduate Medical Education. Our department designed an innovative, competition-based approach to objectively assess the skill level of postgraduate year 1 residents in performing basic anesthesia-related tasks after 1 month of anesthesiology training. We launched an "Olympic" event to assess requisite skills in an environment of friendly competition. A simulation format was chosen to allow standardized objective assessment of the resident's skill level at an early stage of training, with possible identification of and intervention for skills needing improvement. Our experience may serve as a template for other programs and specialties developing processes for assessing and documenting improvement in skill and competency over the course of residency training.

AngioJet rheolytic thrombectomy for thrombotic myocardial injury in a 15-year-old cystic fibrosis patient during lung transplantation.

We report on an acute myocardial injury caused by thrombotic emboli during off-pump bilateral sequential lung transplantation in a 15-year-old female patient with end-stage cystic fibrosis. A few minutes after reperfusion of the right (second sequential) allograft, the patient developed hemodynamic collapse, including hypotension, acute akinesis of the anterior and lateral myocardial walls, and severe mitral regurgitation. Thrombus was noted within the left atrium and ventricle on intraoperative transesophageal echocardiogram. After emergent cannulation for cardiopulmonary bypass, the patient underwent left cardiac catheterization with AngioJet rheolytic thrombectomy and removal of the clot from the distal left anterior descending coronary artery. Reperfusion was established without the need for stent placement or further intervention. The patient required short-term life support with venoarterial extracorporeal membrane oxygenation.