Development and Validation of a Machine Learning Algorithm Using Clinical Pages to Predict Imminent Clinical Deterioration.

BACKGROUND: Early detection of clinical deterioration among hospitalized patients is a clinical priority for patient safety and quality of care. Current automated approaches for identifying these patients perform poorly at identifying imminent events. OBJECTIVE: Develop a machine learning algorithm using pager messages sent between clinical team members to predict imminent clinical deterioration. DESIGN: We conducted a large observational study using long short-term memory machine learning models on the content and frequency of clinical pages. PARTICIPANTS: We included all hospitalizations between January 1, 2018 and December 31, 2020 at Vanderbilt University Medical Center that included at least one page message to physicians. Exclusion criteria included patients receiving palliative care, hospitalizations with a planned intensive care stay, and hospitalizations in the top 2% longest length of stay. MAIN MEASURES: Model classification performance to identify in-hospital cardiac arrest, transfer to intensive care, or Rapid Response activation in the next 3-, 6-, and 12-hours. We compared model performance against three common early warning scores: Modified Early Warning Score, National Early Warning Score, and the Epic Deterioration Index. KEY RESULTS: There were 87,783 patients (mean [SD] age 54.0 [18.8] years; 45,835 [52.2%] women) who experienced 136,778 hospitalizations. 6214 hospitalized patients experienced a deterioration event. The machine learning model accurately identified 62% of deterioration events within 3-hours prior to the event and 47% of events within 12-hours. Across each time horizon, the model surpassed performance of the best early warning score including area under the receiver operating characteristic curve at 6-hours (0.856 vs. 0.781), sensitivity at 6-hours (0.590 vs. 0.505), specificity at 6-hours (0.900 vs. 0.878), and F-score at 6-hours (0.291 vs. 0.220). CONCLUSIONS: Machine learning applied to the content and frequency of clinical pages improves prediction of imminent deterioration. Using clinical pages to monitor patient acuity supports improved detection of imminent deterioration without requiring changes to clinical workflow or nursing documentation.

Development and Validation of Trigger Algorithms to Identify Delays in Diagnostic Evaluation of Gastroenterological Cancer.

BACKGROUND & AIMS: Colorectal cancer (CRC) and hepatocellular cancer (HCC) are common causes of death and morbidity, and patients benefit from early detection. However, delays in follow-up of suspicious findings are common, and methods to efficiently detect such delays are needed. We developed, refined, and tested trigger algorithms that identify patients with delayed follow-up evaluation of findings suspicious of CRC or HCC. METHODS: We developed and validated two trigger algorithms that detect delays in diagnostic evaluation of CRC and HCC using laboratory, diagnosis, procedure, and referral codes from the Department of Veteran Affairs National Corporate Data Warehouse. The algorithm initially identified patients with positive test results for iron deficiency anemia or fecal immunochemical test (for CRC) and elevated α-fetoprotein results (for HCC). Our algorithm then excluded patients for whom follow-up evaluation was unnecessary, such as patients with a terminal illness or those who had already completed a follow-up evaluation within 60 days. Clinicians reviewed samples of both delayed and nondelayed records, and review data were used to calculate trigger performance. RESULTS: We applied the algorithm for CRC to 245,158 patients seen from January 1, 2013, through December 31, 2013 and identified 1073 patients with delayed follow up. In a review of 400 randomly selected records, we found that our algorithm identified patients with delayed follow-up with a positive predictive value of 56.0% (95% CI, 51.0%-61.0%). We applied the algorithm for HCC to 333,828 patients seen from January 1, 2011 through December 31, 2014, and identified 130 patients with delayed follow-up. During manual review of all 130 records, we found that our algorithm identified patients with delayed follow-up with a positive predictive value of 82.3% (95% CI, 74.4%-88.2%). When we extrapolated the findings to all patients with abnormal results, the algorithm identified patients with delayed follow-up evaluation for CRC with 68.6% sensitivity (95% CI, 65.4%-71.6%) and 81.1% specificity (95% CI, 79.5%-82.6%); it identified patients with delayed follow-up evaluation for HCC with 89.1% sensitivity (95% CI, 81.8%-93.8%) and 96.5% specificity (95% CI, 94.8%-97.7%). Compared to nonselective methods, use of the algorithm reduced the number of records required for review to identify a delay by more than 99%. CONCLUSIONS: Using data from the Veterans Affairs electronic health record database, we developed an algorithm that greatly reduces the number of record reviews necessary to identify delays in follow-up evaluations for patients with suspected CRC or HCC. This approach offers a more efficient method to identify delayed diagnostic evaluation of gastroenterological cancers.

Electronic Detection of Delayed Test Result Follow-Up in Patients with Hypothyroidism.

BACKGROUND: Delays in following up abnormal test results are a common problem in outpatient settings. Surveillance systems that use trigger tools to identify delayed follow-up can help reduce missed opportunities in care. OBJECTIVE: To develop and test an electronic health record (EHR)-based trigger algorithm to identify instances of delayed follow-up of abnormal thyroid-stimulating hormone (TSH) results in patients being treated for hypothyroidism. DESIGN: We developed an algorithm using structured EHR data to identify patients with hypothyroidism who had delayed follow-up (>60 days) after an abnormal TSH. We then retrospectively applied the algorithm to a large EHR data warehouse within the Department of Veterans Affairs (VA), on patient records from two large VA networks for the period from January 1, 2011, to December 31, 2011. Identified records were reviewed to confirm the presence of delays in follow-up. KEY RESULTS: During the study period, 645,555 patients were seen in the outpatient setting within the two networks. Of 293,554 patients with at least one TSH test result, the trigger identified 1250 patients on treatment for hypothyroidism with elevated TSH. Of these patients, 271 were flagged as potentially having delayed follow-up of their test result. Chart reviews confirmed delays in 163 of the 271 flagged patients (PPV = 60.1%). CONCLUSIONS: An automated trigger algorithm applied to records in a large EHR data warehouse identified patients with hypothyroidism with potential delays in thyroid function test results follow-up. Future prospective application of the TSH trigger algorithm can be used by clinical teams as a surveillance and quality improvement technique to monitor and improve follow-up.

Development and field testing of a self-assessment guide for computer-based provider order entry.

Despite the benefits of computerized provider order entry (CPOE), numerous reports of unexpected CPOE-related safety concerns have surfaced. As part of a larger project to improve the safety of electronic health records (EHRs), we developed and field tested a CPOE "safety self-assessment" guide through literature searches, expert opinion, and site visits. We then conducted a field test of this guide with nine hospital chief medical informatics officers (CMIOs), who were identified through the Association of Medical Directors of Information Systems. The CPOE safety self-assessment guide was sent electronically to the CMIOs. Once the assessments were returned, we conducted structured telephone interviews for further comments about the guide's format and content. The CMIOs in our study found the CPOE safety guide useful and relatively easy to complete, taking no more than 30 minutes. Analysis of responses to the guide suggest that most recommended practices were implemented inconsistently across facilities. Despite consensus for certain CPOE best practices in the medical literature and among experts, there appeared to be considerable variation among CMIOs' opinions of best practices. Interview data suggested this inconsistency was mostly due to system limitations and/or differing opinions about the necessity of certain EHR-related safety measures. Despite the absence of consensus on best practices, a self-assessment safety guide provides a practical starting point for organizations to assess and improve safety and the effectiveness of their CPOE system.

Special Issue on Informatics Education: Vanderbilt Clinical Informatics Center Education Strategy: To Infinity and Beyond!

BACKGROUND: The Vanderbilt Clinical Informatics Center (VCLIC) is based in the Department of Biomedical Informatics (DBMI) and operates across Vanderbilt University Medical Center (VUMC) and Vanderbilt University (VU) with a goal of enabling and supporting clinical informatics research and practice. VCLIC supports several types of applied clinical informatics teaching, including teaching of students in courses, professional education for staff and faculty throughout VUMC, and workshops and conferences that are open to the public. OBJECTIVES: In this paper, we provide a detailed accounting of our center and institution's methods of educating and training faculty, staff, students, and trainees from across the academic institution and health system on clinical informatics topics, including formal training programs and informal applied learning sessions. METHODS: Through a host of informal learning events, such as workshops, seminars, conference-style events, bite-size instructive videos, and hackathons, as well as several formal education programs, such as the Clinical Informatics Graduate Course, Master's in Applied Clinical Informatics, Medical Student Integrated Science Course, Graduate Medical Education Elective, and Fellowship in Clinical Informatics, VCLIC and VUMC provide opportunities for faculty, students, trainees, and even staff to engage with Clinical Informatics topics and learn related skills. RESULTS: The described programs have trained hundreds of participants from across the academic and clinical enterprises. Of the VCLIC-held events, the majority of attendees indicated through surveys that they were satisfied, with the average satisfaction score being 4.63/5, and all events averaging a satisfaction score of greater than 4. Across the 20 events VCLIC has held, our largest audiences are DBMI, HealthIT operational staff, and students from the medical and nursing schools. CONCLUSIONS: VCLIC has created and delivered a successful suite of formal and informal educational events and programs to disseminate clinical informatics knowledge and skills to learners across the academic institution and healthcare system.

Implementable Prediction of Pressure Injuries in Hospitalized Adults: Model Development and Validation.

Background: Numerous pressure injury prediction models have been developed using electronic health record data, yet hospital-acquired pressure injuries (HAPIs) are increasing, which demonstrates the critical challenge of implementing these models in routine care. Objective: To help bridge the gap between development and implementation, we sought to create a model that was feasible, broadly applicable, dynamic, actionable, and rigorously validated and then compare its performance to usual care (ie, the Braden scale). Methods: We extracted electronic health record data from 197,991 adult hospital admissions with 51 candidate features. For risk prediction and feature selection, we used logistic regression with a least absolute shrinkage and selection operator (LASSO) approach. To compare the model with usual care, we used the area under the receiver operating curve (AUC), Brier score, slope, intercept, and integrated calibration index. The model was validated using a temporally staggered cohort. Results: A total of 5458 HAPIs were identified between January 2018 and July 2022. We determined 22 features were necessary to achieve a parsimonious and highly accurate model. The top 5 features included tracheostomy, edema, central line, first albumin measure, and age. Our model achieved higher discrimination than the Braden scale (AUC 0.897, 95% CI 0.893-0.901 vs AUC 0.798, 95% CI 0.791-0.803). Conclusions: We developed and validated an accurate prediction model for HAPIs that surpassed the standard-of-care risk assessment and fulfilled necessary elements for implementation. Future work includes a pragmatic randomized trial to assess whether our model improves patient outcomes.

POINT: Pipeline for Offline Conversion and Integration of Geocodes and Neighborhood Data.

OBJECTIVES: Geocoding, the process of converting addresses into precise geographic coordinates, allows researchers and health systems to obtain neighborhood-level estimates of social determinants of health. This information supports opportunities to personalize care and interventions for individual patients based on the environments where they live. We developed an integrated offline geocoding pipeline to streamline the process of obtaining address-based variables, which can be integrated into existing data processing pipelines. METHODS: POINT is a web-based, containerized, application for geocoding addresses that can be deployed offline and made available to multiple users across an organization. Our application supports use through both a graphical user interface and application programming interface to query geographic variables, by census tract, without exposing sensitive patient data. We evaluated our application's performance using two datasets: one consisting of 1 million nationally representative addresses sampled from Open Addresses, and the other consisting of 3,096 previously geocoded patient addresses. RESULTS: A total of 99.4 and 99.8% of addresses in the Open Addresses and patient addresses datasets, respectively, were geocoded successfully. Census tract assignment was concordant with reference in greater than 90% of addresses for both datasets. Among successful geocodes, median (interquartile range) distances from reference coordinates were 52.5 (26.5-119.4) and 14.5 (10.9-24.6) m for the two datasets. CONCLUSION: POINT successfully geocodes more addresses and yields similar accuracy to existing solutions, including the U.S. Census Bureau's official geocoder. Addresses are considered protected health information and cannot be shared with common online geocoding services. POINT is an offline solution that enables scalability to multiple users and integrates downstream mapping to neighborhood-level variables with a pipeline that allows users to incorporate additional datasets as they become available. As health systems and researchers continue to explore and improve health equity, it is essential to quickly and accurately obtain neighborhood variables in a Health Insurance Portability and Accountability Act (HIPAA)-compliant way.

Leveraging natural language processing to identify eligible lung cancer screening patients with the electronic health record.

OBJECTIVE: To develop and validate an approach that identifies patients eligible for lung cancer screening (LCS) by combining structured and unstructured smoking data from the electronic health record (EHR). METHODS: We identified patients aged 50-80 years who had at least one encounter in a primary care clinic at Vanderbilt University Medical Center (VUMC) between 2019 and 2022. We fine-tuned an existing natural language processing (NLP) tool to extract quantitative smoking information using clinical notes collected from VUMC. Then, we developed an approach to identify patients who are eligible for LCS by combining smoking information from structured data and clinical narratives. We compared this method with two approaches to identify LCS eligibility only using smoking information from structured EHR. We used 50 patients with a documented history of tobacco use for comparison and validation. RESULTS: 102,475 patients were included. The NLP-based approach achieved an F1-score of 0.909, and accuracy of 0.96. The baseline approach could identify 5,887 patients. Compared to the baseline approach, the number of identified patients using all structured data and the NLP-based algorithm was 7,194 (22.2 %) and 10,231 (73.8 %), respectively. The NLP-based approach identified 589 Black/African Americans, a significant increase of 119 %. CONCLUSION: We present a feasible NLP-based approach to identify LCS eligible patients. It provides a technical basis for the development of clinical decision support tools to potentially improve the utilization of LCS and diminish healthcare disparities.

A multi-site randomized trial of a clinical decision support intervention to improve problem list completeness.

OBJECTIVE: To improve problem list documentation and care quality. MATERIALS AND METHODS: We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS: There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION: The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION: An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.

Conceptualizing clinical decision support as complex interventions: a meta-analysis of comparative effectiveness trials.

OBJECTIVES: Complex interventions with multiple components and behavior change strategies are increasingly implemented as a form of clinical decision support (CDS) using native electronic health record functionality. Objectives of this study were, therefore, to (1) identify the proportion of randomized controlled trials with CDS interventions that were complex, (2) describe common gaps in the reporting of complexity in CDS research, and (3) determine the impact of increased complexity on CDS effectiveness. MATERIALS AND METHODS: To assess CDS complexity and identify reporting gaps for characterizing CDS interventions, we used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting tool for complex interventions. We evaluated the effect of increased complexity using random-effects meta-analysis. RESULTS: Most included studies evaluated a complex CDS intervention (76%). No studies described use of analytical frameworks or causal pathways. Two studies discussed use of theory but only one fully described the rationale and put it in context of a behavior change. A small but positive effect (standardized mean difference, 0.147; 95% CI, 0.039-0.255; P < .01) in favor of increasing intervention complexity was observed. DISCUSSION: While most CDS studies should classify interventions as complex, opportunities persist for documenting and providing resources in a manner that would enable CDS interventions to be replicated and adapted. Unless reporting of the design, implementation, and evaluation of CDS interventions improves, only slight benefits can be expected. CONCLUSION: Conceptualizing CDS as complex interventions may help convey the careful attention that is needed to ensure these interventions are contextually and theoretically informed.

New onset delirium prediction using machine learning and long short-term memory (LSTM) in electronic health record.

OBJECTIVE: To develop and test an accurate deep learning model for predicting new onset delirium in hospitalized adult patients. METHODS: Using electronic health record (EHR) data extracted from a large academic medical center, we developed a model combining long short-term memory (LSTM) and machine learning to predict new onset delirium and compared its performance with machine-learning-only models (logistic regression, random forest, support vector machine, neural network, and LightGBM). The labels of models were confusion assessment method (CAM) assessments. We evaluated models on a hold-out dataset. We calculated Shapley additive explanations (SHAP) measures to gauge the feature impact on the model. RESULTS: A total of 331 489 CAM assessments with 896 features from 34 035 patients were included. The LightGBM model achieved the best performance (AUC 0.927 [0.924, 0.929] and F1 0.626 [0.618, 0.634]) among the machine learning models. When combined with the LSTM model, the final model's performance improved significantly (P = .001) with AUC 0.952 [0.950, 0.955] and F1 0.759 [0.755, 0.765]. The precision value of the combined model improved from 0.497 to 0.751 with a fixed recall of 0.8. Using the mean absolute SHAP values, we identified the top 20 features, including age, heart rate, Richmond Agitation-Sedation Scale score, Morse fall risk score, pulse, respiratory rate, and level of care. CONCLUSION: Leveraging LSTM to capture temporal trends and combining it with the LightGBM model can significantly improve the prediction of new onset delirium, providing an algorithmic basis for the subsequent development of clinical decision support tools for proactive delirium interventions.

Clinician collaboration to improve clinical decision support: the Clickbusters initiative.

OBJECTIVE: We describe the Clickbusters initiative implemented at Vanderbilt University Medical Center (VUMC), which was designed to improve safety and quality and reduce burnout through the optimization of clinical decision support (CDS) alerts. MATERIALS AND METHODS: We developed a 10-step Clickbusting process and implemented a program that included a curriculum, CDS alert inventory, oversight process, and gamification. We carried out two 3-month rounds of the Clickbusters program at VUMC. We completed descriptive analyses of the changes made to alerts during the process, and of alert firing rates before and after the program. RESULTS: Prior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week. After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%). In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS. CONCLUSIONS: At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks. The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.

How can we partner with electronic health record vendors on the complex journey to safer health care?

The Office of the National Coordinator for Health Information Technology released the Safety Assurance Factors for EHR Resilience (SAFER) guides in 2014. Our group developed these guides covering key facets of both electronic health record (EHR) infrastructure (eg, system configuration, contingency planning for downtime, and system-to-system interfaces) and clinical processes (eg, computer-based provider order entry with clinical decision support, test result reporting, patient identification, and clinician-to-clinician communication). The SAFER guides encourage healthy relationships between EHR vendors and users. We conducted a qualitative study over 12 months. We visited 9 health care organizations ranging in size from 1-doctor outpatient clinics to large, multisite, multihospital integrated delivery networks. We interviewed and observed clinicians, IT professionals, and administrators. From the interview transcripts and observation field notes, we identified overarching themes: technical functionality, usability, standards, testing, workflow processes, personnel to support implementation and use, infrastructure, and clinical content. In addition, we identified health care organization-EHR vendor working relationships: marine drill sergeant, mentor, development partner, seller, and parasite. We encourage health care organizations and EHR vendors to develop healthy working relationships to help address the tasks required to design, develop, implement, and maintain EHRs required to achieve safer and higher quality health care.

Application of Human Factors Methods to Understand Missed Follow-up of Abnormal Test Results.

OBJECTIVE: This study demonstrates application of human factors methods for understanding causes for lack of timely follow-up of abnormal test results ("missed results") in outpatient settings. METHODS: We identified 30 cases of missed test results by querying electronic health record data, developed a critical decision method (CDM)-based interview guide to understand decision-making processes, and interviewed physicians who ordered these tests. We analyzed transcribed responses using a contextual inquiry (CI)-based methodology to identify contextual factors contributing to missed results. We then developed a CI-based flow model and conducted a fault tree analysis (FTA) to identify hierarchical relationships between factors that delayed action. RESULTS: The flow model highlighted barriers in information flow and decision making, and the hierarchical model identified relationships between contributing factors for delayed action. Key findings including underdeveloped methods to track follow-up, as well as mismatches, in communication channels, timeframes, and expectations between patients and physicians. CONCLUSION: This case report illustrates how human factors-based approaches can enable analysis of contributing factors that lead to missed results, thus informing development of preventive strategies to address them.

Test results management and distributed cognition in electronic health record-enabled primary care.

Managing abnormal test results in primary care involves coordination across various settings. This study identifies how primary care teams manage test results in a large, computerized healthcare system in order to inform health information technology requirements for test results management and other distributed healthcare services. At five US Veterans Health Administration facilities, we interviewed 37 primary care team members, including 16 primary care providers, 12 registered nurses, and 9 licensed practical nurses. We performed content analysis using a distributed cognition approach, identifying patterns of information transmission across people and artifacts (e.g. electronic health records). Results illustrate challenges (e.g. information overload) as well as strategies used to overcome challenges. Various communication paths were used. Some team members served as intermediaries, processing information before relaying it. Artifacts were used as memory aids. Health information technology should address the risks of distributed work by supporting awareness of team and task status for reliable management of results.

Electronic Triggers to Identify Delays in Follow-Up of Mammography: Harnessing the Power of Big Data in Health Care.

PURPOSE: We previously developed electronic triggers to automatically flag records for patients experiencing potential delays in diagnostic evaluation for certain cancers. Because of the unique clinical, logistic, and legal aspects of mammography, this study was conducted to evaluate the effectiveness of a trigger to flag delayed follow-up on mammography. METHODS: An algorithm was developed to detect delays in follow-up of abnormal mammographic results (>60 days for BI-RADS® 0, 4, and 5 and >7 months for BI-RADS 3) using clinical data in the electronic health record. Flagged records were then manually reviewed to determine the trigger's performance characteristics (positive and negative predictive value, sensitivity, and specificity). The frequency of delays and patient communication related to abnormal results, reasons for lack of follow-up, and whether patients were subsequently diagnosed with breast cancer were also assessed. RESULTS: Of 365,686 patients seen between January 1, 2010, and May 31, 2015, the trigger identified 2,129 patients with abnormal findings on mammography, of whom it flagged 552 as having delays in follow-up. From these, review of 400 randomly selected records revealed 283 true delays (positive predictive value, 71%; 95% confidence interval, 66%-75%), including 280 records without any documented plan and three patients with plans that were not adhered to. Transcription and reporting inconsistencies were identified in 27% of externally performed mammographic reports. Only 335 records (84%) contained specific documentation that the patient was informed of the abnormal result. CONCLUSIONS: Care delays appear to continue despite federal laws requiring patient notification of mammographic results within 30 days. Clinical application of mammography-related triggers could help detect these delays.

Electronic Health Record Alert-Related Workload as a Predictor of Burnout in Primary Care Providers.

BACKGROUND: Electronic health records (EHRs) have been shown to increase physician workload. One EHR feature that contributes to increased workload is asynchronous alerts (also known as inbox notifications) related to test results, referral responses, medication refill requests, and messages from physicians and other health care professionals. This alert-related workload results in negative cognitive outcomes, but its effect on affective outcomes, such as burnout, has been understudied. OBJECTIVES: To examine EHR alert-related workload (both objective and subjective) as a predictor of burnout in primary care providers (PCPs), in order to ultimately inform interventions aimed at reducing burnout due to alert workload. METHODS: A cross-sectional questionnaire and focus group of 16 PCPs at a large medical center in the southern United States. RESULTS: Subjective, but not objective, alert workload was related to two of the three dimensions of burnout, including physical fatigue (p = 0.02) and cognitive weariness (p = 0.04), when controlling for organizational tenure. To reduce alert workload and subsequent burnout, participants indicated a desire to have protected time for alert management, fewer unnecessary alerts, and improvements to the EHR system. CONCLUSIONS: Burnout associated with alert workload may be in part due to subjective differences at an individual level, and not solely a function of the objective work environment. This suggests the need for both individual and organizational-level interventions to improve alert workload and subsequent burnout. Additional research should confirm these findings in larger, more representative samples.

Errors in Diagnosis of Spinal Epidural Abscesses in the Era of Electronic Health Records.

PURPOSE: With this study, we set out to identify missed opportunities in diagnosis of spinal epidural abscesses to outline areas for process improvement. METHODS: Using a large national clinical data repository, we identified all patients with a new diagnosis of spinal epidural abscess in the Department of Veterans Affairs (VA) during 2013. Two physicians independently conducted retrospective chart reviews on 250 randomly selected patients and evaluated their records for red flags (eg, unexplained weight loss, neurological deficits, and fever) 90 days prior to diagnosis. Diagnostic errors were defined as missed opportunities to evaluate red flags in a timely or appropriate manner. Reviewers gathered information about process breakdowns related to patient factors, the patient-provider encounter, test performance and interpretation, test follow-up and tracking, and the referral process. Reviewers also determined harm and time lag between red flags and definitive diagnoses. RESULTS: Of 250 patients, 119 had a new diagnosis of spinal epidural abscess, 66 (55.5%) of which experienced diagnostic error. Median time to diagnosis in error cases was 12 days, compared with 4 days in cases without error (P <.01). Red flags that were frequently not evaluated in error cases included unexplained fever (n = 57; 86.4%), focal neurological deficits with progressive or disabling symptoms (n = 54; 81.8%), and active infection (n = 54; 81.8%). Most errors involved breakdowns during the patient-provider encounter (n = 60; 90.1%), including failures in information gathering/integration, and were associated with temporary harm (n = 43; 65.2%). CONCLUSION: Despite wide availability of clinical data, errors in diagnosis of spinal epidural abscesses are common and involve inadequate history, physical examination, and test ordering. Solutions should include renewed attention to basic clinical skills.

Safety huddles to proactively identify and address electronic health record safety.

OBJECTIVE: Methods to identify and study safety risks of electronic health records (EHRs) are underdeveloped and largely depend on limited end-user reports. "Safety huddles" have been found useful in creating a sense of collective situational awareness that increases an organization's capacity to respond to safety concerns. We explored the use of safety huddles for identifying and learning about EHR-related safety concerns. DESIGN: Data were obtained from daily safety huddle briefing notes recorded at a single midsized tertiary-care hospital in the United States over 1 year. Huddles were attended by key administrative, clinical, and information technology staff. We conducted a content analysis of huddle notes to identify what EHR-related safety concerns were discussed. We expanded a previously developed EHR-related error taxonomy to categorize types of EHR-related safety concerns recorded in the notes. RESULTS: On review of daily huddle notes spanning 249 days, we identified 245 EHR-related safety concerns. For our analysis, we defined EHR technology to include a specific EHR functionality, an entire clinical software application, or the hardware system. Most concerns (41.6%) involved " EHR technology working incorrectly, " followed by 25.7% involving " EHR technology not working at all. " Concerns related to "EHR technology missing or absent" accounted for 16.7%, whereas 15.9% were linked to " user errors ." CONCLUSIONS: Safety huddles promoted discussion of several technology-related issues at the organization level and can serve as a promising technique to identify and address EHR-related safety concerns. Based on our findings, we recommend that health care organizations consider huddles as a strategy to promote understanding and improvement of EHR safety.