How Safe are Outpatient Electronic Health Records? An Evaluation of Medication-Related Decision Support using the Ambulatory Electronic Health Record Evaluation Tool.

BACKGROUND: The purpose of the Ambulatory Electronic Health Record (EHR) Evaluation Tool is to provide outpatient clinics with an assessment that they can use to measure the ability of the EHR system to detect and prevent common prescriber errors. The tool consists of a medication safety test and a medication reconciliation module. OBJECTIVES: The goal of this study was to perform a broad evaluation of outpatient medication-related decision support using the Ambulatory EHR Evaluation Tool. METHODS: We performed a cross-sectional study with 10 outpatient clinics using the Ambulatory EHR Evaluation Tool. For the medication safety test, clinics were provided test patients and associated medication test orders to enter in their EHR, where they recorded any advice or information they received. Once finished, clinics received an overall percentage score of unsafe orders detected and individual order category scores. For the medication reconciliation module, clinics were asked to electronically reconcile two medication lists, where modifications were made by adding and removing medications and changing the dosage of select medications. RESULTS: For the medication safety test, the mean overall score was 57%, with the highest score being 70%, and the lowest score being 40%. Clinics performed well in the drug allergy (100%), drug dose daily (85%), and inappropriate medication combinations (74%) order categories. Order categories with the lowest performance were drug laboratory (10%) and drug monitoring (3%). Most clinics (90%) scored a 0% in at least one order category. For the medication reconciliation module, only one clinic (10%) could reconcile medication lists electronically; however, there was no clinical decision support available that checked for drug interactions. CONCLUSION: We evaluated a sample of ambulatory practices around their medication-related decision support and found that advanced capabilities within these systems have yet to be widely implemented. The tool was practical to use and identified substantial opportunities for improvement in outpatient medication safety.

Comprehensive Pharmacist-led Transitions-of-care Medication Management around Hospital Discharge Adds Modest Cost Relative to Usual Care: Time-and-Motion Cost Analysis.

Optimal medication management is important during hospitalization and at discharge because post-discharge adverse drug events (ADEs) are common, often preventable, and contribute to patient harms, healthcare utilization, and costs. Conduct a cost analysis of a comprehensive pharmacist-led transitions-of-care medication management intervention for older adults during and after hospital discharge. Twelve intervention components addressed medication reconciliation, medication review, and medication adherence. Trained, experienced pharmacists delivered the intervention to older adults with chronic comorbidities at 2 large U.S. academic centers. To quantify and categorize time spent on the intervention, we conducted a time-and-motion analysis of study pharmacists over 36 sequential workdays (14 519 min) involving 117 patients. For 40 patients' hospitalizations, we observed all intervention activities. We used the median minutes spent and pharmacist wages nationally to calculate cost per hospitalization (2020 U.S. dollars) from the hospital perspective, relative to usual care. Pharmacists spent a median of 66.9 min per hospitalization (interquartile range 46.1-90.1), equating to $101 ($86 to $116 in sensitivity analyses). In unadjusted analyses, study site was associated with time spent (medians 111 and 51.8 min) while patient primary language, discharge disposition, number of outpatient medications, and patient age were not. In this cost analysis, comprehensive medication management around discharge cost about $101 per hospitalization, with variation across sites. This cost is at least an order of magnitude less than published costs associated with ADEs, hospital readmissions, or other interventions designed to reduce readmissions. Work is ongoing to assess the current intervention's effectiveness.

Key use cases for artificial intelligence to reduce the frequency of adverse drug events: a scoping review.

Adverse drug events (ADEs) represent one of the most prevalent types of health-care-related harm, and there is substantial room for improvement in the way that they are currently predicted and detected. We conducted a scoping review to identify key use cases in which artificial intelligence (AI) could be leveraged to reduce the frequency of ADEs. We focused on modern machine learning techniques and natural language processing. 78 articles were included in the scoping review. Studies were heterogeneous and applied various AI techniques covering a wide range of medications and ADEs. We identified several key use cases in which AI could contribute to reducing the frequency and consequences of ADEs, through prediction to prevent ADEs and early detection to mitigate the effects. Most studies (73 [94%] of 78) assessed technical algorithm performance, and few studies evaluated the use of AI in clinical settings. Most articles (58 [74%] of 78) were published within the past 5 years, highlighting an emerging area of study. Availability of new types of data, such as genetic information, and access to unstructured clinical notes might further advance the field.

The Development and Piloting of the Ambulatory Electronic Health Record Evaluation Tool: Lessons Learned.

BACKGROUND: Substantial research has been performed about the impact of computerized physician order entry on medication safety in the inpatient setting; however, relatively little has been done in ambulatory care, where most medications are prescribed. OBJECTIVE: To outline the development and piloting process of the Ambulatory Electronic Health Record (EHR) Evaluation Tool and to report the quantitative and qualitative results from the pilot. METHODS: The Ambulatory EHR Evaluation Tool closely mirrors the inpatient version of the tool, which is administered by The Leapfrog Group. The tool was piloted with seven clinics in the United States, each using a different EHR. The tool consists of a medication safety test and a medication reconciliation module. For the medication test, clinics entered test patients and associated test orders into their EHR and recorded any decision support they received. An overall percentage score of unsafe orders detected, and order category scores were provided to clinics. For the medication reconciliation module, clinics demonstrated how their EHR electronically detected discrepancies between two medication lists. RESULTS: For the medication safety test, the clinics correctly alerted on 54.6% of unsafe medication orders. Clinics scored highest in the drug allergy (100%) and drug-drug interaction (89.3%) categories. Lower scoring categories included drug age (39.3%) and therapeutic duplication (39.3%). None of the clinics alerted for the drug laboratory or drug monitoring orders. In the medication reconciliation module, three (42.8%) clinics had an EHR-based medication reconciliation function; however, only one of those clinics could demonstrate it during the pilot. CONCLUSION: Clinics struggled in areas of advanced decision support such as drug age, drug laboratory, and drub monitoring. Most clinics did not have an EHR-based medication reconciliation function and this process was dependent on accessing patients' medication lists. Wider use of this tool could improve outpatient medication safety and can inform vendors about areas of improvement.

Impact of a problem-oriented view on clinical data retrieval.

OBJECTIVE: The electronic health record (EHR) data deluge makes data retrieval more difficult, escalating cognitive load and exacerbating clinician burnout. New auto-summarization techniques are needed. The study goal was to determine if problem-oriented view (POV) auto-summaries improve data retrieval workflows. We hypothesized that POV users would perform tasks faster, make fewer errors, be more satisfied with EHR use, and experience less cognitive load as compared with users of the standard view (SV). METHODS: Simple data retrieval tasks were performed in an EHR simulation environment. A randomized block design was used. In the control group (SV), subjects retrieved lab results and medications by navigating to corresponding sections of the electronic record. In the intervention group (POV), subjects clicked on the name of the problem and immediately saw lab results and medications relevant to that problem. RESULTS: With POV, mean completion time was faster (173 seconds for POV vs 205 seconds for SV; P < .0001), the error rate was lower (3.4% for POV vs 7.7% for SV; P = .0010), user satisfaction was greater (System Usability Scale score 58.5 for POV vs 41.3 for SV; P < .0001), and cognitive task load was less (NASA Task Load Index score 0.72 for POV vs 0.99 for SV; P < .0001). DISCUSSION: The study demonstrates that using a problem-based auto-summary has a positive impact on 4 aspects of EHR data retrieval, including cognitive load. CONCLUSION: EHRs have brought on a data deluge, with increased cognitive load and physician burnout. To mitigate these increases, further development and implementation of auto-summarization functionality and the requisite knowledge base are needed.

Derivation of a Clinical Risk Score to Predict 14-Day Occurrence of Hypoxia, ICU Admission, and Death Among Patients with Coronavirus Disease 2019.

BACKGROUND: Uncertainty surrounding COVID-19 regarding rapid progression to acute respiratory distress syndrome and unusual clinical characteristics make discharge from a monitored setting challenging. A clinical risk score to predict 14-day occurrence of hypoxia, ICU admission, and death is unavailable. OBJECTIVE: Derive and validate a risk score to predict suitability for discharge from a monitored setting among an early cohort of patients with COVID-19. DESIGN: Model derivation and validation in a retrospective cohort. We built a manual forward stepwise logistic regression model to identify variables associated with suitability for discharge and assigned points to each variable. Event-free patients were included after at least 14 days of follow-up. PARTICIPANTS: All adult patients with a COVID-19 diagnosis between March 1, 2020, and April 12, 2020, in 10 hospitals in Massachusetts, USA. MAIN MEASURES: Fourteen-day composite predicting hypoxia, ICU admission, and death. We calculated a risk score for each patient as a predictor of suitability for discharge evaluated by area under the curve. KEY RESULTS: Of 2059 patients with COVID-19, 1326 met inclusion. The 1014-patient training cohort had a mean age of 58 years, was 56% female, and 65% had at least one comorbidity. A total of 255 (25%) patients were suitable for discharge. Variables associated with suitability for discharge were age, oxygen saturation, and albumin level, yielding a risk score between 0 and 55. At a cut point of 30, the score had a sensitivity of 83% and specificity of 82%. The respective c-statistic for the derivation and validation cohorts were 0.8939 (95% CI, 0.8687 to 0.9192) and 0.8685 (95% CI, 0.8095 to 0.9275). The score performed similarly for inpatients and emergency department patients. CONCLUSIONS: A 3-item risk score for patients with COVID-19 consisting of age, oxygen saturation, and an acute phase reactant (albumin) using point of care data predicts suitability for discharge and may optimize scarce resources.

The tradeoffs between safety and alert fatigue: Data from a national evaluation of hospital medication-related clinical decision support.

OBJECTIVE: The study sought to evaluate the overall performance of hospitals that used the Computerized Physician Order Entry Evaluation Tool in both 2017 and 2018, along with their performance against fatal orders and nuisance orders. MATERIALS AND METHODS: We evaluated 1599 hospitals that took the test in both 2017 and 2018 by using their overall percentage scores on the test, along with the percentage of fatal orders appropriately alerted on, and the percentage of nuisance orders incorrectly alerted on. RESULTS: Hospitals showed overall improvement; the mean score in 2017 was 58.1%, and this increased to 66.2% in 2018. Fatal order performance improved slightly from 78.8% to 83.0% (P < .001), though there was almost no change in nuisance order performance (89.0% to 89.7%; P = .43). Hospitals alerting on one or more nuisance orders had a 3-percentage-point increase in their overall score. DISCUSSION: Despite the improvement of overall scores in 2017 and 2018, there was little improvement in fatal order performance, suggesting that hospitals are not targeting the deadliest orders first. Nuisance order performance showed almost no improvement, and some hospitals may be achieving higher scores by overalerting, suggesting that the thresholds for which alerts are fired from are too low. CONCLUSIONS: Although hospitals improved overall from 2017 to 2018, there is still important room for improvement for both fatal and nuisance orders. Hospitals that incorrectly alerted on one or more nuisance orders had slightly higher overall performance, suggesting that some hospitals may be achieving higher scores at the cost of overalerting, which has the potential to cause clinician burnout and even worsen safety.

Design and testing of a mobile health application rating tool.

Mobile health applications ("apps") have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps' potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with ≥4 stars and <4-stars and then rated them with THESIS to examine the tool's interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96-3.09]), but especially poorly for privacy/security (2.21 out of 5 [95% CI, 2.11-2.32]), interoperability (1.75 [95% CI, 1.59-1.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30-1.56]). Ratings using THESIS had fair interrater reliability (κ = 0.3-0.6) and excellent scale reliability (ɑ = 0.85). Correlation with traditional star ratings was low (r = 0.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.

National Trends in the Safety Performance of Electronic Health Record Systems From 2009 to 2018.

Importance: Despite the broad adoption of electronic health record (EHR) systems across the continuum of care, safety problems persist. Objective: To measure the safety performance of operational EHRs in hospitals across the country during a 10-year period. Design, Setting, and Participants: This case series included all US adult hospitals nationwide that used the National Quality Forum Health IT Safety Measure EHR computerized physician order entry safety test administered by the Leapfrog Group between 2009 and 2018. Data were analyzed from July 1, 2018 to December 1, 2019. Exposure: The Health IT Safety Measure test, which uses simulated medication orders that have either injured or killed patients previously to evaluate how well hospital EHRs could identify medication errors with potential for patient harm. Main Outcomes and Measures: Descriptive statistics for performance on the assessment test over time were calculated at the overall test score level, type of decision support category level, and EHR vendor level. Results: Among 8657 hospital-years observed during the study, mean (SD) scores on the overall test increased from 53.9% (18.3%) in 2009 to 65.6% (15.4%) in 2018. Mean (SD) hospital score for the categories representing basic clinical decision support increased from 69.8% (20.8%) in 2009 to 85.6% (14.9%) in 2018. For the categories representing advanced clinical decision support, the mean (SD) score increased from 29.6% (22.4%) in 2009 to 46.1% (21.6%) in 2018. There was considerable variation in test performance by EHR. Conclusions and Relevance: These findings suggest that despite broad adoption and optimization of EHR systems in hospitals, wide variation in the safety performance of operational EHR systems remains across a large sample of hospitals and EHR vendors. Hospitals using some EHR vendors had significantly higher test scores. Overall, substantial safety risk persists in current hospital EHR systems.

Assessing the safety of electronic health records: a national longitudinal study of medication-related decision support.

BACKGROUND: Electronic health records (EHR) can improve safety via computerised physician order entry with clinical decision support, designed in part to alert providers and prevent potential adverse drug events at entry and before they reach the patient. However, early evidence suggested performance at preventing adverse drug events was mixed. METHODS: We used data from a national, longitudinal sample of 1527 hospitals in the USA from 2009 to 2016 who took a safety performance assessment test using simulated medication orders to test how well their EHR prevented medication errors with potential for patient harm. We calculated the descriptive statistics on performance on the assessment over time, by years of hospital experience with the test and across hospital characteristics. Finally, we used ordinary least squares regression to identify hospital characteristics associated with higher test performance. RESULTS: The average hospital EHR system correctly prevented only 54.0% of potential adverse drug events tested on the 44-order safety performance assessment in 2009; this rose to 61.6% in 2016. Hospitals that took the assessment multiple times performed better in subsequent years than those taking the test the first time, from 55.2% in the first year of test experience to 70.3% in the eighth, suggesting efforts to participate in voluntary self-assessment and improvement may be helpful in improving medication safety performance. CONCLUSION: Hospital medication order safety performance has improved over time but is far from perfect. The specifics of EHR medication safety implementation and improvement play a key role in realising the benefits of computerising prescribing, as organisations have substantial latitude in terms of what they implement. Intentional quality improvement efforts appear to be a critical part of high safety performance and may indicate the importance of a culture of safety.