A multi-site randomized trial of a clinical decision support intervention to improve problem list completeness.

OBJECTIVE: To improve problem list documentation and care quality. MATERIALS AND METHODS: We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS: There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION: The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION: An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.

Trends in buprenorphine treatment disparities during the COVID pandemic in Massachusetts.

Background: Racial, sex, and age disparities in buprenorphine treatment have previously been demonstrated. We evaluated trends in buprenorphine treatment disparities before and after the onset of the COVID pandemic in Massachusetts. Methods: This cross-sectional study used data from an integrated health system comparing 12-months before and after the March 2020 Massachusetts COVID state of emergency declaration, excluding March as a washout period. Among patients with a clinical encounter during the study periods with a diagnosis of opioid use disorder or opioid poisoning, we extracted outpatient buprenorphine prescription rates by age, sex, race and ethnicity, and language. Generating univariable and multivariable Poisson regression models, we calculated the probability of receiving buprenorphine. Results: Among 4,530 patients seen in the period before the COVID emergency declaration, 57.9% received buprenorphine. Among 3,653 patients seen in the second time period, 55.1% received buprenorphine. Younger patients (<24) had a lower likelihood of receiving buprenorphine in both time periods (adjusted prevalence ratio (aPR), 0.56; 95% CI, 0.42-0.75 before vs. aPR, 0.76; 95% CI, 0.60-0.96 after). Male patients had a greater likelihood of receiving buprenorphine compared to female patients in both time periods (aPR: 1.05; 95% CI, 1.00-1.11 vs. aPR: 1.09; 95% CI, 1.02-1.16). Racial disparities emerged in the time period following the COVID pandemic, with non-Hispanic Black patients having a lower likelihood of receiving buprenorphine compared to non-Hispanic white patients in the second time period (aPR, 0.85; 95% CI, 0.72-0.99). Conclusions: Following the onset of the COVID pandemic in Massachusetts, ongoing racial, age, and gender disparities were evident in buprenorphine treatment with younger, Black, and female patients less likely to be treated with buprenorphine across an integrated health system.

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

Background: Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited. Objective: Investigate whether user override comments can be used to discover malfunctions. Methods: We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: "broken," "not broken, but could be improved," and "not broken." We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth. Results: Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738. Discussion: Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts. Conclusion: Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.

Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study.

OBJECTIVE: Developing effective and reliable rule-based clinical decision support (CDS) alerts and reminders is challenging. Using a previously developed taxonomy for alert malfunctions, we identified best practices for developing, testing, implementing, and maintaining alerts and avoiding malfunctions. MATERIALS AND METHODS: We identified 72 initial practices from the literature, interviews with subject matter experts, and prior research. To refine, enrich, and prioritize the list of practices, we used the Delphi method with two rounds of consensus-building and refinement. We used a larger than normal panel of experts to include a wide representation of CDS subject matter experts from various disciplines. RESULTS: 28 experts completed Round 1 and 25 completed Round 2. Round 1 narrowed the list to 47 best practices in 7 categories: knowledge management, designing and specifying, building, testing, deployment, monitoring and feedback, and people and governance. Round 2 developed consensus on the importance and feasibility of each best practice. DISCUSSION: The Delphi panel identified a range of best practices that may help to improve implementation of rule-based CDS and avert malfunctions. Due to limitations on resources and personnel, not everyone can implement all best practices. The most robust processes require investing in a data warehouse. Experts also pointed to the issue of shared responsibility between the healthcare organization and the electronic health record vendor. CONCLUSION: These 47 best practices represent an ideal situation. The research identifies the balance between importance and difficulty, highlights the challenges faced by organizations seeking to implement CDS, and describes several opportunities for future research to reduce alert malfunctions.

Using statistical anomaly detection models to find clinical decision support malfunctions.

Objective: Malfunctions in Clinical Decision Support (CDS) systems occur due to a multitude of reasons, and often go unnoticed, leading to potentially poor outcomes. Our goal was to identify malfunctions within CDS systems. Methods: We evaluated 6 anomaly detection models: (1) Poisson Changepoint Model, (2) Autoregressive Integrated Moving Average (ARIMA) Model, (3) Hierarchical Divisive Changepoint (HDC) Model, (4) Bayesian Changepoint Model, (5) Seasonal Hybrid Extreme Studentized Deviate (SHESD) Model, and (6) E-Divisive with Median (EDM) Model and characterized their ability to find known anomalies. We analyzed 4 CDS alerts with known malfunctions from the Longitudinal Medical Record (LMR) and Epic® (Epic Systems Corporation, Madison, WI, USA) at Brigham and Women's Hospital, Boston, MA. The 4 rules recommend lead testing in children, aspirin therapy in patients with coronary artery disease, pneumococcal vaccination in immunocompromised adults and thyroid testing in patients taking amiodarone. Results: Poisson changepoint, ARIMA, HDC, Bayesian changepoint and the SHESD model were able to detect anomalies in an alert for lead screening in children and in an alert for pneumococcal conjugate vaccine in immunocompromised adults. EDM was able to detect anomalies in an alert for monitoring thyroid function in patients on amiodarone. Conclusions: Malfunctions/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.

Development and evaluation of a novel user interface for reviewing clinical microbiology results.

Background: Microbiology laboratory results are complex and cumbersome to review. We sought to develop a new review tool to improve the ease and accuracy of microbiology results review. Methods: We observed and informally interviewed clinicians to determine areas in which existing microbiology review tools were lacking. We developed a new tool that reorganizes microbiology results by time and organism. We conducted a scenario-based usability evaluation to compare the new tool to existing legacy tools, using a balanced block design. Results: The average time-on-task decreased from 45.3 min for the legacy tools to 27.1 min for the new tool (P < .0001). Total errors decreased from 41 with the legacy tools to 19 with the new tool (P = .0068). The average Single Ease Question score was 5.65 (out of 7) for the new tool, compared to 3.78 for the legacy tools (P < .0001). The new tool scored 88 ("Excellent") on the System Usability Scale. Conclusions: The new tool substantially improved efficiency, accuracy, and usability. It was subsequently integrated into the electronic health record and rolled out system-wide. This project provides an example of how clinical and informatics teams can innovative alongside a commercial Electronic Health Record (EHR).

Outpatient CPOE orders discontinued due to 'erroneous entry': prospective survey of prescribers' explanations for errors.

BACKGROUND: Computerised prescriber order entry (CPOE) systems users often discontinue medications because the initial order was erroneous. OBJECTIVE: To elucidate error types by querying prescribers about their reasons for discontinuing outpatient medication orders that they had self-identified as erroneous. METHODS: During a nearly 3 year retrospective data collection period, we identified 57 972 drugs discontinued with the reason 'Error (erroneous entry)." Because chart reviews revealed limited information about these errors, we prospectively studied consecutive, discontinued erroneous orders by querying prescribers in near-real-time to learn more about the erroneous orders. RESULTS: From January 2014 to April 2014, we prospectively emailed prescribers about outpatient drug orders that they had discontinued due to erroneous initial order entry. Of 2 50 806 medication orders in these 4 months, 1133 (0.45%) of these were discontinued due to error. From these 1133, we emailed 542 unique prescribers to ask about their reason(s) for discontinuing these mediation orders in error. We received 312 responses (58% response rate). We categorised these responses using a previously published taxonomy. The top reasons for these discontinued erroneous orders included: medication ordered for wrong patient (27.8%, n=60); wrong drug ordered (18.5%, n=40); and duplicate order placed (14.4%, n=31). Other common discontinued erroneous orders related to drug dosage and formulation (eg, extended release versus not). Oxycodone (3%) was the most frequent drug discontinued error. CONCLUSION: Drugs are not infrequently discontinued 'in error.' Wrong patient and wrong drug errors constitute the leading types of erroneous prescriptions recognised and discontinued by prescribers. Data regarding erroneous medication entries represent an important source of intelligence about how CPOE systems are functioning and malfunctioning, providing important insights regarding areas for designing CPOE more safely in the future.

Clinical decision support alert malfunctions: analysis and empirically derived taxonomy.

Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

A Picture is Worth 1,000 Words. The Use of Clinical Images in Electronic Medical Records.

OBJECTIVE: To understand how clinicians utilize image uploading tools in a home grown electronic health records (EHR) system. METHODS: A content analysis of patient notes containing non-radiological images from the EHR was conducted. Images from 4,000 random notes from July 1, 2009 - June 30, 2010 were reviewed and manually coded. Codes were assigned to four properties of the image: (1) image type, (2) role of image uploader (e.g. MD, NP, PA, RN), (3) practice type (e.g. internal medicine, dermatology, ophthalmology), and (4) image subject. RESULTS: 3,815 images from image-containing notes stored in the EHR were reviewed and manually coded. Of those images, 32.8% were clinical and 66.2% were non-clinical. The most common types of the clinical images were photographs (38.0%), diagrams (19.1%), and scanned documents (14.4%). MDs uploaded 67.9% of clinical images, followed by RNs with 10.2%, and genetic counselors with 6.8%. Dermatology (34.9%), ophthalmology (16.1%), and general surgery (10.8%) uploaded the most clinical images. The content of clinical images referencing body parts varied, with 49.8% of those images focusing on the head and neck region, 15.3% focusing on the thorax, and 13.8% focusing on the lower extremities. CONCLUSION: The diversity of image types, content, and uploaders within a home grown EHR system reflected the versatility and importance of the image uploading tool. Understanding how users utilize image uploading tools in a clinical setting highlights important considerations for designing better EHR tools and the importance of interoperability between EHR systems and other health technology.

Analysis of variations in the display of drug names in computerized prescriber-order-entry systems.

PURPOSE: The variations in how drug names are displayed in computerized prescriber-order-entry (CPOE) systems were analyzed to determine their contribution to potential medication errors. METHODS: A diverse set of 10 inpatient and outpatient CPOE system vendors and self-developed CPOE systems in 6 U.S. healthcare institutions was evaluated. A team of pharmacists, physicians, patient-safety experts, and informatics experts created a CPOE assessment tool to standardize the assessment of CPOE features across the systems studied. Hypothetical scenarios were conducted with test patients to study the medication ordering workflow and ways in which medications were displayed in each system. Brand versus generic drug name ordering was studied at 1 large outpatient system to understand why prescribers ordered both brand and generic forms of the same drug. RESULTS: Widespread variations in the display of drug names were observed both within and across the 6 study sites and 10 systems, including the inconsistent display of brand and generic names. Some displayed drugs differently even on the same screen. Combination products were often displayed inconsistently, and some systems required prescribers to know the first drug listed in the combination in order for the correct product to appear in a search. It also appeared that prescribers may have prescribed both brand and generic forms of the same medication, creating the potential for drug duplication errors. CONCLUSION: A review of 10 CPOE systems revealed that medication names were displayed inconsistently, which can result in confusion or errors in reviewing, selecting, and ordering medications.

Computerized prescriber order entry-related patient safety reports: analysis of 2522 medication errors.

Objective: To examine medication errors potentially related to computerized prescriber order entry (CPOE) and refine a previously published taxonomy to classify them. Materials and Methods: We reviewed all patient safety medication reports that occurred in the medication ordering phase from 6 sites participating in a United States Food and Drug Administration-sponsored project examining CPOE safety. Two pharmacists independently reviewed each report to confirm whether the error occurred in the ordering/prescribing phase and was related to CPOE. For those related to CPOE, we assessed whether CPOE facilitated (actively contributed to) the error or failed to prevent the error (did not directly cause it, but optimal systems could have potentially prevented it). A previously developed taxonomy was iteratively refined to classify the reports. Results: Of 2522 medication error reports, 1308 (51.9%) were related to CPOE. Of these, CPOE facilitated the error in 171 (13.1%) and potentially could have prevented the error in 1137 (86.9%). The most frequent categories of "what happened to the patient" were delays in medication reaching the patient, potentially receiving duplicate drugs, or receiving a higher dose than indicated. The most frequent categories for "what happened in CPOE" included orders not routed to or received at the intended location, wrong dose ordered, and duplicate orders. Variations were seen in the format, categorization, and quality of reports, resulting in error causation being assignable in only 403 instances (31%). Discussion and Conclusion: Errors related to CPOE commonly involved transmission errors, erroneous dosing, and duplicate orders. More standardized safety reporting using a common taxonomy could help health care systems and vendors learn and implement prevention strategies.

Variation in high-priority drug-drug interaction alerts across institutions and electronic health records.

Objective: The United States Office of the National Coordinator for Health Information Technology sponsored the development of a "high-priority" list of drug-drug interactions (DDIs) to be used for clinical decision support. We assessed current adoption of this list and current alerting practice for these DDIs with regard to alert implementation (presence or absence of an alert) and display (alert appearance as interruptive or passive). Materials and methods: We conducted evaluations of electronic health records (EHRs) at a convenience sample of health care organizations across the United States using a standardized testing protocol with simulated orders. Results: Evaluations of 19 systems were conducted at 13 sites using 14 different EHRs. Across systems, 69% of the high-priority DDI pairs produced alerts. Implementation and display of the DDI alerts tested varied between systems, even when the same EHR vendor was used. Across the drug pairs evaluated, implementation and display of DDI alerts differed, ranging from 27% (4/15) to 93% (14/15) implementation. Discussion: Currently, there is no standard of care covering which DDI alerts to implement or how to display them to providers. Opportunities to improve DDI alerting include using differential displays based on DDI severity, establishing improved lists of clinically significant DDIs, and thoroughly reviewing organizational implementation decisions regarding DDIs. Conclusion: DDI alerting is clinically important but not standardized. There is significant room for improvement and standardization around evidence-based DDIs.

Association between shunt-responsive idiopathic normal pressure hydrocephalus and alcohol.

OBJECTIVE Idiopathic normal pressure hydrocephalus (iNPH) is characterized by ventriculomegaly, gait difficulty, incontinence, and dementia. The symptoms can be ameliorated by CSF drainage. The object of this study was to identify factors associated with shunt-responsive iNPH. METHODS The authors reviewed the medical records of 529 patients who underwent shunt placement for iNPH at their institution between July 2001 and March 2015. Variables associated with shunt-responsive iNPH were identified using bivariate and multivariate analyses. Detailed alcohol consumption information was obtained for 328 patients and was used to examine the relationship between alcohol and shunt-responsive iNPH. A computerized patient registry from 2 academic medical centers was queried to determine the prevalence of alcohol abuse among 1665 iNPH patients. RESULTS Bivariate analysis identified associations between shunt-responsive iNPH and gait difficulty (OR 4.59, 95% CI 2.32-9.09; p < 0.0001), dementia (OR 1.79, 95% CI 1.14-2.80; p = 0.01), incontinence (OR 1.77, 95% CI 1.13-2.76; p = 0.01), and alcohol use (OR 1.98, 95% CI 1.23-3.16; p = 0.03). Borderline significance was observed for hyperlipidemia (OR 1.56, 95% CI 0.99-2.45; p = 0.054), a family history of hyperlipidemia (OR 3.09, 95% CI 0.93-10.26, p = 0.054), and diabetes (OR 1.83, 95% CI 0.96-3.51; p = 0.064). Multivariate analysis identified associations with gait difficulty (OR 3.98, 95% CI 1.81-8.77; p = 0.0006) and alcohol (OR 1.94, 95% CI 1.10-3.39; p = 0.04). Increased alcohol intake correlated with greater improvement after CSF drainage. Alcohol abuse was 2.5 times more prevalent among iNPH patients than matched controls. CONCLUSIONS Alcohol consumption is associated with the development of shunt-responsive iNPH.

Analysis of clinical decision support system malfunctions: a case series and survey.

OBJECTIVE: To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. MATERIALS AND METHODS: We identified and investigated several CDSS malfunctions at Brigham and Women's Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. RESULTS: We identified four CDSS malfunctions at Brigham and Women's Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually. DISCUSSION: CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate. CONCLUSION: CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

Quantitative evaluation of changes in gait after extended cerebrospinal fluid drainage for normal pressure hydrocephalus.

Idiopathic normal pressure hydrocephalus (iNPH) is characterized by gait instability, urinary incontinence and cognitive dysfunction. These symptoms can be relieved by cerebrospinal fluid (CSF) drainage, but the time course and nature of the improvements are poorly characterized. Attempts to prospectively identify iNPH patients responsive to CSF drainage by evaluating presenting gait quality or via extended lumbar cerebrospinal fluid drainage (eLCD) trials are common, but the reliability of such approaches is unclear. Here we combine eLCD trials with computerized quantitative gait measurements to predict shunt responsiveness in patients undergoing evaluation for possible iNPH. In this prospective cohort study, 50 patients presenting with enlarged cerebral ventricles and gait, urinary, and/or cognitive difficulties were evaluated for iNPH using a computerized gait analysis system during a 3day trial of eLCD. Gait speed, stride length, cadence, and the Timed Up and Go test were quantified before and during eLCD. Qualitative assessments of incontinence and cognition were obtained throughout the eLCD trial. Patients who improved after eLCD underwent ventriculoperitoneal shunt placement, and symptoms were reassessed serially over the next 3 to 15months. There was no significant difference in presenting gait characteristics between patients who improved after drainage and those who did not. Gait improvement was not observed until 2 or more days of continuous drainage in most cases. Symptoms improved after eLCD in 60% of patients, and all patients who improved after eLCD also improved after shunt placement. The degree of improvement after eLCD correlated closely with that observed after shunt placement.

Problem list completeness in electronic health records: A multi-site study and assessment of success factors.

OBJECTIVE: To assess problem list completeness using an objective measure across a range of sites, and to identify success factors for problem list completeness. METHODS: We conducted a retrospective analysis of electronic health record data and interviews at ten healthcare organizations within the United States, United Kingdom, and Argentina who use a variety of electronic health record systems: four self-developed and six commercial. At each site, we assessed the proportion of patients who have diabetes recorded on their problem list out of all patients with a hemoglobin A1c elevation>=7.0%, which is diagnostic of diabetes. We then conducted interviews with informatics leaders at the four highest performing sites to determine factors associated with success. Finally, we surveyed all the sites about common practices implemented at the top performing sites to determine whether there was an association between problem list management practices and problem list completeness. RESULTS: Problem list completeness across the ten sites ranged from 60.2% to 99.4%, with a mean of 78.2%. Financial incentives, problem-oriented charting, gap reporting, shared responsibility, links to billing codes, and organizational culture were identified as success factors at the four hospitals with problem list completeness at or near 90.0%. DISCUSSION: Incomplete problem lists represent a global data integrity problem that could compromise quality of care and put patients at risk. There was a wide range of problem list completeness across the healthcare facilities. Nevertheless, some facilities have achieved high levels of problem list completeness, and it is important to better understand the factors that contribute to success to improve patient safety. CONCLUSION: Problem list completeness varies substantially across healthcare facilities. In our review of EHR systems at ten healthcare facilities, we identified six success factors which may be useful for healthcare organizations seeking to improve the quality of their problem list documentation: financial incentives, problem oriented charting, gap reporting, shared responsibility, links to billing codes, and organizational culture.