Single-reviewer electronic phenotyping validation in operational settings: Comparison of strategies and recommendations.

OBJECTIVE: Develop evidence-based recommendations for single-reviewer validation of electronic phenotyping results in operational settings. MATERIAL AND METHODS: We conducted a randomized controlled study to evaluate whether electronic phenotyping results should be used to support manual chart review during single-reviewer electronic phenotyping validation (N=3104). We evaluated the accuracy, duration and cost of manual chart review with and without the availability of electronic phenotyping results, including relevant patient-specific details. The cost of identification of an erroneous electronic phenotyping result was calculated based on the personnel time required for the initial chart review and subsequent adjudication of discrepancies between manual chart review results and electronic phenotype determinations. RESULTS: Providing electronic phenotyping results (vs not providing those results) was associated with improved overall accuracy of manual chart review (98.90% vs 92.46%, p<0.001), decreased review duration per test case (62.43 vs 76.78s, p<0.001), and insignificantly reduced estimated marginal costs of identification of an erroneous electronic phenotyping result ($48.54 vs $63.56, p=0.16). The agreement between chart review and electronic phenotyping results was higher when the phenotyping results were provided (Cohen's kappa 0.98 vs 0.88, p<0.001). As a result, while accuracy improved when initial electronic phenotyping results were correct (99.74% vs 92.67%, N=3049, p<0.001), there was a trend towards decreased accuracy when initial electronic phenotyping results were erroneous (56.67% vs 80.00%, N=55, p=0.07). Electronic phenotyping results provided the greatest benefit for the accurate identification of rare exclusion criteria. DISCUSSION: Single-reviewer chart review of electronic phenotyping can be conducted more accurately, quickly, and at lower cost when supported by electronic phenotyping results. However, human reviewers tend to agree with electronic phenotyping results even when those results are wrong. Thus, the value of providing electronic phenotyping results depends on the accuracy of the underlying electronic phenotyping algorithm. CONCLUSION: We recommend using a mix of phenotyping validation strategies, with the balance of strategies based on the anticipated electronic phenotyping error rate, the tolerance for missed electronic phenotyping errors, as well as the expertise, cost, and availability of personnel involved in chart review and discrepancy adjudication.

GARDE: a standards-based clinical decision support platform for identifying population health management cohorts.

Population health management (PHM) is an important approach to promote wellness and deliver health care to targeted individuals who meet criteria for preventive measures or treatment. A critical component for any PHM program is a data analytics platform that can target those eligible individuals. OBJECTIVE: The aim of this study was to design and implement a scalable standards-based clinical decision support (CDS) approach to identify patient cohorts for PHM and maximize opportunities for multi-site dissemination. MATERIALS AND METHODS: An architecture was established to support bidirectional data exchanges between heterogeneous electronic health record (EHR) data sources, PHM systems, and CDS components. HL7 Fast Healthcare Interoperability Resources and CDS Hooks were used to facilitate interoperability and dissemination. The approach was validated by deploying the platform at multiple sites to identify patients who meet the criteria for genetic evaluation of familial cancer. RESULTS: The Genetic Cancer Risk Detector (GARDE) platform was created and is comprised of four components: (1) an open-source CDS Hooks server for computing patient eligibility for PHM cohorts, (2) an open-source Population Coordinator that processes GARDE requests and communicates results to a PHM system, (3) an EHR Patient Data Repository, and (4) EHR PHM Tools to manage patients and perform outreach functions. Site-specific deployments were performed on onsite virtual machines and cloud-based Amazon Web Services. DISCUSSION: GARDE's component architecture establishes generalizable standards-based methods for computing PHM cohorts. Replicating deployments using one of the established deployment methods requires minimal local customization. Most of the deployment effort was related to obtaining site-specific information technology governance approvals.

Leveraging Artificial Intelligence to Improve Chronic Disease Care: Methods and Application to Pharmacotherapy Decision Support for Type-2 Diabetes Mellitus.

OBJECTIVES: Artificial intelligence (AI), including predictive analytics, has great potential to improve the care of common chronic conditions with high morbidity and mortality. However, there are still many challenges to achieving this vision. The goal of this project was to develop and apply methods for enhancing chronic disease care using AI. METHODS: Using a dataset of 27,904 patients with diabetes, an analytical method was developed and validated for generating a treatment pathway graph which consists of models that predict the likelihood of alternate treatment strategies achieving care goals. An AI-driven clinical decision support system (CDSS) integrated with the electronic health record (EHR) was developed by encapsulating the prediction models in an OpenCDS Web service module and delivering the model outputs through a SMART on FHIR (Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources) web-based dashboard. This CDSS enables clinicians and patients to review relevant patient parameters, select treatment goals, and review alternate treatment strategies based on prediction results. RESULTS: The proposed analytical method outperformed previous machine-learning algorithms on prediction accuracy. The CDSS was successfully integrated with the Epic EHR at the University of Utah. CONCLUSION: A predictive analytics-based CDSS was developed and successfully integrated with the EHR through standards-based interoperability frameworks. The approach used could potentially be applied to many other chronic conditions to bring AI-driven CDSS to the point of care.

Establishing a multidisciplinary initiative for interoperable electronic health record innovations at an academic medical center.

OBJECTIVE: To establish an enterprise initiative for improving health and health care through interoperable electronic health record (EHR) innovations. MATERIALS AND METHODS: We developed a unifying mission and vision, established multidisciplinary governance, and formulated a strategic plan. Key elements of our strategy include establishing a world-class team; creating shared infrastructure to support individual innovations; developing and implementing innovations with high anticipated impact and a clear path to adoption; incorporating best practices such as the use of Fast Healthcare Interoperability Resources (FHIR) and related interoperability standards; and maximizing synergies across research and operations and with partner organizations. RESULTS: University of Utah Health launched the ReImagine EHR initiative in 2016. Supportive infrastructure developed by the initiative include various FHIR-related tooling and a systematic evaluation framework. More than 10 EHR-integrated digital innovations have been implemented to support preventive care, shared decision-making, chronic disease management, and acute clinical care. Initial evaluations of these innovations have demonstrated positive impact on user satisfaction, provider efficiency, and compliance with evidence-based guidelines. Return on investment has included improvements in care; over $35 million in external grant funding; commercial opportunities; and increased ability to adapt to a changing healthcare landscape. DISCUSSION: Key lessons learned include the value of investing in digital innovation initiatives leveraging FHIR; the importance of supportive infrastructure for accelerating innovation; and the critical role of user-centered design, implementation science, and evaluation. CONCLUSION: EHR-integrated digital innovation initiatives can be key assets for enhancing the EHR user experience, improving patient care, and reducing provider burnout.

Integrated displays to improve chronic disease management in ambulatory care: A SMART on FHIR application informed by mixed-methods user testing.

OBJECTIVE: The study sought to evaluate a novel electronic health record (EHR) add-on application for chronic disease management that uses an integrated display to decrease user cognitive load, improve efficiency, and support clinical decision making. MATERIALS AND METHODS: We designed a chronic disease management application using the technology framework known as SMART on FHIR (Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources). We used mixed methods to obtain user feedback on a prototype to support ambulatory providers managing chronic obstructive pulmonary disease. Each participant managed 2 patient scenarios using the regular EHR with and without access to our prototype in block-randomized order. The primary outcome was the percentage of expert-recommended ideal care tasks completed. Timing, keyboard and mouse use, and participant surveys were also collected. User experiences were captured using a retrospective think-aloud interview analyzed by concept coding. RESULTS: With our prototype, the 13 participants completed more recommended care (81% vs 48%; P < .001) and recommended tasks per minute (0.8 vs 0.6; P = .03) over longer sessions (7.0 minutes vs 5.4 minutes; P = .006). Keystrokes per task were lower with the prototype (6 vs 18; P < .001). Qualitative themes elicited included the desire for reliable presentation of information which matches participants' mental models of disease and for intuitive navigation in order to decrease cognitive load. DISCUSSION: Participants completed more recommended care by taking more time when using our prototype. Interviews identified a tension between using the inefficient but familiar EHR vs learning to use our novel prototype. Concept coding of user feedback generated actionable insights. CONCLUSIONS: Mixed methods can support the design and evaluation of SMART on FHIR EHR add-on applications by enhancing understanding of the user experience.

Design, Development, and Initial Evaluation of a Terminology for Clinical Decision Support and Electronic Clinical Quality Measurement.

When coupled with a common information model, a common terminology for clinical decision support (CDS) and electronic clinical quality measurement (eCQM) could greatly facilitate the distributed development and sharing of CDS and eCQM knowledge resources. To enable such scalable knowledge authoring and sharing, we systematically developed an extensible and standards-based terminology for CDS and eCQM in the context of the HL7 Virtual Medical Record (vMR) information model. The development of this terminology entailed three steps: (1) systematic, physician-curated concept identification from sources such as the Health Information Technology Standards Panel (HITSP) and the SNOMED-CT CORE problem list; (2) concept de-duplication leveraging the Unified Medical Language System (UMLS) MetaMap and Metathesaurus; and (3) systematic concept naming using standard terminologies and heuristic algorithms. This process generated 3,046 concepts spanning 68 domains. Evaluation against representative CDS and eCQM resources revealed approximately 50-70% concept coverage, indicating the need for continued expansion of the terminology.

Challenges and Solutions in Optimizing Execution Performance of a Clinical Decision Support-Based Quality Measurement (CDS-QM) Framework.

Given the close relationship between clinical decision support (CDS) and quality measurement (QM), it has been proposed that a standards-based CDS Web service could be leveraged to enable QM. Benefits of such a CDS-QM framework include semantic consistency and implementation efficiency. However, earlier research has identified execution performance as a critical barrier when CDS-QM is applied to large populations. Here, we describe challenges encountered and solutions devised to optimize CDS-QM execution performance. Through these optimizations, the CDS-QM execution time was optimized approximately three orders of magnitude, such that approximately 370,000 patient records can now be evaluated for 22 quality measure groups in less than 5 hours (approximately 2 milliseconds per measure group per patient). Several key optimization methods were identified, with the most impact achieved through population-based retrieval of relevant data, multi-step data staging, and parallel processing. These optimizations have enabled CDS-QM to be operationally deployed at an enterprise level.

Clinical Decision Support-based Quality Measurement (CDS-QM) Framework: Prototype Implementation, Evaluation, and Future Directions.

Electronic quality measurement (QM) and clinical decision support (CDS) are closely related but are typically implemented independently, resulting in significant duplication of effort. While it seems intuitive that technical approaches could be re-used across these two related use cases, such reuse is seldom reported in the literature, especially for standards-based approaches. Therefore, we evaluated the feasibility of using a standards-based CDS framework aligned with anticipated EHR certification criteria to implement electronic QM. The CDS-QM framework was used to automate a complex national quality measure (SCIP-VTE-2) at an academic healthcare system which had previously relied on time-consuming manual chart abstractions. Compared with 305 manually-reviewed reference cases, the recall of automated measurement was 100%. The precision was 96.3% (CI:92.6%-98.5%) for ascertaining the denominator and 96.2% (CI:92.3%-98.4%) for the numerator. We therefore validated that a standards-based CDS-QM framework can successfully enable automated QM, and we identified benefits and challenges with this approach.

Chemical stability of admixtures combining ziconotide with fentanyl or sufentanil during simulated intrathecal administration.

Although the U.S. Food and Drug Administration has not approved the combined use of intrathecal medications, practitioners frequently prescribe combination intrathecal therapy for patients who do not experience adequate analgesia with a single intrathecal agent; however, the chemical stability of an analgesic combination may influence the frequency of pump refills necessary to maintain safe and efective pain control. This investigation was performed to evaluate the chemical stability of admixtures containing 25 mcg/mL ziconotide and either 1000 mcg/mL fentanyl citrate or 1000 mcg/mL sufentanil citrate during simulated intrathecal infusion under laboratory conditions at 37 deg C. Admixtures were prepared from commercial ziconotide (25 mcg/mL) and lyophilized powders of the opioid drugs, sparged with nitrogen to remove dissolved oxygen, and stored in implantable intrathecal pumps at 37 deg C. Samples obtained at various intervals over the course of 40 days were assessed for drug concentrations by using high-performance liquid chromatography. The periods of time that the admixtures retained > or = 90% and > or = 80% of the initial concentrations of each drug (i.e., the 90% and 80% stabilities) were determined by using linear regression and 95% confidence intervals. At study end, ziconotide concentrations averaged 87.5% of the initial concentration in the ziconotide/fentanyl admixture and 89.3% of the initial concentration in the ziconotide/sufentanil admixture; opioid concentrations were unchanged. Ziconotide was 90% stable for 26 days and 80% stable for 58 days (extrapolated) when combined with fentanyl; when combined with sufentanil, ziconotide was 90% stable for 33 days and 80% stable for 68 days (extrapolated). The opioids were stable throughout the study. At the concentrations used in this study, ziconotide/fentanyl and ziconotide/sufentanil admixtures were relatively stable.

Chemical Stability of Admixtures Containing Ziconotide 25 mcg/mL and Morphine Sulfate 10 mg/mL or 20 mg/mL During Simulated Intrathecal Administration.

The chemical stability of an intrathecally administered analgesic combination may influence the frequency of pump refills necessary to maintain safe and effective analgesia. Previous work has shown that the stability of ziconotide at body temperature is reduced substantially by the presence of morphine sulfate 35 mg/mL. The current study was performed to evaluate the chemical stability of admixtures combining ziconotide with lower concentrations of morphine sulfate during simulated intrathecal infusion under laboratory conditions at 37 deg C. Admixtures containing ziconotide 25 mcg/mL and morphine sulfate 10 mg/mL or 20 mg/mL were stored in implantable intrathecal pumps at 37 deg C and in control vials at 37 deg C or 5 deg C. Samples were obtained over 60 days (admixture containing morphine sulfate 10 mg/mL) or 28 days (admixture containing morphine sulfate 20 mg/mL) and drug concentrations were assessed by high-performance liquid chromatography. Estimates of the time intervals that each admixture retained > or= 90% and > or = 80% of the initial concentrations of both drugs (i.e., the 90% and 80% stabilites) were based on 95% confidence bounds obtained via linear regression. Morphine sulfate 10 mg/mL, the mean ziconotide concentration declined to 81.4% of the initial concentration in 60 days, and 90% and 80% stabilites were maintained for 34 days and 65 days, respctively. In the admixture containing morphine sulfate 20 mg/mL, the mean ziconotide concentration declined to 85.3% of the initial concentration in 28 days, and 90% and 80% stabilities were maintained for 19 days and 37 days, respectively. Decreasing the concentration of morphine in an admixture containing ziconotide improves the stablity of ziconotide.