DQAgui: a graphical user interface for the MIRACUM data quality assessment tool.

BACKGROUND: With the growing impact of observational research studies, there is also a growing focus on data quality (DQ). As opposed to experimental study designs, observational research studies are performed using data mostly collected in a non-research context (secondary use). Depending on the number of data elements to be analyzed, DQ reports of data stored within research networks can grow very large. They might be cumbersome to read and important information could be overseen quickly. To address this issue, a DQ assessment (DQA) tool with a graphical user interface (GUI) was developed and provided as a web application. METHODS: The aim was to provide an easy-to-use interface for users without prior programming knowledge to carry out DQ checks and to present the results in a clearly structured way. This interface serves as a starting point for a more detailed investigation of possible DQ irregularities. A user-centered development process ensured the practical feasibility of the interactive GUI. The interface was implemented in the R programming language and aligned to Kahn et al.'s DQ categories conformance, completeness and plausibility. RESULTS: With DQAgui, an R package with a web-app frontend for DQ assessment was developed. The GUI allows users to perform DQ analyses of tabular data sets and to systematically evaluate the results. During the development of the GUI, additional features were implemented, such as analyzing a subset of the data by defining time periods and restricting the analyses to certain data elements. CONCLUSIONS: As part of the MIRACUM project, DQAgui is now being used at ten German university hospitals for DQ assessment and to provide a central overview of the availability of important data elements in a datamap over 2 years. Future development efforts should focus on design optimization and include a usability evaluation.

ToolPool Gesundheitsforschung - A Repository for Software and Services Focused on Supporting Clinical and Epidemiological Research.

The academic research environment is characterized by self-developed, innovative, customized solutions, which are often free to use for third parties with open-source code and open licenses. On the other hand, they are maintained only to a very limited extent after the end of project funding. The ToolPool Gesundheitsforschung addresses the problem of finding ready to use solutions by building a registry of proven and supported tools, services, concepts and consulting offers. The goal is to provide an up-to-date selection of "relevant" solutions for a given domain that are immediately usable and that are actually used by third parties, rather than aiming at a complete list of all solutions which belong to that domain. Proof of relevance and usage must be provided, for example, by concrete application scenarios, experience reports by uninvolved third parties, references in publications or workshops held. Quality assurance is carried out for new entries by an agreed list of admission criteria, for existing entries at least once a year by a special task force. Currently, 79 solutions are represented, this number is to be significantly expanded by involving of new editors from current national funding initiatives in Germany.

Linking a Consortium-Wide Data Quality Assessment Tool with the MIRACUM Metadata Repository.

BACKGROUND: Many research initiatives aim at using data from electronic health records (EHRs) in observational studies. Participating sites of the German Medical Informatics Initiative (MII) established data integration centers to integrate EHR data within research data repositories to support local and federated analyses. To address concerns regarding possible data quality (DQ) issues of hospital routine data compared with data specifically collected for scientific purposes, we have previously presented a data quality assessment (DQA) tool providing a standardized approach to assess DQ of the research data repositories at the MIRACUM consortium's partner sites. OBJECTIVES: Major limitations of the former approach included manual interpretation of the results and hard coding of analyses, making their expansion to new data elements and databases time-consuming and error prone. We here present an enhanced version of the DQA tool by linking it to common data element definitions stored in a metadata repository (MDR), adopting the harmonized DQA framework from Kahn et al and its application within the MIRACUM consortium. METHODS: Data quality checks were consequently aligned to a harmonized DQA terminology. Database-specific information were systematically identified and represented in an MDR. Furthermore, a structured representation of logical relations between data elements was developed to model plausibility-statements in the MDR. RESULTS: The MIRACUM DQA tool was linked to data element definitions stored in a consortium-wide MDR. Additional databases used within MIRACUM were linked to the DQ checks by extending the respective data elements in the MDR with the required information. The evaluation of DQ checks was automated. An adaptable software implementation is provided with the R package DQAstats. CONCLUSION: The enhancements of the DQA tool facilitate the future integration of new data elements and make the tool scalable to other databases and data models. It has been provided to all ten MIRACUM partners and was successfully deployed and integrated into their respective data integration center infrastructure.

A Multi-User Terminology Mapping Toolbox.

Semantic interoperability is a major challenge in multi-center data sharing projects, a challenge that the German Initiative for Medical Informatics is taking up. With respect to laboratory data, enriching site-specific tests and measurements with LOINC codes appears to be a crucial step in supporting cross-institutional research. However, this effort is very time-consuming, as it requires expert knowledge of local site specifics. To ease this process, we developed a generic manual collaborative terminology mapping tool, the MIRACUM Mapper. It allows the creation of arbitrary mapping workflows involving different user roles. A mapping workflow with two user roles has been implemented at University Hospital Erlangen to support the local LOINC mapping. Additionally, the MIRACUM LabVisualizeR provides summary statistics and visualizations of analyte data. We developed a toolbox that facilitates the collaborative creation of mappings and streamlines the review as well as the validation process. The two tools are available under an open source license.

First Steps to Evaluate an NLP Tool's Medication Extraction Accuracy from Discharge Letters.

INTRODUCTION: The aim of this study is to evaluate the use of a natural language processing (NLP) software to extract medication statements from unstructured medical discharge letters. METHODS: Ten randomly selected discharge letters were extracted from the data warehouse of the University Hospital Erlangen (UHE) and manually annotated to create a gold standard. The AHD NLP tool, provided by MIRACUM's industry partner was used to annotate these discharge letters. Annotations by the NLP tool where then compared to the gold standard on two levels: phrase precision (whether or not the whole medication statement has been identified correctly) and token precision (whether or not the medication name has been identified correctly within correctly discovered medication phrases). RESULTS: The NLP tool detected medication related phrases with an overall F-measure of 0.852. The medication name has been identified correctly with an overall F-measure of 0.936. DISCUSSION: This proof-of-concept study is a first step towards an automated scalable evaluation system for MIRACUM's industry partner's NLP tool by using a gold standard. Medication phrases and names have been correctly identified in most cases by the NLP system. Future effort needs to be put into extending and validating the gold standard.

International Comparisons of Harmonized Laboratory Value Trajectories to Predict Severe COVID-19: Leveraging the 4CE Collaborative Across 342 Hospitals and 6 Countries: A Retrospective Cohort Study.

Objectives: To perform an international comparison of the trajectory of laboratory values among hospitalized patients with COVID-19 who develop severe disease and identify optimal timing of laboratory value collection to predict severity across hospitals and regions. Design: Retrospective cohort study. Setting: The Consortium for Clinical Characterization of COVID-19 by EHR (4CE), an international multi-site data-sharing collaborative of 342 hospitals in the US and in Europe. Participants: Patients hospitalized with COVID-19, admitted before or after PCR-confirmed result for SARS-CoV-2. Primary and secondary outcome measures: Patients were categorized as "ever-severe" or "never-severe" using the validated 4CE severity criteria. Eighteen laboratory tests associated with poor COVID-19-related outcomes were evaluated for predictive accuracy by area under the curve (AUC), compared between the severity categories. Subgroup analysis was performed to validate a subset of laboratory values as predictive of severity against a published algorithm. A subset of laboratory values (CRP, albumin, LDH, neutrophil count, D-dimer, and procalcitonin) was compared between North American and European sites for severity prediction. Results: Of 36,447 patients with COVID-19, 19,953 (43.7%) were categorized as ever-severe. Most patients (78.7%) were 50 years of age or older and male (60.5%). Longitudinal trajectories of CRP, albumin, LDH, neutrophil count, D-dimer, and procalcitonin showed association with disease severity. Significant differences of laboratory values at admission were found between the two groups. With the exception of D-dimer, predictive discrimination of laboratory values did not improve after admission. Sub-group analysis using age, D-dimer, CRP, and lymphocyte count as predictive of severity at admission showed similar discrimination to a published algorithm (AUC=0.88 and 0.91, respectively). Both models deteriorated in predictive accuracy as the disease progressed. On average, no difference in severity prediction was found between North American and European sites. Conclusions: Laboratory test values at admission can be used to predict severity in patients with COVID-19. Prediction models show consistency across international sites highlighting the potential generalizability of these models.

International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium.

We leveraged the largely untapped resource of electronic health record data to address critical clinical and epidemiological questions about Coronavirus Disease 2019 (COVID-19). To do this, we formed an international consortium (4CE) of 96 hospitals across five countries (www.covidclinical.net). Contributors utilized the Informatics for Integrating Biology and the Bedside (i2b2) or Observational Medical Outcomes Partnership (OMOP) platforms to map to a common data model. The group focused on temporal changes in key laboratory test values. Harmonized data were analyzed locally and converted to a shared aggregate form for rapid analysis and visualization of regional differences and global commonalities. Data covered 27,584 COVID-19 cases with 187,802 laboratory tests. Case counts and laboratory trajectories were concordant with existing literature. Laboratory tests at the time of diagnosis showed hospital-level differences equivalent to country-level variation across the consortium partners. Despite the limitations of decentralized data generation, we established a framework to capture the trajectory of COVID-19 disease in patients and their response to interventions.

Moving Towards an EHR Data Quality Framework: The MIRACUM Approach.

INTRODUCTION: Data quality (DQ) is an important prerequisite for secondary use of electronic health record (EHR) data in clinical research, particularly with regards to progressing towards a learning health system, one of the MIRACUM consortium's goals. Following the successful integration of the i2b2 research data repository in MIRACUM, we present a standardized and generic DQ framework. STATE OF THE ART: Already established DQ evaluation methods do not cover all of MIRACUM's requirements. CONCEPT: A data quality analysis plan was developed to assess common data quality dimensions for demographic-, condition-, procedure- and department-related variables of MIRACUM's research data repository. IMPLEMENTATION: A data quality analysis (DQA) tool was developed using R scripts packaged in a Docker image with all the necessary dependencies and R libraries for easy distribution. It integrates with the i2b2 data repository at each MIRACUM site, executes an analysis on the data and generates a DQ report. LESSONS LEARNED: Our DQA tool brings the analysis to the data and thus meets the MIRACUM data protection requirements. It evaluates established DQ dimensions of data repositories in a standardized and easily distributable way. This analysis allowed us to reveal and revise inconsistencies in earlier versions of the ETL jobs. The framework is portable, easy to deploy across different sites and even further adaptable to other database schemes. CONCLUSION: The presented framework provides the first step towards a unified, standardized and harmonized EHR DQ assessment in MIRACUM. DQ issues can now be systematically identified by individual hospitals to subsequently implement site- or consortium-wide feedback loops to increase data quality.

In Search for Methods to Support Electronic Patient Recruitment in a Multi-ICU Clinical Trial.

Antimycotics are substances to treat fungal infections, a frequent cause of death on intensive care units. It is of great importance to administer such drugs only to patients who actually need them, since the unnecessary application leads to the selection of multiresistent fungi, making future therapy more difficult, and represents a significant financial burden for the health care system. Within the scope of a prospective study, which analyses the premature discontinuation of the administration of unnecessary antifungal drugs, two software platforms for recruitment support were implemented and compared at the University Hospital Erlangen. Besides technical aspects, such as the necessary development time and query runtimes, their usability and user friendliness for the clinical users were compared. We found that the practical identification of patient cohorts is possible both with a full featured business intelligence application and with a low effort approach based on language constructs of the Arden Syntax. Furthermore, this pilot evaluation led to important insights related to the clinical documentation context and data quality issues. A comprehensive analysis of the clinical environment and documentation context is essential for the final decision on the tool to be used.

An Analysis of Erlangen University Hospital's Billing Data on Utility-Based De-Identification.

BACKGROUND: To make patient care data more accessible for research, German university hospitals join forces in the course of the Medical Informatics Initiative. In a first step, the administrative data of university hospitals is made available for federated utilization. Project-specific de-identification of this data is necessary to satisfy privacy laws. OBJECTIVE: We want to make a statement about the population uniqueness of the data. By generalizing the data, we try to reduce uniqueness and improve k-anonymity. METHODS: We analyze quasi-identifying attributes of the Erlangen University Hospital's billing data regarding population uniqueness and re-identification risk. We count individuals per equality class (k) to measure uniqueness. RESULTS: Because of the diagnoses and procedures being particularly unique in combination with sex and age of the patients, the data set is not anonymized in matters of k-anonymity with k > 1 . We are able to reduce population uniqueness with generalization and suppression of unique domains. CONCLUSION: To create k-anonymity with k > 1 while still maintaining a particular utility of the data, we need to apply further established strategies of de-identification.