DQAgui: a graphical user interface for the MIRACUM data quality assessment tool.

BACKGROUND: With the growing impact of observational research studies, there is also a growing focus on data quality (DQ). As opposed to experimental study designs, observational research studies are performed using data mostly collected in a non-research context (secondary use). Depending on the number of data elements to be analyzed, DQ reports of data stored within research networks can grow very large. They might be cumbersome to read and important information could be overseen quickly. To address this issue, a DQ assessment (DQA) tool with a graphical user interface (GUI) was developed and provided as a web application. METHODS: The aim was to provide an easy-to-use interface for users without prior programming knowledge to carry out DQ checks and to present the results in a clearly structured way. This interface serves as a starting point for a more detailed investigation of possible DQ irregularities. A user-centered development process ensured the practical feasibility of the interactive GUI. The interface was implemented in the R programming language and aligned to Kahn et al.'s DQ categories conformance, completeness and plausibility. RESULTS: With DQAgui, an R package with a web-app frontend for DQ assessment was developed. The GUI allows users to perform DQ analyses of tabular data sets and to systematically evaluate the results. During the development of the GUI, additional features were implemented, such as analyzing a subset of the data by defining time periods and restricting the analyses to certain data elements. CONCLUSIONS: As part of the MIRACUM project, DQAgui is now being used at ten German university hospitals for DQ assessment and to provide a central overview of the availability of important data elements in a datamap over 2 years. Future development efforts should focus on design optimization and include a usability evaluation.

Effectiveness of IT-supported patient recruitment: study protocol for an interrupted time series study at ten German university hospitals.

BACKGROUND: As part of the German Medical Informatics Initiative, the MIRACUM project establishes data integration centers across ten German university hospitals. The embedded MIRACUM Use Case "Alerting in Care - IT Support for Patient Recruitment", aims to support the recruitment into clinical trials by automatically querying the repositories for patients satisfying eligibility criteria and presenting them as screening candidates. The objective of this study is to investigate whether the developed recruitment tool has a positive effect on study recruitment within a multi-center environment by increasing the number of participants. Its secondary objective is the measurement of organizational burden and user satisfaction of the provided IT solution. METHODS: The study uses an Interrupted Time Series Design with a duration of 15 months. All trials start in the control phase of randomized length with regular recruitment and change to the intervention phase with additional IT support. The intervention consists of the application of a recruitment-support system which uses patient data collected in general care for screening according to specific criteria. The inclusion and exclusion criteria of all selected trials are translated into a machine-readable format using the OHDSI ATLAS tool. All patient data from the data integration centers is regularly checked against these criteria. The primary outcome is the number of participants recruited per trial and week standardized by the targeted number of participants per week and the expected recruitment duration of the specific trial. Secondary outcomes are usability, usefulness, and efficacy of the recruitment support. Sample size calculation based on simple parallel group assumption can demonstrate an effect size of d=0.57 on a significance level of 5% and a power of 80% with a total number of 100 trials (10 per site). Data describing the included trials and the recruitment process is collected at each site. The primary analysis will be conducted using linear mixed models with the actual recruitment number per week and trial standardized by the expected recruitment number per week and trial as the dependent variable. DISCUSSION: The application of an IT-supported recruitment solution developed in the MIRACUM consortium leads to an increased number of recruited participants in studies at German university hospitals. It supports employees engaged in the recruitment of trial participants and is easy to integrate in their daily work.

recruIT: A cloud-native clinical trial recruitment support system based on Health Level 7 Fast Healthcare Interoperability Resources (HL7 FHIR) and the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM).

BACKGROUND: Clinical trials (CTs) are foundational to the advancement of evidence-based medicine and recruiting a sufficient number of participants is one of the crucial steps to their successful conduct. Yet, poor recruitment remains the most frequent reason for premature discontinuation or costly extension of clinical trials. METHODS: We designed and implemented a novel, open-source software system to support the recruitment process in clinical trials by generating automatic recruitment recommendations. The development is guided by modern, cloud-native design principles and based on Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) as an interoperability standard with the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) being used as a source of patient data. We evaluated the usability using the system usability scale (SUS) after deploying the application for use by study personnel. RESULTS: The implementation is based on the OMOP CDM as a repository of patient data that is continuously queried for possible trial candidates based on given clinical trial eligibility criteria. A web-based screening list can be used to display the candidates and email notifications about possible new trial participants can be sent automatically. All interactions between services use HL7 FHIR as the communication standard. The system can be installed using standard container technology and supports more sophisticated deployments on Kubernetes clusters. End-users (n = 19) rated the system with a SUS score of 79.9/100. CONCLUSION: We contribute a novel, open-source implementation to support the patient recruitment process in clinical trials that can be deployed using state-of-the art technologies. According to the SUS score, the system provides good usability.

Evaluation of Eligibility Criteria Relevance for the Purpose of IT-Supported Trial Recruitment: Descriptive Quantitative Analysis.

BACKGROUND: Clinical trials (CTs) are crucial for medical research; however, they frequently fall short of the requisite number of participants who meet all eligibility criteria (EC). A clinical trial recruitment support system (CTRSS) is developed to help identify potential participants by performing a search on a specific data pool. The accuracy of the search results is directly related to the quality of the data used for comparison. Data accessibility can present challenges, making it crucial to identify the necessary data for a CTRSS to query. Prior research has examined the data elements frequently used in CT EC but has not evaluated which criteria are actually used to search for participants. Although all EC must be met to enroll a person in a CT, not all criteria have the same importance when searching for potential participants in an existing data pool, such as an electronic health record, because some of the criteria are only relevant at the time of enrollment. OBJECTIVE: In this study, we investigated which groups of data elements are relevant in practice for finding suitable participants and whether there are typical elements that are not relevant and can therefore be omitted. METHODS: We asked trial experts and CTRSS developers to first categorize the EC of their CTs according to data element groups and then to classify them into 1 of 3 categories: necessary, complementary, and irrelevant. In addition, the experts assessed whether a criterion was documented (on paper or digitally) or whether it was information known only to the treating physicians or patients. RESULTS: We reviewed 82 CTs with 1132 unique EC. Of these 1132 EC, 350 (30.9%) were considered necessary, 224 (19.8%) complementary, and 341 (30.1%) total irrelevant. To identify the most relevant data elements, we introduced the data element relevance index (DERI). This describes the percentage of studies in which the corresponding data element occurs and is also classified as necessary or supplementary. We found that the query of "diagnosis" was relevant for finding participants in 79 (96.3%) of the CTs. This group was followed by "date of birth/age" with a DERI of 85.4% (n=70) and "procedure" with a DERI of 35.4% (n=29). CONCLUSIONS: The distribution of data element groups in CTs has been heterogeneously described in previous works. Therefore, we recommend identifying the percentage of CTs in which data element groups can be found as a more reliable way to determine the relevance of EC. Only necessary and complementary criteria should be included in this DERI.

Availability of Structured Data Elements in Electronic Health Records for Supporting Patient Recruitment in Clinical Trials.

Automated identification of eligible patients for clinical trials is an evident secondary use for electronic health records (EHR) data accumulated during routine care. This task requires relevant data elements to be both available in the EHR and in a structured form. This work analyzes these data quality dimensions of EHR data elements corresponding to a selection of frequent eligibility criteria over a total of 436 patient records at 10 university hospitals within the MIRACUM consortium. Data elements from demographics, diagnosis and laboratory findings are typically structured with a completeness of 73 % to 88 % while medication as well as procedures are rather untructured with a completeness of only 44 %. The results can be used to derive suggestions for data quality improvement measures with respect to patient recruitment as well as to serve as a baseline to quantify future developments.

AHD2FHIR: A Tool for Mapping of Natural Language Annotations to Fast Healthcare Interoperability Resources - A Technical Case Report.

A significant portion of data in Electronic Health Records is only available as unstructured text, such as surgical or finding reports, clinical notes and discharge summaries. To use this data for secondary purposes, natural language processing (NLP) tools are required to extract structured information. Furthermore, for interoperable use, harmonization of the data is necessary. HL7 Fast Healthcare Interoperability Resources (FHIR), an emerging standard for exchanging healthcare data, defines such a structured format. For German-language medical NLP, the tool Averbis Health Discovery (AHD) represents a comprehensive solution. AHD offers a proprietary REST interface for text analysis pipelines. To build a bridge between FHIR and this interface, we created a service that translates the communication around AHD from and to FHIR. The application is available under an open source license.

Patient Recruitment System for Clinical Trials: Mixed Methods Study About Requirements at Ten University Hospitals.

BACKGROUND: Clinical trials are the gold standard for advancing medical knowledge and improving patient outcomes. For their success, an appropriately sized cohort is required. However, patient recruitment remains one of the most challenging aspects of clinical trials. Information technology (IT) support systems-for instance, patient recruitment systems-may help overcome existing challenges and improve recruitment rates, when customized to the user needs and environment. OBJECTIVE: The goal of our study is to describe the status quo of patient recruitment processes and to identify user requirements for the development of a patient recruitment system. METHODS: We conducted a web-based survey with 56 participants as well as semistructured interviews with 33 participants from 10 German university hospitals. RESULTS: We here report the recruitment procedures and challenges of 10 university hospitals. The recruitment process was influenced by diverse factors such as the ward, use of software, and the study inclusion criteria. Overall, clinical staff seemed more involved in patient identification, while the research staff focused on screening tasks. Ad hoc and planned screenings were common. Identifying eligible patients was still associated with significant manual efforts. The recruitment staff used Microsoft Office suite because tailored software were not available. To implement such software, data from disparate sources will need to be made available. We discussed concrete technical challenges concerning patient recruitment systems, including requirements for features, data, infrastructure, and workflow integration, and we contributed to the support of developing a successful system. CONCLUSIONS: Identifying eligible patients is still associated with significant manual efforts. To fully make use of the high potential of IT in patient recruitment, many technical and process challenges have to be solved first. We contribute and discuss concrete technical challenges for patient recruitment systems, including requirements for features, data, infrastructure, and workflow integration.

A Framework for Criteria-Based Selection and Processing of Fast Healthcare Interoperability Resources (FHIR) Data for Statistical Analysis: Design and Implementation Study.

BACKGROUND: The harmonization and standardization of digital medical information for research purposes is a challenging and ongoing collaborative effort. Current research data repositories typically require extensive efforts in harmonizing and transforming original clinical data. The Fast Healthcare Interoperability Resources (FHIR) format was designed primarily to represent clinical processes; therefore, it closely resembles the clinical data model and is more widely available across modern electronic health records. However, no common standardized data format is directly suitable for statistical analyses, and data need to be preprocessed before statistical analysis. OBJECTIVE: This study aimed to elucidate how FHIR data can be queried directly with a preprocessing service and be used for statistical analyses. METHODS: We propose that the binary JavaScript Object Notation format of the PostgreSQL (PSQL) open source database is suitable for not only storing FHIR data, but also extending it with preprocessing and filtering services, which directly transform data stored in FHIR format into prepared data subsets for statistical analysis. We specified an interface for this preprocessor, implemented and deployed it at University Hospital Erlangen-Nürnberg, generated 3 sample data sets, and analyzed the available data. RESULTS: We imported real-world patient data from 2016 to 2018 into a standard PSQL database, generating a dataset of approximately 35.5 million FHIR resources, including "Patient," "Encounter," "Condition" (diagnoses specified using International Classification of Diseases codes), "Procedure," and "Observation" (laboratory test results). We then integrated the developed preprocessing service with the PSQL database and the locally installed web-based KETOS analysis platform. Advanced statistical analyses were feasible using the developed framework using 3 clinically relevant scenarios (data-driven establishment of hemoglobin reference intervals, assessment of anemia prevalence in patients with cancer, and investigation of the adverse effects of drugs). CONCLUSIONS: This study shows how the standard open source database PSQL can be used to store FHIR data and be integrated with a specifically developed preprocessing and analysis framework. This enables dataset generation with advanced medical criteria and the integration of subsequent statistical analysis. The web-based preprocessing service can be deployed locally at the hospital level, protecting patients' privacy while being integrated with existing open source data analysis tools currently being developed across Germany.

Prototypical Clinical Trial Registry Based on Fast Healthcare Interoperability Resources (FHIR): Design and Implementation Study.

BACKGROUND: Clinical trial registries increase transparency in medical research by making information and results of planned, ongoing, and completed studies publicly available. However, the registration of clinical trials remains a time-consuming manual task complicated by the fact that the same studies often need to be registered in different registries with different data entry requirements and interfaces. OBJECTIVE: This study investigates how Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) may be used as a standardized format for exchanging and storing clinical trial records. METHODS: We designed and prototypically implemented an open-source central trial registry containing records from university hospitals, which are automatically exported and updated by local study management systems. RESULTS: We provided an architecture and implementation of a multisite clinical trials registry based on HL7 FHIR as a data storage and exchange format. CONCLUSIONS: The results show that FHIR resources establish a harmonized view of study information from heterogeneous sources by enabling automated data exchange between trial centers and central study registries.

Data Model Requirements for a Digital Cognitive Aid for Anesthesia to Support Intraoperative Crisis Management.

OBJECTIVE: The aim of this study is to define data model requirements supporting the development of a digital cognitive aid (CA) for intraoperative crisis management in anesthesia, including medical emergency text modules (text elements) and branches or loops within emergency instructions (control structures) as well as their properties, data types, and value ranges. METHODS: The analysis process comprised three steps: reviewing the structure of paper-based CAs to identify common text elements and control structures, identifying requirements derived from content, design, and purpose of a digital CA, and validating requirements by loading exemplary emergency checklist data into the resulting prototype data model. RESULTS: The analysis of paper-based CAs identified 19 general text elements and two control structures. Aggregating these elements and analyzing the content, design and purpose of a digital CA revealed 20 relevant data model requirements. These included checklist tags to enable different search options, structured checklist action steps (items) in groups and subgroups, and additional information on each item. Checklist and Item were identified as two main classes of the prototype data model. A data object built according to this model was successfully integrated into a digital CA prototype. CONCLUSION: To enable consistent design and interactivity with the content, presentation of critical medical information in a digital CA for crisis management requires a uniform structure. So far it has not been investigated which requirements need to be met by a data model for this purpose. The results of this study define the requirements and structure that enable the presentation of critical medical information. Further research is needed to develop a comprehensive data model for a digital CA for crisis management in anesthesia, including supplementation of requirements resulting from simulation studies and feasibility analyses regarding existing data models. This model may also be a useful template for developing data models for CAs in other medical domains.

Design for a Modular Clinical Trial Recruitment Support System Based on FHIR and OMOP.

The MIRACUM consortium is developing a Clinical Trials Recruitment Support System to support the data-driven recruitment of patients for clinical trials. The design of the prototype includes both open source solutions (OMOP CDM, Atlas) and open standards for interoperability (FHIR). The aim of the prototype is to create a patient screening list of potential participants for a clinical study. The paper shows the modular structure and functionality of the prototype building the foundation for the practical implementation of the CTRSS and, at the same time, demonstrating the use of open source solutions and standards for the development of clinical support systems.

Reduced Rate of Inpatient Hospital Admissions in 18 German University Hospitals During the COVID-19 Lockdown.

The COVID-19 pandemic has caused strains on health systems worldwide disrupting routine hospital services for all non-COVID patients. Within this retrospective study, we analyzed inpatient hospital admissions across 18 German university hospitals during the 2020 lockdown period compared to 2018. Patients admitted to hospital between January 1 and May 31, 2020 and the corresponding periods in 2018 and 2019 were included in this study. Data derived from electronic health records were collected and analyzed using the data integration center infrastructure implemented in the university hospitals that are part of the four consortia funded by the German Medical Informatics Initiative. Admissions were grouped and counted by ICD 10 chapters and specific reasons for treatment at each site. Pooled aggregated data were centrally analyzed with descriptive statistics to compare absolute and relative differences between time periods of different years. The results illustrate how care process adoptions depended on the COVID-19 epidemiological situation and the criticality of the disease. Overall inpatient hospital admissions decreased by 35% in weeks 1 to 4 and by 30.3% in weeks 5 to 8 after the lockdown announcement compared to 2018. Even hospital admissions for critical care conditions such as malignant cancer treatments were reduced. We also noted a high reduction of emergency admissions such as myocardial infarction (38.7%), whereas the reduction in stroke admissions was smaller (19.6%). In contrast, we observed a considerable reduction in admissions for non-critical clinical situations, such as hysterectomies for benign tumors (78.8%) and hip replacements due to arthrosis (82.4%). In summary, our study shows that the university hospital admission rates in Germany were substantially reduced following the national COVID-19 lockdown. These included critical care or emergency conditions in which deferral is expected to impair clinical outcomes. Future studies are needed to delineate how appropriate medical care of critically ill patients can be maintained during a pandemic.

Extraction and Prevalence of Structured Data Elements in Free-Text Clinical Trial Eligibility Criteria.

Understanding the prevalence of structured data elements within clinical trial eligibility criteria is a crucial step for prioritizing integration efforts to supported automated patient recruitment into clinical trials based on electronic health record data. In this work, we extract data elements from 50 clinical trials using a collaborative, crowd-sourced, and iterative method. A total of 1.120 criteria were analyzed, and 204 unique data elements were extracted. The most prevalent elements were diagnosis code, procedure code, and medication code, occurring in 414 (37 %), 112 (10 %), and 91 (8 %) of eligibility criteria respectively. The results of this study may aid in optimizing data integration and documentation efforts in the EHR to support clinical trial eligibility determination.

Extractive summarization of clinical trial descriptions.

PURPOSE: Text summarization of clinical trial descriptions has the potential to reduce the time required to familiarize oneself with the subject of studies by condensing long-form detailed descriptions to concise, meaning-preserving synopses. This work describes the process and quality of automatically generated summaries of clinical trial descriptions using extractive text summarization methods. METHODS: We generated a novel dataset from the detailed descriptions and brief summaries of trials registered on clinicaltrials.gov. We executed several text summarization algorithms on the detailed descriptions in this corpus and calculated the standard ROUGE metrics using the brief summaries included in the record as a reference. To investigate the correlation of these metrics with human sentiments, four reviewers assessed the content-completeness of the generated summaries and the helpfulness of both the generated and reference summaries via a Likert scale questionnaire. RESULTS: The filtering stages of the dataset generation process reduce the 277,228 trials registered on clinicaltrials.gov to 101,016 records usable for the summarization task. On average, the summaries in this corpus are 25% the length of the detailed descriptions. Of the evaluated text summarization methods, the TextRank algorithm exhibits the overall best performance with a ROUGE-1 F1 score of 0.3531, ROUGE-2 F1 score of 0.1723, and ROUGE-L F1 score of 0.3003. These scores correlate with the assessment of the helpfulness and content similarity by the human reviewers. Inter-rater agreement for the helpfulness and content similarity was slight and fair respectively (Fleiss' kappa of 0.12 and 0.22). CONCLUSIONS: Extractive summarization is a viable tool for generating meaning-preserving synopses of detailed clinical trial descriptions. Further, the human evaluation has shown that the ROUGE-L F1 score is useful for rating the general quality of generated summaries of clinical trial descriptions in an automated way.

Design and Implementation of a Single Source Multipurpose Hospital-Wide Clinical Trial Registry.

IT-supported patient recruitment, serious adverse event reporting as well as making information about clinical trials accessible to the public on websites, are still major challenges in clinical trials. Too often the distribution of information about trials being performed within a hospital across numerous institutions and IT systems is a barrier to provide efficient IT support for such scenarios. Thus, the essential prerequisite to mastering those challenges is to achieve one single point of truth with adequate, complete, accurate and up-to-date information about all clinical trials running at a hospital. We describe the design and implementation of such a single source clinical trial registry serving multiple purposes at a university hospital.

Correction: KETOS: Clinical decision support and machine learning as a service - A training and deployment platform based on Docker, OMOP-CDM, and FHIR Web Services.

[This corrects the article DOI: 10.1371/journal.pone.0223010.].

KETOS: Clinical decision support and machine learning as a service - A training and deployment platform based on Docker, OMOP-CDM, and FHIR Web Services.

BACKGROUND AND OBJECTIVE: To take full advantage of decision support, machine learning, and patient-level prediction models, it is important that models are not only created, but also deployed in a clinical setting. The KETOS platform demonstrated in this work implements a tool for researchers allowing them to perform statistical analyses and deploy resulting models in a secure environment. METHODS: The proposed system uses Docker virtualization to provide researchers with reproducible data analysis and development environments, accessible via Jupyter Notebook, to perform statistical analysis and develop, train and deploy models based on standardized input data. The platform is built in a modular fashion and interfaces with web services using the Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) standard to access patient data. In our prototypical implementation we use an OMOP common data model (OMOP-CDM) database. The architecture supports the entire research lifecycle from creating a data analysis environment, retrieving data, and training to final deployment in a hospital setting. RESULTS: We evaluated the platform by establishing and deploying an analysis and end user application for hemoglobin reference intervals within the University Hospital Erlangen. To demonstrate the potential of the system to deploy arbitrary models, we loaded a colorectal cancer dataset into an OMOP database and built machine learning models to predict patient outcomes and made them available via a web service. We demonstrated both the integration with FHIR as well as an example end user application. Finally, we integrated the platform with the open source DataSHIELD architecture to allow for distributed privacy preserving data analysis and training across networks of hospitals. CONCLUSION: The KETOS platform takes a novel approach to data analysis, training and deploying decision support models in a hospital or healthcare setting. It does so in a secure and privacy-preserving manner, combining the flexibility of Docker virtualization with the advantages of standardized vocabularies, a widely applied database schema (OMOP-CDM), and a standardized way to exchange medical data (FHIR).

Investigating the Capabilities of FHIR Search for Clinical Trial Phenotyping.

Clinical trials are the foundation of evidence-based medicine and their computerized support has been a recurring theme in medical informatics. One challenging aspect is the representation of eligibility criteria in a machine-readable format to automate the identification of suitable participants. In this study, we investigate the capabilities for expressing trial eligibility criteria via the search functionality specified in HL7 FHIR, an emerging standard for exchanging healthcare information electronically which also defines a set of operations for searching for health record data. Using a randomly sampled subset of 303 eligibility criteria from ClinicalTrials.gov yielded a 34 % success rate in representing them using the FHIR search semantics. While limitations are present, the FHIR search semantics are a viable tool for supporting preliminary trial eligibility assessment.