Multi-Method Formative Evaluation of a Digital Online Grocery Shopping Assistant Among Special Supplemental Nutrition Program for Women, Infants, and Children Participants.

OBJECTIVE: Assess the acceptability of a digital grocery shopping assistant among rural women with low income. DESIGN: Simulated shopping experience, semistructured interviews, and a choice experiment. SETTING: Rural central North Carolina Special Supplemental Nutrition Program for Women, Infants, and Children clinic. PARTICIPANTS: Thirty adults (aged ≥18 years) recruited from a Special Supplemental Nutrition Program for Women, Infants, and Children clinic. PHENOMENON OF INTEREST: A simulated grocery shopping experience with the Retail Online Shopping Assistant (ROSA) and mixed-methods feedback on the experience. ANALYSIS: Deductive and inductive qualitative content analysis to independently code and identify themes and patterns among interview responses and quantitative analysis of simulated shopping experience and choice experiment. RESULTS: Most participants liked ROSA (28/30, 93%) and found it helpful and likely to change their purchase across various food categories and at checkout. Retail Online Shopping Assistant's reminders and suggestions could reduce less healthy shopping habits and diversify food options. Participants desired dynamic suggestions and help with various health conditions. Participants preferred a racially inclusive, approachable, cartoon-like, and clinically dressed character. CONCLUSIONS AND IMPLICATIONS: This formative study suggests ROSA could be a beneficial tool for facilitating healthy online grocery shopping among rural shoppers. Future research should investigate the impact of ROSA on dietary behaviors further.

EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension.

OBJECTIVE: To develop a computable representation for medical evidence and to contribute a gold standard dataset of annotated randomized controlled trial (RCT) abstracts, along with a natural language processing (NLP) pipeline for transforming free-text RCT evidence in PubMed into the structured representation. MATERIALS AND METHODS: Our representation, EvidenceMap, consists of 3 levels of abstraction: Medical Evidence Entity, Proposition and Map, to represent the hierarchical structure of medical evidence composition. Randomly selected RCT abstracts were annotated following EvidenceMap based on the consensus of 2 independent annotators to train an NLP pipeline. Via a user study, we measured how the EvidenceMap improved evidence comprehension and analyzed its representative capacity by comparing the evidence annotation with EvidenceMap representation and without following any specific guidelines. RESULTS: Two corpora including 229 disease-agnostic and 80 COVID-19 RCT abstracts were annotated, yielding 12 725 entities and 1602 propositions. EvidenceMap saves users 51.9% of the time compared to reading raw-text abstracts. Most evidence elements identified during the freeform annotation were successfully represented by EvidenceMap, and users gave the enrollment, study design, and study Results sections mean 5-scale Likert ratings of 4.85, 4.70, and 4.20, respectively. The end-to-end evaluations of the pipeline show that the evidence proposition formulation achieves F1 scores of 0.84 and 0.86 in the adjusted random index score. CONCLUSIONS: EvidenceMap extends the participant, intervention, comparator, and outcome framework into 3 levels of abstraction for transforming free-text evidence from the clinical literature into a computable structure. It can be used as an interoperable format for better evidence retrieval and synthesis and an interpretable representation to efficiently comprehend RCT findings.

Deep learning for rare disease: A scoping review.

Although individually rare, collectively more than 7,000 rare diseases affect about 10% of patients. Each of the rare diseases impacts the quality of life for patients and their families, and incurs significant societal costs. The low prevalence of each rare disease causes formidable challenges in accurately diagnosing and caring for these patients and engaging participants in research to advance treatments. Deep learning has advanced many scientific fields and has been applied to many healthcare tasks. This study reviewed the current uses of deep learning to advance rare disease research. Among the 332 reviewed articles, we found that deep learning has been actively used for rare neoplastic diseases (250/332), followed by rare genetic diseases (170/332) and rare neurological diseases (127/332). Convolutional neural networks (307/332) were the most frequently used deep learning architecture, presumably because image data were the most commonly available data type in rare disease research. Diagnosis is the main focus of rare disease research using deep learning (263/332). We summarized the challenges and future research directions for leveraging deep learning to advance rare disease research.

Interactive Similarity-Based Search of Clinical Trials.

The rapid growth of clinical trials launched in recent years poses significant challenges for accurate and efficient trial search. Keyword-based clinical trial search engines require users to construct effective queries, which can be a difficult task given complex information needs. In this study, we present an interactive clinical trial search interface that retrieves trials similar to a target clinical trial. It enables user configuration of 13 clinical trial features and 4 metrics (Jaccard similarity, semantic-based similarity, temporal overlap and geographical distance) to measure pairwise trial similarities. Among 1,007 coronavirus disease 2019 (COVID-19) trials conducted in the United States, 91.9% were found to have similar trials with the similarity threshold being 0.85 and 43.8% were highly similar with the threshold 0.95. A simulation study using 3 groups of similar trials curated by COVID-19 clinical trial reviews demonstrates the precision and recall of the search interface.

Combining human and machine intelligence for clinical trial eligibility querying.

OBJECTIVE: To combine machine efficiency and human intelligence for converting complex clinical trial eligibility criteria text into cohort queries. MATERIALS AND METHODS: Criteria2Query (C2Q) 2.0 was developed to enable real-time user intervention for criteria selection and simplification, parsing error correction, and concept mapping. The accuracy, precision, recall, and F1 score of enhanced modules for negation scope detection, temporal and value normalization were evaluated using a previously curated gold standard, the annotated eligibility criteria of 1010 COVID-19 clinical trials. The usability and usefulness were evaluated by 10 research coordinators in a task-oriented usability evaluation using 5 Alzheimer's disease trials. Data were collected by user interaction logging, a demographic questionnaire, the Health Information Technology Usability Evaluation Scale (Health-ITUES), and a feature-specific questionnaire. RESULTS: The accuracies of negation scope detection, temporal and value normalization were 0.924, 0.916, and 0.966, respectively. C2Q 2.0 achieved a moderate usability score (3.84 out of 5) and a high learnability score (4.54 out of 5). On average, 9.9 modifications were made for a clinical study. Experienced researchers made more modifications than novice researchers. The most frequent modification was deletion (5.35 per study). Furthermore, the evaluators favored cohort queries resulting from modifications (score 4.1 out of 5) and the user engagement features (score 4.3 out of 5). DISCUSSION AND CONCLUSION: Features to engage domain experts and to overcome the limitations in automated machine output are shown to be useful and user-friendly. We concluded that human-computer collaboration is key to improving the adoption and user-friendliness of natural language processing.

The potential role of EHR data in optimizing eligibility criteria definition for cardiovascular outcome trials.

BACKGROUND: Cardiovascular outcome trials (CVOTs) include patients with high risks for cardiovascular events based on specific inclusion criteria. Little is known about the impact of such inclusion criteria on patient accrual and the incidence rate of cardiovascular events. MATERIALS AND METHODS: We evaluated the impact of criteria on the accrual and the number of cardiovascular events in a cohort of 1544 diabetes patients identified from the clinical data warehouse of New York Presbyterian Hospital / Columbia University Irving Medical Center. RESULTS: The highest incidence rate of the composite events (i.e., cardiovascular mortality, stroke, and myocardial infarction) was observed when the inclusion criteria seek patients with underlying cardiovascular diseases or age ≥ 60 with at least two of the risk factors including duration of diabetes, hypertension, dyslipidemia, smoking status, and albuminuria. CONCLUSION: Our study shows that the electronic health records could be utilized to optimize the inclusion criteria while balancing study inclusiveness and number of events.

A Framework for Systematic Assessment of Clinical Trial Population Representativeness Using Electronic Health Records Data.

BACKGROUND: Clinical trials are the gold standard for generating robust medical evidence, but clinical trial results often raise generalizability concerns, which can be attributed to the lack of population representativeness. The electronic health records (EHRs) data are useful for estimating the population representativeness of clinical trial study population. OBJECTIVES: This research aims to estimate the population representativeness of clinical trials systematically using EHR data during the early design stage. METHODS: We present an end-to-end analytical framework for transforming free-text clinical trial eligibility criteria into executable database queries conformant with the Observational Medical Outcomes Partnership Common Data Model and for systematically quantifying the population representativeness for each clinical trial. RESULTS: We calculated the population representativeness of 782 novel coronavirus disease 2019 (COVID-19) trials and 3,827 type 2 diabetes mellitus (T2DM) trials in the United States respectively using this framework. With the use of overly restrictive eligibility criteria, 85.7% of the COVID-19 trials and 30.1% of T2DM trials had poor population representativeness. CONCLUSION: This research demonstrates the potential of using the EHR data to assess the clinical trials population representativeness, providing data-driven metrics to inform the selection and optimization of eligibility criteria.

A Comparison between Human and NLP-based Annotation of Clinical Trial Eligibility Criteria Text Using The OMOP Common Data Model.

Human annotations are the established gold standard for evaluating natural language processing (NLP) methods. The goals of this study are to quantify and qualify the disagreement between human and NLP. We developed an NLP system for annotating clinical trial eligibility criteria text and constructed a manually annotated corpus, both following the OMOP Common Data Model (CDM). We analyzed the discrepancies between the human and NLP annotations and their causes (e.g., ambiguities in concept categorization and tacit decisions on inclusion of qualifiers and temporal attributes during concept annotation). This study initially reported complexities in clinical trial eligibility criteria text that complicate NLP and the limitations of the OMOP CDM. The disagreement between and human and NLP annotations may be generalizable. We discuss implications for NLP evaluation.

Participatory Design of a Clinical Trial Eligibility Criteria Simplification Method.

Clinical trial eligibility criteria are important for selecting the right participants for clinical trials. However, they are often complex and not computable. This paper presents the participatory design of a human-computer collaboration method for criteria simplification that includes natural language processing followed by user-centered eligibility criteria simplification. A case study on the ARCADIA trial shows how criteria were simplified for structured database querying by clinical researchers and identifies rules for criteria simplification and concept normalization.

Building an OMOP common data model-compliant annotated corpus for COVID-19 clinical trials.

Clinical trials are essential for generating reliable medical evidence, but often suffer from expensive and delayed patient recruitment because the unstructured eligibility criteria description prevents automatic query generation for eligibility screening. In response to the COVID-19 pandemic, many trials have been created but their information is not computable. We included 700 COVID-19 trials available at the point of study and developed a semi-automatic approach to generate an annotated corpus for COVID-19 clinical trial eligibility criteria called COVIC. A hierarchical annotation schema based on the OMOP Common Data Model was developed to accommodate four levels of annotation granularity: i.e., study cohort, eligibility criteria, named entity and standard concept. In COVIC, 39 trials with more than one study cohorts were identified and labelled with an identifier for each cohort. 1,943 criteria for non-clinical characteristics such as "informed consent", "exclusivity of participation" were annotated. 9767 criteria were represented by 18,161 entities in 8 domains, 7,743 attributes of 7 attribute types and 16,443 relationships of 11 relationship types. 17,171 entities were mapped to standard medical concepts and 1,009 attributes were normalized into computable representations. COVIC can serve as a corpus indexed by semantic tags for COVID-19 trial search and analytics, and a benchmark for machine learning based criteria extraction.

A knowledge base of clinical trial eligibility criteria.

OBJECTIVE: We present the Clinical Trial Knowledge Base, a regularly updated knowledge base of discrete clinical trial eligibility criteria equipped with a web-based user interface for querying and aggregate analysis of common eligibility criteria. MATERIALS AND METHODS: We used a natural language processing (NLP) tool named Criteria2Query (Yuan et al., 2019) to transform free text clinical trial eligibility criteria from ClinicalTrials.gov into discrete criteria concepts and attributes encoded using the widely adopted Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) and stored in a relational SQL database. A web application accessible via RESTful APIs was implemented to enable queries and visual aggregate analyses. We demonstrate CTKB's potential role in EHR phenotype knowledge engineering using ten validated phenotyping algorithms. RESULTS: At the time of writing, CTKB contained 87,504 distinctive OMOP CDM standard concepts, including Condition (47.82%), Drug (23.01%), Procedure (13.73%), Measurement (24.70%) and Observation (5.28%), with 34.78% for inclusion criteria and 65.22% for exclusion criteria, extracted from 352,110 clinical trials. The average hit rate of criteria concepts in eMERGE phenotype algorithms is 77.56%. CONCLUSION: CTKB is a novel comprehensive knowledge base of discrete eligibility criteria concepts with the potential to enable knowledge engineering for clinical trial cohort definition, clinical trial population representativeness assessment, electronical phenotyping, and data gap analyses for using electronic health records to support clinical trial recruitment.

The COVID-19 Trial Finder.

Clinical trials are the gold standard for generating reliable medical evidence. The biggest bottleneck in clinical trials is recruitment. To facilitate recruitment, tools for patient search of relevant clinical trials have been developed, but users often suffer from information overload. With nearly 700 coronavirus disease 2019 (COVID-19) trials conducted in the United States as of August 2020, it is imperative to enable rapid recruitment to these studies. The COVID-19 Trial Finder was designed to facilitate patient-centered search of COVID-19 trials, first by location and radius distance from trial sites, and then by brief, dynamically generated medical questions to allow users to prescreen their eligibility for nearby COVID-19 trials with minimum human computer interaction. A simulation study using 20 publicly available patient case reports demonstrates its precision and effectiveness.

Chia, a large annotated corpus of clinical trial eligibility criteria.

We present Chia, a novel, large annotated corpus of patient eligibility criteria extracted from 1,000 interventional, Phase IV clinical trials registered in ClinicalTrials.gov. This dataset includes 12,409 annotated eligibility criteria, represented by 41,487 distinctive entities of 15 entity types and 25,017 relationships of 12 relationship types. Each criterion is represented as a directed acyclic graph, which can be easily transformed into Boolean logic to form a database query. Chia can serve as a shared benchmark to develop and test future machine learning, rule-based, or hybrid methods for information extraction from free-text clinical trial eligibility criteria.

INSMA: An integrated system for multimodal data acquisition and analysis in the intensive care unit.

Modern intensive care units (ICU) are equipped with a variety of different medical devices to monitor the physiological status of patients. These devices can generate large amounts of multimodal data daily that include physiological waveform signals (arterial blood pressure, electrocardiogram, respiration), patient alarm messages, numeric vitals data, etc. In order to provide opportunities for increasingly improved patient care, it is necessary to develop an effective data acquisition and analysis system that can assist clinicians and provide decision support at the patient bedside. Previous research has discussed various data collection methods, but a comprehensive solution for bedside data acquisition to analysis has not been achieved. In this paper, we proposed a multimodal data acquisition and analysis system called INSMA, with the ability to acquire, store, process, and visualize multiple types of data from the Philips IntelliVue patient monitor. We also discuss how the acquired data can be used for patient state tracking. INSMA is being tested in the ICU at University Hospitals Cleveland Medical Center.