Deep learning for rare disease: A scoping review.

Although individually rare, collectively more than 7,000 rare diseases affect about 10% of patients. Each of the rare diseases impacts the quality of life for patients and their families, and incurs significant societal costs. The low prevalence of each rare disease causes formidable challenges in accurately diagnosing and caring for these patients and engaging participants in research to advance treatments. Deep learning has advanced many scientific fields and has been applied to many healthcare tasks. This study reviewed the current uses of deep learning to advance rare disease research. Among the 332 reviewed articles, we found that deep learning has been actively used for rare neoplastic diseases (250/332), followed by rare genetic diseases (170/332) and rare neurological diseases (127/332). Convolutional neural networks (307/332) were the most frequently used deep learning architecture, presumably because image data were the most commonly available data type in rare disease research. Diagnosis is the main focus of rare disease research using deep learning (263/332). We summarized the challenges and future research directions for leveraging deep learning to advance rare disease research.

Leveraging electronic health record data for clinical trial planning by assessing eligibility criteria's impact on patient count and safety.

OBJECTIVE: To present an approach on using electronic health record (EHR) data that assesses how different eligibility criteria, either individually or in combination, can impact patient count and safety (exemplified by all-cause hospitalization risk) and further assist with criteria selection for prospective clinical trials. MATERIALS AND METHODS: Trials in three disease domains - relapsed/refractory (r/r) lymphoma/leukemia; hepatitis C virus (HCV); stages 3 and 4 chronic kidney disease (CKD) - were analyzed as case studies for this approach. For each disease domain, criteria were identified and all criteria combinations were used to create EHR cohorts. Per combination, two values were derived: (1) number of eligible patients meeting the selected criteria; (2) hospitalization risk, measured as the hazard ratio between those that qualified and those that did not. From these values, k-means clustering was applied to derive which criteria combinations maximized patient counts but minimized hospitalization risk. RESULTS: Criteria combinations that reduced hospitalization risk without substantial reductions on patient counts were as follows: for r/r lymphoma/leukemia (23 trials; 9 criteria; 623 patients), applying no infection and adequate absolute neutrophil count while forgoing no prior malignancy; for HCV (15; 7; 751), applying no human immunodeficiency virus and no hepatocellular carcinoma while forgoing no decompensated liver disease/cirrhosis; for CKD (10; 9; 23893), applying no congestive heart failure. CONCLUSIONS: Within each disease domain, the more drastic effects were generally driven by a few criteria. Similar criteria across different disease domains introduce different changes. Although results are contingent on the trial sample and the EHR data used, this approach demonstrates how EHR data can inform the impact on safety and available patients when exploring different criteria combinations for designing clinical trials.

Clinical comparison between trial participants and potentially eligible patients using electronic health record data: A generalizability assessment method.

OBJECTIVE: To present a generalizability assessment method that compares baseline clinical characteristics of trial participants (TP) to potentially eligible (PE) patients as presented in their electronic health record (EHR) data while controlling for clinical setting and recruitment period. METHODS: For each clinical trial, a clinical event was defined to identify patients of interest using available EHR data from one clinical setting during the trial's recruitment timeframe. The trial's eligibility criteria were then applied and patients were separated into two mutually exclusive groups: (1) TP, which were patients that participated in the trial per trial enrollment data; (2) PE, the remaining patients. The primary outcome was standardized differences in clinical characteristics between TP and PE per trial. A standardized difference was considered prominent if its absolute value was greater than or equal to 0.1. The secondary outcome was the difference in mean propensity scores (PS) between TP and PE per trial, in which the PS represented prediction for a patient to be in the trial. Three diverse trials were selected for illustration: one focused on hepatitis C virus (HCV) patients receiving a liver transplantation; one focused on leukemia patients and lymphoma patients; and one focused on appendicitis patients. RESULTS: For the HCV trial, 43 TP and 83 PE were found, with 61 characteristics evaluated. Prominent differences were found among 69% of characteristics, with a mean PS difference of 0.13. For the leukemia/lymphoma trial, 23 TP and 23 PE were found, with 39 characteristics evaluated. Prominent differences were found among 82% of characteristics, with a mean PS difference of 0.76. For the appendicitis trial, 123 TP and 242 PE were found, with 52 characteristics evaluated. Prominent differences were found among 52% of characteristics, with a mean PS difference of 0.15. CONCLUSIONS: Differences in clinical characteristics were observed between TP and PE among all three trials. In two of the three trials, not all of the differences necessarily compromised trial generalizability and subsets of PE could be considered similar to their corresponding TP. In the remaining trial, lack of generalizability appeared present, but may be a result of other factors such as small sample size or site recruitment strategy. These inconsistent findings suggest eligibility criteria alone are sometimes insufficient in defining a target group to generalize to. With caveats in limited scalability, EHR data quality, and lack of patient perspective on trial participation, this generalizability assessment method that incorporates control for temporality and clinical setting promise to better pinpoint clinical patterns and trial considerations.

Comparative effectiveness of generic and brand-name medication use: A database study of US health insurance claims.

BACKGROUND: To the extent that outcomes are mediated through negative perceptions of generics (the nocebo effect), observational studies comparing brand-name and generic drugs are susceptible to bias favoring the brand-name drugs. We used authorized generic (AG) products, which are identical in composition and appearance to brand-name products but are marketed as generics, as a control group to address this bias in an evaluation aiming to compare the effectiveness of generic versus brand medications. METHODS AND FINDINGS: For commercial health insurance enrollees from the US, administrative claims data were derived from 2 databases: (1) Optum Clinformatics Data Mart (years: 2004-2013) and (2) Truven MarketScan (years: 2003-2015). For a total of 8 drug products, the following groups were compared using a cohort study design: (1) patients switching from brand-name products to AGs versus generics, and patients initiating treatment with AGs versus generics, where AG use proxied brand-name use, addressing negative perception bias, and (2) patients initiating generic versus brand-name products (bias-prone direct comparison) and patients initiating AG versus brand-name products (negative control). Using Cox proportional hazards regression after 1:1 propensity-score matching, we compared a composite cardiovascular endpoint (for amlodipine, amlodipine-benazepril, and quinapril), non-vertebral fracture (for alendronate and calcitonin), psychiatric hospitalization rate (for sertraline and escitalopram), and insulin initiation (for glipizide) between the groups. Inverse variance meta-analytic methods were used to pool adjusted hazard ratios (HRs) for each comparison between the 2 databases. Across 8 products, 2,264,774 matched pairs of patients were included in the comparisons of AGs versus generics. A majority (12 out of 16) of the clinical endpoint estimates showed similar outcomes between AGs and generics. Among the other 4 estimates that did have significantly different outcomes, 3 suggested improved outcomes with generics and 1 favored AGs (patients switching from amlodipine brand-name: HR [95% CI] 0.92 [0.88-0.97]). The comparison between generic and brand-name initiators involved 1,313,161 matched pairs, and no differences in outcomes were noted for alendronate, calcitonin, glipizide, or quinapril. We observed a lower risk of the composite cardiovascular endpoint with generics versus brand-name products for amlodipine and amlodipine-benazepril (HR [95% CI]: 0.91 [0.84-0.99] and 0.84 [0.76-0.94], respectively). For escitalopram and sertraline, we observed higher rates of psychiatric hospitalizations with generics (HR [95% CI]: 1.05 [1.01-1.10] and 1.07 [1.01-1.14], respectively). The negative control comparisons also indicated potentially higher rates of similar magnitude with AG compared to brand-name initiation for escitalopram and sertraline (HR [95% CI]: 1.06 [0.98-1.13] and 1.11 [1.05-1.18], respectively), suggesting that the differences observed between brand and generic users in these outcomes are likely explained by either residual confounding or generic perception bias. Limitations of this study include potential residual confounding due to the unavailability of certain clinical parameters in administrative claims data and the inability to evaluate surrogate outcomes, such as immediate changes in blood pressure, upon switching from brand products to generics. CONCLUSIONS: In this study, we observed that use of generics was associated with comparable clinical outcomes to use of brand-name products. These results could help in promoting educational interventions aimed at increasing patient and provider confidence in the ability of generic medicines to manage chronic diseases.

Ensembles of natural language processing systems for portable phenotyping solutions.

BACKGROUND: Manually curating standardized phenotypic concepts such as Human Phenotype Ontology (HPO) terms from narrative text in electronic health records (EHRs) is time consuming and error prone. Natural language processing (NLP) techniques can facilitate automated phenotype extraction and thus improve the efficiency of curating clinical phenotypes from clinical texts. While individual NLP systems can perform well for a single cohort, an ensemble-based method might shed light on increasing the portability of NLP pipelines across different cohorts. METHODS: We compared four NLP systems, MetaMapLite, MedLEE, ClinPhen and cTAKES, and four ensemble techniques, including intersection, union, majority-voting and machine learning, for extracting generic phenotypic concepts. We addressed two important research questions regarding automated phenotype recognition. First, we evaluated the performance of different approaches in identifying generic phenotypic concepts. Second, we compared the performance of different methods to identify patient-specific phenotypic concepts. To better quantify the effects caused by concept granularity differences on performance, we developed a novel evaluation metric that considered concept hierarchies and frequencies. Each of the approaches was evaluated on a gold standard set of clinical documents annotated by clinical experts. One dataset containing 1,609 concepts derived from 50 clinical notes from two different institutions was used in both evaluations, and an additional dataset of 608 concepts derived from 50 case report abstracts obtained from PubMed was used for evaluation of identifying generic phenotypic concepts only. RESULTS: For generic phenotypic concept recognition, the top three performers in the NYP/CUIMC dataset are union ensemble (F1, 0.634), training-based ensemble (F1, 0.632), and majority vote-based ensemble (F1, 0.622). In the Mayo dataset, the top three are majority vote-based ensemble (F1, 0.642), cTAKES (F1, 0.615), and MedLEE (F1, 0.559). In the PubMed dataset, the top three are majority vote-based ensemble (F1, 0.719), training-based (F1, 0.696) and MetaMapLite (F1, 0.694). For identifying patient specific phenotypes, the top three performers in the NYP/CUIMC dataset are majority vote-based ensemble (F1, 0.610), MedLEE (F1, 0.609), and training-based ensemble (F1, 0.585). In the Mayo dataset, the top three are majority vote-based ensemble (F1, 0.604), cTAKES (F1, 0.531) and MedLEE (F1, 0.527). CONCLUSIONS: Our study demonstrates that ensembles of natural language processing can improve both generic phenotypic concept recognition and patient specific phenotypic concept identification over individual systems. Among the individual NLP systems, each individual system performed best when they were applied in the dataset that they were primary designed for. However, combining multiple NLP systems to create an ensemble can generally improve the performance. Specifically, the ensemble can increase the results reproducibility across different cohorts and tasks, and thus provide a more portable phenotyping solution compared to individual NLP systems.

The relative benefits of claims and electronic health record data for predicting medication adherence trajectory.

BACKGROUND: Healthcare providers are increasingly encouraged to improve their patients' adherence to chronic disease medications. Prediction of adherence can identify patients in need of intervention, but most prediction efforts have focused on claims data, which may be unavailable to providers. Electronic health records (EHR) are readily available and may provide richer information with which to predict adherence than is currently available through claims. METHODS: In a linked database of complete Medicare Advantage claims and comprehensive EHR from a multi-specialty outpatient practice, we identified patients who filled a prescription for a statin, antihypertensive, or oral antidiabetic during 2011 to 2012. We followed patients to identify subsequent medication filling patterns and used group-based trajectory models to assign patients to adherence trajectories. We then identified potential predictors from both claims and EHR data and fit a series of models to evaluate the accuracy of each data source in predicting medication adherence. RESULTS: Claims were highly predictive of patients in the worst adherence trajectory (C=0.78), but EHR data also provided good predictions (C=0.72). Among claims predictors, presence of a prior gap in filling of at least 6 days was by far the most influential predictor. In contrast, good predictions from EHR data required complex models with many variables. CONCLUSION: EHR data can provide good predictions of adherence trajectory and therefore may be useful for providers seeking to deploy resource-intensive interventions. However, prior adherence information derived from claims is most predictive, and can supplement EHR data when it is available.

Data Mining for Adverse Drug Events With a Propensity Score-matched Tree-based Scan Statistic.

The tree-based scan statistic is a statistical data mining tool that has been used for signal detection with a self-controlled design in vaccine safety studies. This disproportionality statistic adjusts for multiple testing in evaluation of thousands of potential adverse events. However, many drug safety questions are not well suited for self-controlled analysis. We propose a method that combines tree-based scan statistics with propensity score-matched analysis of new initiator cohorts, a robust design for investigations of drug safety. We conducted plasmode simulations to evaluate performance. In multiple realistic scenarios, tree-based scan statistics in cohorts that were propensity score matched to adjust for confounding outperformed tree-based scan statistics in unmatched cohorts. In scenarios where confounding moved point estimates away from the null, adjusted analyses recovered the prespecified type 1 error while unadjusted analyses inflated type 1 error. In scenarios where confounding moved point estimates toward the null, adjusted analyses preserved power, whereas unadjusted analyses greatly reduced power. Although complete adjustment of true confounders had the best performance, matching on a moderately mis-specified propensity score substantially improved type 1 error and power compared with no adjustment. When there was true elevation in risk of an adverse event, there were often co-occurring signals for clinically related concepts. TreeScan with propensity score matching shows promise as a method for screening and prioritization of potential adverse events. It should be followed by clinical review and safety studies specifically designed to quantify the magnitude of effect, with confounding control targeted to the outcome of interest.

Adaptation and Validation of the Combined Comorbidity Score for ICD-10-CM.

BACKGROUND: The combined comorbidity score, which merges the Charlson and Elixhauser comorbidity indices, uses the ninth revision of the International Classification of Diseases, Clinical Modification (ICD-9-CM). In October 2015, the United States adopted the 10th revision (ICD-10-CM). OBJECTIVE: The objective of this study is to examine different coding algorithms for the ICD-10-CM combined comorbidity score and compare their performance to the original ICD-9-CM score. METHODS: Four ICD-10-CM coding algorithms were defined: 2 using General Equivalence Mappings (GEMs), one based on ICD-10-CA (Canadian modification) codes for Charlson and Elixhauser measures, and one including codes from all 3 algorithms. We used claims data from the Clinfomatics Data Mart to identify 2 cohorts. The ICD-10-CM cohort comprised patients who had a hospitalization between January 1, 2016 and March 1, 2016. The ICD-9-CM cohort comprised patients who had a hospitalization between January 1, 2015 and March 1, 2015. We used logistic regression models to predict 30-day hospital readmission for the original score in the ICD-9-CM cohort and for each ICD-10-CM algorithm in the ICD-10-CM cohort. RESULTS: Distributions of each version of the score were similar. The algorithm based on ICD-10-CA codes [c-statistic, 0.646; 95% confidence interval (CI), 0.640-0.653] had the most similar discrimination for readmission to the ICD-9-CM version (c, 0.646; 95% CI, 0.639-0.653), but combining all identified ICD-10-CM codes had the highest c-statistic (c, 0.651; 95% CI, 0.644-0.657). CONCLUSIONS: We propose an ICD-10-CM version of the combined comorbidity score that includes codes identified by ICD-10-CA and GEMs. Compared with the original score, it has similar performance in predicting readmission in a population of United States commercially insured individuals.

Survey of Living Organ Donors' Experience and Directions for Process Improvement.

INTRODUCTION: Understanding living organ donors' experience with donation and challenges faced during the process is necessary to guide the development of effective strategies to maximize donor benefit and increase the number of living donors. METHODS: An anonymous self-administered survey, specifically designed for this population based on key informant interviews, was mailed to 426 individuals who donated a kidney or liver at our institution. Quantitative and qualitative methods including open and axial coding were used to analyze donor responses. FINDINGS: Of the 141 survey respondents, 94% would encourage others to become donors; however, nearly half (44%) thought the donation process could be improved and offered numerous suggestions. Five major themes arose: (1) desire for greater convenience in testing and scheduling; (2) involvement of previous donors throughout the process; (3) education and promotion of donation through social media; (4) unanticipated difficulties, specifically pain; and (5) financial concerns. DISCUSSION: Donor feedback has been translated into performance improvements at our hospital, many of which are applicable to other institutions. Population-specific survey development helps to identify vital patient concerns and provides valuable feedback to enhance the delivery of care.

Comparison of Benefit-Risk Assessment Methods for Prospective Monitoring of Newly Marketed Drugs: A Simulation Study.

OBJECTIVES: To compare benefit-risk assessment (BRA) methods for determining whether and when sufficient evidence exists to indicate that one drug is favorable over another in prospective monitoring. METHODS: We simulated prospective monitoring of a new drug (A) versus an alternative drug (B) with respect to two beneficial and three harmful outcomes. We generated data for 1000 iterations of six scenarios and applied four BRA metrics: number needed to treat and number needed to harm (NNT|NNH), incremental net benefit (INB) with maximum acceptable risk, INB with relative-value-adjusted life-years, and INB with quality-adjusted life-years. We determined the proportion of iterations in which the 99% confidence interval for each metric included and excluded the null and we calculated mean time to alerting. RESULTS: With no true difference in any outcome between drugs A and B, the proportion of iterations including the null was lowest for INB with relative-value-adjusted life-years (64%) and highest for INB with quality-adjusted life-years (76%). When drug A was more effective and the drugs were equally safe, all metrics indicated net favorability of A in more than 70% of the iterations. When drug A was safer than drug B, NNT|NNH had the highest proportion of iterations indicating net favorability of drug A (65%). Mean time to alerting was similar among methods across the six scenarios. CONCLUSIONS: BRA metrics can be useful for identifying net favorability when applied to prospective monitoring of a new drug versus an alternative drug. INB-based approaches similarly outperform unweighted NNT|NNH approaches. Time to alerting was similar across approaches.

Prospective Benefit-Risk Monitoring of New Drugs for Rapid Assessment of Net Favorability in Electronic Health Care Data.

BACKGROUND: Benefit-risk assessment (BRA) methods can combine measures of benefits and risks into a single value. OBJECTIVES: To examine BRA metrics for prospective monitoring of new drugs in electronic health care data. METHODS: Using two electronic health care databases, we emulated prospective monitoring of three drugs (rofecoxib vs. nonselective nonsteroidal anti-inflammatory drugs, prasugrel vs. clopidogrel, and denosumab vs. bisphosphonates) using a sequential propensity score-matched cohort design. We applied four BRA metrics: number needed to treat and number needed to harm; incremental net benefit (INB) with maximum acceptable risk; INB with relative-value-adjusted life-years; and INB with quality-adjusted life-years (QALYs). We determined whether and when the bootstrapped 99% confidence interval (CI) for each metric excluded zero, indicating net favorability of one drug over the other. RESULTS: For rofecoxib, all four metrics yielded a negative value, suggesting net favorability of nonselective nonsteroidal anti-inflammatory drugs over rofecoxib, and the 99% CI for all but the number needed to treat and number needed to harm excluded the null during follow-up. For prasugrel, only the 99% CI for INB-QALY excluded the null, but trends in values over time were similar across the four metrics, suggesting overall net favorability of prasugrel versus clopidogrel. The 99% CI for INB-relative-value-adjusted life-years and INB-QALY excluded the null in the denosumab example, suggesting net favorability of denosumab over bisphosphonates. CONCLUSIONS: Prospective benefit-risk monitoring can be used to determine net favorability of a new drug in electronic health care data. In three examples, existing BRA metrics produced qualitatively similar results but differed with respect to alert generation.

Risk Factors Associated With SARS-CoV-2 Breakthrough Infections in Fully mRNA-Vaccinated Individuals: Retrospective Analysis.

BACKGROUND: COVID-19 messenger RNA (mRNA) vaccines have demonstrated efficacy and effectiveness in preventing symptomatic COVID-19, while being relatively safe in trial studies. However, vaccine breakthrough infections have been reported. OBJECTIVE: This study aims to identify risk factors associated with COVID-19 breakthrough infections among fully mRNA-vaccinated individuals. METHODS: We conducted a series of observational retrospective analyses using the electronic health records (EHRs) of the Columbia University Irving Medical Center/New York Presbyterian (CUIMC/NYP) up to September 21, 2021. New York City (NYC) adult residences with at least 1 polymerase chain reaction (PCR) record were included in this analysis. Poisson regression was performed to assess the association between the breakthrough infection rate in vaccinated individuals and multiple risk factors-including vaccine brand, demographics, and underlying conditions-while adjusting for calendar month, prior number of visits, and observational days in the EHR. RESULTS: The overall estimated breakthrough infection rate was 0.16 (95% CI 0.14-0.18). Individuals who were vaccinated with Pfizer/BNT162b2 (incidence rate ratio [IRR] against Moderna/mRNA-1273=1.66, 95% CI 1.17-2.35) were male (IRR against female=1.47, 95% CI 1.11-1.94) and had compromised immune systems (IRR=1.48, 95% CI 1.09-2.00) were at the highest risk for breakthrough infections. Among all underlying conditions, those with primary immunodeficiency, a history of organ transplant, an active tumor, use of immunosuppressant medications, or Alzheimer disease were at the highest risk. CONCLUSIONS: Although we found both mRNA vaccines were effective, Moderna/mRNA-1273 had a lower incidence rate of breakthrough infections. Immunocompromised and male individuals were among the highest risk groups experiencing breakthrough infections. Given the rapidly changing nature of the SARS-CoV-2 pandemic, continued monitoring and a generalizable analysis pipeline are warranted to inform quick updates on vaccine effectiveness in real time.

Comparison of Clinical Characteristics Between Clinical Trial Participants and Nonparticipants Using Electronic Health Record Data.

Importance: Assessing generalizability of clinical trials is important to ensure appropriate application of interventions, but most assessments provide minimal granularity on comparisons of clinical characteristics. Objective: To assess the extent of underlying clinical differences between clinical trial participants and nonparticipants by using a combination of electronic health record and trial enrollment data. Design, Setting, and Participants: This cross-sectional study used data obtained from a single academic medical center between September 1996 and January 2019 to identify 1645 clinical trial participants from a diverse set of 202 available trials conducted at the center. Using an aggregated resampling procedure, nonparticipants were matched to participants 1:1 based on trial conditions, number of recent visits to a health care professional, and calendar time. Exposures: Clinical trial enrollment vs no enrollment. Main Outcomes and Measures: The primary outcome was standardized differences in clinical characteristics between participants and nonparticipants in clinical trials stratified into the 4 most common disease domains. Results: This cross-sectional study included 1645 participants from 202 trials (929 [56.5%] male; mean [SD] age, 54.65 [21.38] years) and an aggregated set of 1645 nonparticipants (855 [52.0%] male; mean [SD] age, 57.24 [21.91] years). The most common disease domains for the selected trials were neoplastic disease (86 trials; 737 participants), disorders of the digestive system (31 trials; 321 participants), inflammatory disorders (28 trials; 276 participants), and disorders of the cardiovascular system (27 trials; 319 participants); trials could qualify for multiple disease domains. Among 31 conditions, the percentage of conditions for which the prevalence was lower among participants than among nonparticipants per standardized differences was 64.5% (20 conditions) for neoplastic disease trials, 61.3% (19) for digestive system trials, 58.1% (18) for inflammatory disorder trials, and 38.7% (12) for cardiovascular system trials. Among 17 medications, the percentage of medications for which use was less among participants than among nonparticipants per standardized differences was 64.7% (11) for neoplastic disease trials, 58.8% (10) for digestive system trials, 88.2% (15) for inflammatory disorder trials, and 52.9% (9) for cardiovascular system trials. Conclusions and Relevance: Using a combination of electronic health record and trial enrollment data, this study found that clinical trial participants had fewer comorbidities and less use of medication than nonparticipants across a variety of disease domains. Combining trial enrollment data with electronic health record data may be useful for better understanding of the generalizability of trial results.

Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review.

OBJECTIVE: Real-world data (RWD), defined as routinely collected healthcare data, can be a potential catalyst for addressing challenges faced in clinical trials. We performed a scoping review of database-specific RWD applications within clinical trial contexts, synthesizing prominent uses and themes. MATERIALS AND METHODS: Querying 3 biomedical literature databases, research articles using electronic health records, administrative claims databases, or clinical registries either within a clinical trial or in tandem with methodology related to clinical trials were included. Articles were required to use at least 1 US RWD source. All abstract screening, full-text screening, and data extraction was performed by 1 reviewer. Two reviewers independently verified all decisions. RESULTS: Of 2020 screened articles, 89 qualified: 59 articles used electronic health records, 29 used administrative claims, and 26 used registries. Our synthesis was driven by the general life cycle of a clinical trial, culminating into 3 major themes: trial process tasks (51 articles); dissemination strategies (6); and generalizability assessments (34). Despite a diverse set of diseases studied, <10% of trials using RWD for trial process tasks evaluated medications or procedures (5/51). All articles highlighted data-related challenges, such as missing values. DISCUSSION: Database-specific RWD have been occasionally leveraged for various clinical trial tasks. We observed underuse of RWD within conducted medication or procedure trials, though it is subject to the confounder of implicit report of RWD use. CONCLUSION: Enhanced incorporation of RWD should be further explored for medication or procedure trials, including better understanding of how to handle related data quality issues to facilitate RWD use.

Published models that predict hospital readmission: a critical appraisal.

INTRODUCTION: The number of readmission risk prediction models available has increased rapidly, and these models are used extensively for health decision-making. Unfortunately, readmission models can be subject to flaws in their development and validation, as well as limitations in their clinical usefulness. OBJECTIVE: To critically appraise readmission models in the published literature using Delphi-based recommendations for their development and validation. METHODS: We used the modified Delphi process to create Critical Appraisal of Models that Predict Readmission (CAMPR), which lists expert recommendations focused on development and validation of readmission models. Guided by CAMPR, two researchers independently appraised published readmission models in two recent systematic reviews and concurrently extracted data to generate reference lists of eligibility criteria and risk factors. RESULTS: We found that published models (n=81) followed 6.8 recommendations (45%) on average. Many models had weaknesses in their development, including failure to internally validate (12%), failure to account for readmission at other institutions (93%), failure to account for missing data (68%), failure to discuss data preprocessing (67%) and failure to state the model's eligibility criteria (33%). CONCLUSIONS: The high prevalence of weaknesses in model development identified in the published literature is concerning, as these weaknesses are known to compromise predictive validity. CAMPR may support researchers, clinicians and administrators to identify and prevent future weaknesses in model development.

A Retrospective Analysis of COVID-19 mRNA Vaccine Breakthrough Infections - Risk Factors and Vaccine Effectiveness.

IMPORTANCE: Little is known about COVID vaccine breakthrough infections and their risk factors. OBJECTIVE: To identify risk factors associated with COVID-19 breakthrough infections among vaccinated individuals and to reassess the effectiveness of COVID-19 vaccination against severe outcomes using real-world data. DESIGN SETTING AND PARTICIPANTS: We conducted a series of observational retrospective analyses using the electronic health records (EHRs) of Columbia University Irving Medical Center/New York Presbyterian (CUIMC/NYP) up to September 21, 2021. New York adult residence with PCR test records were included in this analysis. MAIN OUTCOMES AND MEASURES: Poisson regression was used to assess the association between breakthrough infection rate in vaccinated individuals and multiple risk factors - including vaccine brand, demographics, and underlying conditions - while adjusting for calendar month, prior number of visits and observational days. Logistic regression was used to assess the association between vaccine administration and infection rate by comparing a vaccinated cohort to a historically matched cohort in the pre-vaccinated period. Infection incident rate was also compared between vaccinated individuals and longitudinally matched unvaccinated individuals. Cox regression was used to estimate the association of the vaccine and COVID-19 associated severe outcomes by comparing breakthrough cohort and two matched unvaccinated infection cohorts. RESULTS: Individuals vaccinated with Pfizer/BNT162b2 (IRR against Moderna/mRNA-1273 [95% CI]: 1.66 [1.17 - 2.35]); were male (1.47 [1.11 - 1.94%]); and had compromised immune systems (1.48 [1.09 - 2.00]) were at the highest risk for breakthrough infections. Vaccinated individuals had a significant lower infection rate among all subgroups. An increased incidence rate was found in both vaccines over the time. Among individuals infected with COVID-19, vaccination significantly reduced the risk of death (adj. HR: 0.20 [0.08 - 0.49]). CONCLUSION AND RELEVANCE: While we found both mRNA vaccines were effective, Moderna/mRNA-1273 had a lower incidence rate of breakthrough infections. Both vaccines had increased incidence rates over the time. Immunocompromised individuals were among the highest risk groups experiencing breakthrough infections. Given the rapidly changing nature of the SARS-CoV-2, continued monitoring and a generalizable analysis pipeline are warranted to inform quick updates on vaccine effectiveness in real time. KEY POINTS: Question: What risk factors contribute to COVID-19 breakthrough infections among mRNA vaccinated individuals? How do clinical outcomes differ between vaccinated but still SARS-CoV-2 infected individuals and non-vaccinated, infected individuals?Findings: This retrospective study uses CUIMC/NYP EHR data up to September 21, 2021. Individuals who were vaccinated with Pfizer/BNT162b2, male, and had compromised immune systems had significantly higher incidence rate ratios of breakthrough infections. Comparing demographically matched pre-vaccinated and unvaccinated individuals, vaccinated individuals had a lower incidence rate of SARS-CoV-2 infection among all subgroups.Meaning: Leveraging real-world EHR data provides insight on who may optimally benefit from a booster COVID-19 vaccination.

A Framework for Systematic Assessment of Clinical Trial Population Representativeness Using Electronic Health Records Data.

BACKGROUND: Clinical trials are the gold standard for generating robust medical evidence, but clinical trial results often raise generalizability concerns, which can be attributed to the lack of population representativeness. The electronic health records (EHRs) data are useful for estimating the population representativeness of clinical trial study population. OBJECTIVES: This research aims to estimate the population representativeness of clinical trials systematically using EHR data during the early design stage. METHODS: We present an end-to-end analytical framework for transforming free-text clinical trial eligibility criteria into executable database queries conformant with the Observational Medical Outcomes Partnership Common Data Model and for systematically quantifying the population representativeness for each clinical trial. RESULTS: We calculated the population representativeness of 782 novel coronavirus disease 2019 (COVID-19) trials and 3,827 type 2 diabetes mellitus (T2DM) trials in the United States respectively using this framework. With the use of overly restrictive eligibility criteria, 85.7% of the COVID-19 trials and 30.1% of T2DM trials had poor population representativeness. CONCLUSION: This research demonstrates the potential of using the EHR data to assess the clinical trials population representativeness, providing data-driven metrics to inform the selection and optimization of eligibility criteria.

Towards clinical data-driven eligibility criteria optimization for interventional COVID-19 clinical trials.

OBJECTIVE: This research aims to evaluate the impact of eligibility criteria on recruitment and observable clinical outcomes of COVID-19 clinical trials using electronic health record (EHR) data. MATERIALS AND METHODS: On June 18, 2020, we identified frequently used eligibility criteria from all the interventional COVID-19 trials in ClinicalTrials.gov (n = 288), including age, pregnancy, oxygen saturation, alanine/aspartate aminotransferase, platelets, and estimated glomerular filtration rate. We applied the frequently used criteria to the EHR data of COVID-19 patients in Columbia University Irving Medical Center (CUIMC) (March 2020-June 2020) and evaluated their impact on patient accrual and the occurrence of a composite endpoint of mechanical ventilation, tracheostomy, and in-hospital death. RESULTS: There were 3251 patients diagnosed with COVID-19 from the CUIMC EHR included in the analysis. The median follow-up period was 10 days (interquartile range 4-28 days). The composite events occurred in 18.1% (n = 587) of the COVID-19 cohort during the follow-up. In a hypothetical trial with common eligibility criteria, 33.6% (690/2051) were eligible among patients with evaluable data and 22.2% (153/690) had the composite event. DISCUSSION: By adjusting the thresholds of common eligibility criteria based on the characteristics of COVID-19 patients, we could observe more composite events from fewer patients. CONCLUSIONS: This research demonstrated the potential of using the EHR data of COVID-19 patients to inform the selection of eligibility criteria and their thresholds, supporting data-driven optimization of participant selection towards improved statistical power of COVID-19 trials.