Accurate, Robust, and Scalable Machine Abstraction of Mayo Endoscopic Subscores From Colonoscopy Reports.

BACKGROUND: The Mayo endoscopic subscore (MES) is an important quantitative measure of disease activity in ulcerative colitis. Colonoscopy reports in routine clinical care usually characterize ulcerative colitis disease activity using free text description, limiting their utility for clinical research and quality improvement. We sought to develop algorithms to classify colonoscopy reports according to their MES. METHODS: We annotated 500 colonoscopy reports from 2 health systems. We trained and evaluated 4 classes of algorithms. Our primary outcome was accuracy in identifying scorable reports (binary) and assigning an MES (ordinal). Secondary outcomes included learning efficiency, generalizability, and fairness. RESULTS: Automated machine learning models achieved 98% and 97% accuracy on the binary and ordinal prediction tasks, outperforming other models. Binary models trained on the University of California, San Francisco data alone maintained accuracy (96%) on validation data from Zuckerberg San Francisco General. When using 80% of the training data, models remained accurate for the binary task (97% [n = 320]) but lost accuracy on the ordinal task (67% [n = 194]). We found no evidence of bias by gender (P = .65) or area deprivation index (P = .80). CONCLUSIONS: We derived a highly accurate pair of models capable of classifying reports by their MES and recognizing when to abstain from prediction. Our models were generalizable on outside institution validation. There was no evidence of algorithmic bias. Our methods have the potential to enable retrospective studies of treatment effectiveness, prospective identification of patients meeting study criteria, and quality improvement efforts in inflammatory bowel diseases.

Our accurate pair of models automatically classify colonoscopy reports by Mayo endoscopic subscore and abstain from prediction appropriately. Our methods can enable large-scale electronic health record studies of treatment effectiveness, prospective identification of patients for clinical trials, and quality improvement efforts in ulcerative colitis.

Algorithmic Identification of Treatment-Emergent Adverse Events From Clinical Notes Using Large Language Models: A Pilot Study in Inflammatory Bowel Disease.

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California - San Francisco (UCSF)-BERT, to identify serious AEs (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF-BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF-BERT was significantly superior at identifying hospitalization events emergent to medication use (P < 0.01). LLMs like UCSF-BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre-trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.

Real-world effectiveness of ustekinumab and vedolizumab in TNF-exposed pediatric patients with ulcerative colitis.

OBJECTIVES: Vedolizumab (VDZ) and ustekinumab (UST) are second-line treatments in pediatric patients with ulcerative colitis (UC) refractory to antitumor necrosis factor (anti-TNF) therapy. Pediatric studies comparing the effectiveness of these medications are lacking. Using a registry from ImproveCareNow (ICN), a global research network in pediatric inflammatory bowel disease, we compared the effectiveness of UST and VDZ in anti-TNF refractory UC. METHODS: We performed a propensity-score weighted regression analysis to compare corticosteroid-free clinical remission (CFCR) at 6 months from starting second-line therapy. Sensitivity analyses tested the robustness of our findings to different ways of handling missing outcome data. Secondary analyses evaluated alternative proxies of response and infection risk. RESULTS: Our cohort included 262 patients on VDZ and 74 patients on UST. At baseline, the two groups differed on their mean pediatric UC activity index (PUCAI) (p = 0.03) but were otherwise similar. At Month 6, 28.3% of patients on VDZ and 25.8% of those on UST achieved CFCR (p = 0.76). Our primary model showed no difference in CFCR (odds ratio: 0.81; 95% confidence interval [CI]: 0.41-1.59) (p = 0.54). The time to biologic discontinuation was similar in both groups (hazard ratio: 1.26; 95% CI: 0.76-2.08) (p = 0.36), with the reference group being VDZ, and we found no differences in clinical response, growth parameters, hospitalizations, surgeries, infections, or malignancy risk. Sensitivity analyses supported these findings of similar effectiveness. CONCLUSIONS: UST and VDZ are similarly effective for inducing clinical remission in anti-TNF refractory UC in pediatric patients. Providers should consider safety, tolerability, cost, and comorbidities when deciding between these therapies.

Personalizing treatment selection in Crohn's disease: a meta-analysis of individual participant data from fifteen randomized controlled trials.

BACKGROUND: Meta-analyses have found anti-TNF drugs to be the best treatment, on average, for Crohn's disease. We performed a subgroup analysis to determine if it is possible to achieve more efficacious outcomes by individualizing treatment selection. METHODS: We obtained participant-level data from 15 trials of FDA-approved treatments (N=5703). We used sequential regression and simulation to model week six disease activity as a function of drug class, demographics, and disease-related features. We performed hypothesis testing to define subgroups based on rank-ordered preferences for treatments. We queried health records from University of California Health (UCH) to estimate the impacts these models could have on practice. We computed the sample size needed to prospectively test a prediction of our models. RESULTS: 45% of the participants (N=2561) showed greater efficacy with at least one drug class (anti-TNF, anti-IL-12/23, anti-integrin) over another. They were classifiable into 6 subgroups, two showing greatest efficacy with anti-TNFs (36%, N=2064). Women over 50 showed superior responses with anti-IL-12/23s. Although they represented only 2% of the trial-based cohort, 25% of Crohn's patients at UCH are women over 50 (N=5,647), consistent with potential selection bias in trials. Moreover, 75% of biologic-exposed women over 50 did not receive an anti-IL12/23 first-line, supporting the potential value of these models. A future trial with 250 patients per arm will have 97% power to confirm the superiority of anti-IL-12/23s over anti-TNFs in these patients. A treatment recommendation tool is available at https://crohnsrx.org. CONCLUSIONS: Personalizing treatment can improve outcomes in Crohn's disease. Future work is needed to confirm these findings, and improve representativeness in Crohn's trials.

Sequential regression and simulation: a method for estimating causal effects from heterogeneous clinical trials without a common control group.

BACKGROUND: The advent of clinical trial data sharing platforms has created opportunities for making new discoveries and answering important questions using already collected data. However, existing methods for meta-analyzing these data require the presence of shared control groups across studies, significantly limiting the number of questions that can be confidently addressed. We sought to develop a method for meta-analyzing potentially heterogeneous clinical trials even in the absence of a common control group. METHODS: This work was conducted within the context of a broader effort to study comparative efficacy in Crohn's disease. Following a search of clnicaltrials.gov we obtained access to the individual participant data from nine trials of FDA-approved treatments in Crohn's Disease (N = 3392). We developed a method involving sequences of regression and simulation to separately model the placebo- and drug-attributable effects, and to simulate head-to-head trials against an appropriately normalized background. We validated this method by comparing the outcome of a simulated trial comparing the efficacies of adalimumab and ustekinumab against the recently published results of SEAVUE, an actual head-to-head trial of these drugs. This study was pre-registered on PROSPERO (#157,827) prior to the completion of SEAVUE. RESULTS: Using our method of sequential regression and simulation, we compared the week eight outcomes of two virtual cohorts subject to the same patient selection criteria as SEAVUE and treated with adalimumab or ustekinumab. Our primary analysis replicated the corresponding published results from SEAVUE (p = 0.9). This finding proved stable under multiple sensitivity analyses. CONCLUSIONS: This new method may help reduce the bias of individual participant data meta-analyses, expand the scope of what can be learned from these already-collected data, and reduce the costs of obtaining high-quality evidence to guide patient care.

Reducing diagnostic delays in Acute Hepatic Porphyria using electronic health records data and machine learning: a multicenter development and validation study.

Importance: Acute Hepatic Porphyria (AHP) is a group of rare but treatable conditions associated with diagnostic delays of fifteen years on average. The advent of electronic health records (EHR) data and machine learning (ML) may improve the timely recognition of rare diseases like AHP. However, prediction models can be difficult to train given the limited case numbers, unstructured EHR data, and selection biases intrinsic to healthcare delivery. Objective: To train and characterize models for identifying patients with AHP. Design Setting and Participants: This diagnostic study used structured and notes-based EHR data from two centers at the University of California, UCSF (2012-2022) and UCLA (2019-2022). The data were split into two cohorts (referral, diagnosis) and used to develop models that predict: 1) who will be referred for testing of acute porphyria, amongst those who presented with abdominal pain (a cardinal symptom of AHP), and 2) who will test positive, amongst those referred. The referral cohort consisted of 747 patients referred for testing and 99,849 contemporaneous patients who were not. The diagnosis cohort consisted of 72 confirmed AHP cases and 347 patients who tested negative. Cases were female predominant and 6-75 years old at the time of diagnosis. Candidate models used a range of architectures. Feature selection was semi-automated and incorporated publicly available data from knowledge graphs. Main Outcomes and Measures: F-score on an outcome-stratified test set. Results: The best center-specific referral models achieved an F-score of 86-91%. The best diagnosis model achieved an F-score of 92%. To further test our model, we contacted 372 current patients who lack an AHP diagnosis but were predicted by our models as potentially having it (≥ 10% probability of referral, ≥ 50% of testing positive). However, we were only able to recruit 10 of these patients for biochemical testing, all of whom were negative. Nonetheless, post hoc evaluations suggested that these models could identify 71% of cases earlier than their diagnosis date, saving 1.2 years. Conclusions and Relevance: ML can reduce diagnostic delays in AHP and other rare diseases. Robust recruitment strategies and multicenter coordination will be needed to validate these models before they can be deployed.

Algorithmic identification of treatment-emergent adverse events from clinical notes using large language models: a pilot study in inflammatory bowel disease.

Background and Aims: Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLM) like BERT have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event detection. Methods: We adapted a new clinical LLM, UCSF BERT, to identify serious adverse events (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. Results: We annotated 928 outpatient IBD notes corresponding to 928 individual IBD patients for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of 8 candidate models, UCSF BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF BERT was significantly superior at identifying hospitalization events emergent to medication use (p < 0.01). Conclusions: LLMs like UCSF BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared to prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multi-center data and newer architectures like GPT. Our findings support the potential value of using large language models to enhance pharmacovigilance.

ALK Inhibitor Treatment Patterns and Outcomes in Real-World Patients with ALK-Positive Non-Small-Cell Lung Cancer: A Retrospective Cohort Study.

BACKGROUND: Randomized trials have demonstrated that anaplastic lymphoma kinase (ALK) tyrosine kinase inhibitors (TKIs) can be safe and efficacious treatments for patients with ALK-positive advanced non-small-cell lung cancer (aNSCLC). However, their safety, tolerability, effectiveness, and patterns of use in real-world patients remain understudied. OBJECTIVE: We sought to assess the overall treatment pattern characteristics, safety, and effectiveness outcomes of real-world patients with ALK-positive aNSCLC receiving ALK TKIs. PATIENTS AND METHODS: This retrospective cohort study using electronic health record data included adult patients with ALK-positive aNSCLC receiving ALK TKIs between January 2012 and November 2021 at a large tertiary medical center, University of California, San Francisco (UCSF), with alectinib or crizotinib as the initial ALK TKI therapy. Our primary endpoints included the incidence of treatment changes (treatment dose adjustments, interruptions, and discontinuations) during the initial ALK TKI treatment, the count and type of subsequent treatments, rates of serious adverse events (sAEs), and major adverse events (mAEs) leading to any ALK TKI treatment changes. Secondary endpoints included the hazard ratios (HRs) for median mAE-free survival (mAEFS), real-world progression-free survival (rwPFS), and overall survival (OS) when comparing alectinib with crizotinib. RESULTS: The cohort consisted of 117 adult patients (70 alectinib and 47 crizotinib) with ALK-positive aNSCLC, with 24.8%, 17.9%, and 6.0% experiencing treatment dose adjustments, interruptions, and discontinuation, respectively. Of the 73 patients whose ALK TKI treatments were discontinued, 68 received subsequent treatments including newer generations of ALK TKIs, immune checkpoint inhibitors, and chemotherapies. The most common mAEs were rash (9.9%) and bradycardia (7.0%) for alectinib and liver toxicity (19.1%) for crizotinib. The most common sAEs were pericardial effusion (5.6%) and pleural effusion (5.6%) for alectinib and pulmonary embolism (6.4%) for crizotinib. Patients receiving alectinib versus crizotinib as their first ALK TKI treatment experienced significantly prolonged median rwPFS (29.3 versus 10.4 months) with an HR of 0.38 (95% CI 0.21-0.67), while prolonged median mAEFS (not reached versus 91.3 months) and OS (54.1 versus 45.8 months) were observed in patients receiving alectinib versus crizotinib but did not reach statistical significance. Yet, it is worth noting that there was a high degree of cross-over post-progression, which could significantly confound the overall survival measures. CONCLUSIONS: We found that ALK TKIs were highly tolerable, and alectinib was associated with favorable survival outcomes with longer time to adverse events (AE) requiring medical interventions, disease progression, and death, in the context of real-world use. Proactive monitoring for adverse events such as rash, bradycardia, and hepatotoxicity may help further promote the safe and optimal use of ALK TKIs in the treatment of patients with aNSCLC.

Creation of an ustekinumab external control arm for Crohn's disease using electronic health records data: A pilot study.

BACKGROUND: Randomized trials are the gold-standard for clinical evidence generation, but they can sometimes be limited by infeasibility and unclear generalizability to real-world practice. External control arm (ECA) studies may help address this evidence gaps by constructing retrospective cohorts that closely emulate prospective ones. Experience in constructing these outside the context of rare diseases or cancer is limited. We piloted an approach for developing an ECA in Crohn's disease using electronic health records (EHR) data. METHODS: We queried EHR databases and manually screened records at the University of California, San Francisco to identify patients meeting the eligibility criteria of TRIDENT, a recently completed interventional trial involving an ustekinumab reference arm. We defined timepoints to balance missing data and bias. We compared imputation models by their impacts on cohort membership and outcomes. We assessed the accuracy of algorithmic data curation against manual review. Lastly, we assessed disease activity following treatment with ustekinumab. RESULTS: Screening identified 183 patients. 30% of the cohort had missing baseline data. Nonetheless, cohort membership and outcomes were robust to the method of imputation. Algorithms for ascertaining non-symptom-based elements of disease activity using structured data were accurate against manual review. The cohort consisted of 56 patients, exceeding planned enrollment in TRIDENT. 34% of the cohort was in steroid-free remission at week 24. CONCLUSION: We piloted an approach for creating an ECA in Crohn's disease from EHR data by using a combination of informatics and manual methods. However, our study reveals significant missing data when standard-of-care clinical data are repurposed. More work will be needed to improve the alignment of trial design with typical patterns of clinical practice, and thereby enable a future of more robust ECAs in chronic diseases like Crohn's disease.

CovidCounties is an interactive real time tracker of the COVID19 pandemic at the level of US counties.

Management of the COVID-19 pandemic has proven to be a significant challenge to policy makers. This is in large part due to uneven reporting and the absence of open-access visualization tools to present local trends and infer healthcare needs. Here we report the development of CovidCounties.org, an interactive web application that depicts daily disease trends at the level of US counties using time series plots and maps. This application is accompanied by a manually curated dataset that catalogs all major public policy actions made at the state-level, as well as technical validation of the primary data. Finally, the underlying code for the site is also provided as open source, enabling others to validate and learn from this work.

CovidCounties - an interactive, real-time tracker of the COVID-19 pandemic at the level of US counties.

Management of the COVID-19 pandemic has proven to be a significant challenge to policy makers. This is in large part due to uneven reporting and the absence of open-access visualization tools to present local trends and infer healthcare needs. Here we report the development of CovidCounties.org, an interactive web application that depicts daily disease trends at the level of US counties using time series plots and maps. This application is accompanied by a manually curated dataset that catalogs all major public policy actions made at the state-level, as well as technical validation of the primary data. Finally, the underlying code for the site is also provided as open source, enabling others to validate and learn from this work.

Accuracy of medical billing data against the electronic health record in the measurement of colorectal cancer screening rates.

OBJECTIVE: Medical billing data are an attractive source of secondary analysis because of their ease of use and potential to answer population-health questions with statistical power. Although these datasets have known susceptibilities to biases, the degree to which they can distort the assessment of quality measures such as colorectal cancer screening rates are not widely appreciated, nor are their causes and possible solutions. METHODS: Using a billing code database derived from our institution's electronic health records, we estimated the colorectal cancer screening rate of average-risk patients aged 50-74 years seen in primary care or gastroenterology clinic in 2016-2017. 200 records (150 unscreened, 50 screened) were sampled to quantify the accuracy against manual review. RESULTS: Out of 4611 patients, an analysis of billing data suggested a 61% screening rate, an estimate that matches the estimate by the Centers for Disease Control. Manual review revealed a positive predictive value of 96% (86%-100%), negative predictive value of 21% (15%-29%) and a corrected screening rate of 85% (81%-90%). Most false negatives occurred due to examinations performed outside the scope of the database-both within and outside of our institution-but 21% of false negatives fell within the database's scope. False positives occurred due to incomplete examinations and inadequate bowel preparation. Reasons for screening failure include ordered but incomplete examinations (48%), lack of or incorrect documentation by primary care (29%) including incorrect screening intervals (13%) and patients declining screening (13%). CONCLUSIONS: Billing databases are prone to substantial bias that may go undetected even in the presence of confirmatory external estimates. Caution is recommended when performing population-level inference from these data. We propose several solutions to improve the use of these data for the assessment of healthcare quality.

Opportunities and challenges in using real-world data for health care.

Real-world data (RWD) continue to emerge as a new source of clinical evidence. Although the best-known use case of RWD has been in drug regulation, RWD are being generated and used by many other parties, including biopharmaceutical companies, payors, clinical researchers, providers, and patients. In this Review, we describe 21 potential uses for RWD across the spectrum of health care. We also discuss important challenges and limitations relevant to the translation of these data into evidence.

Safety of Flexible Sigmoidoscopy in Pregnant Patients with Known or Suspected Inflammatory Bowel Disease.

BACKGROUND AND AIMS: Lower gastrointestinal endoscopy is crucial in the diagnosis and staging of inflammatory bowel disease (IBD). However, there are limited safety data in pregnant populations, resulting in conservative society guidelines and practice patterns favoring diagnostic delay. We studied whether performance of flexible sigmoidoscopy is associated with adverse events in pregnant patients with known or suspected IBD. METHODS: A retrospective cohort study was conducted at the University of California San Francisco (UCSF) between April 2008 and April 2019. Female patients aged between 18 and 48 years who were pregnant at the time of endoscopy were identified. All patient records were reviewed to determine disease, pregnancy outcomes, and lifestyle factors. Two independent reviewers performed the data abstraction. Adverse events were assessed for temporal relation with endoscopy. RESULTS: We report the outcomes of 48 pregnant patients across all trimesters who underwent lower endoscopy for suspected or established IBD. There were no hospitalizations or adverse obstetric events temporally associated with sigmoidoscopy. 78% (38/50) of lower endoscopies performed in the patients resulted in a change in treatment following sigmoidoscopy. 12% (5/43) of the lower endoscopies performed in patients with known IBD showed no endoscopic evidence of disease activity despite symptoms. CONCLUSIONS: Lower endoscopy in the pregnant patient with known or suspected IBD is low risk and affects therapeutic decision making. It should not be delayed in patients with appropriate indications.

Biosimilars for the Treatment of Inflammatory Bowel Disease.

Biosimilars are biologic products that are highly similar to a previously approved reference (or originator) biologic drug in terms of safety, purity, and potency (efficacy). These medications are increasingly being approved by global regulatory agencies in the hopes of reducing treatment costs. To date, six biosimilars in the United States have been approved for the treatment of inflammatory bowel disease (IBD). Despite their approval by regulatory bodies and several years-worth of 'real world' evidence supporting their use, this class of medications remain somewhat unfamiliar to many clinicians and patients. This review aims to answer common questions regarding biosimilars and their use for IBD. It is written in a question/answer format for easy reference and guides the reader from the basics of biosimilars, to clinically relevant questions encountered in the clinic, to their policy implications, among other topics.

PatientExploreR: an extensible application for dynamic visualization of patient clinical history from electronic health records in the OMOP common data model.

MOTIVATION: Electronic health records (EHRs) are quickly becoming omnipresent in healthcare, but interoperability issues and technical demands limit their use for biomedical and clinical research. Interactive and flexible software that interfaces directly with EHR data structured around a common data model (CDM) could accelerate more EHR-based research by making the data more accessible to researchers who lack computational expertise and/or domain knowledge. RESULTS: We present PatientExploreR, an extensible application built on the R/Shiny framework that interfaces with a relational database of EHR data in the Observational Medical Outcomes Partnership CDM format. PatientExploreR produces patient-level interactive and dynamic reports and facilitates visualization of clinical data without any programming required. It allows researchers to easily construct and export patient cohorts from the EHR for analysis with other software. This application could enable easier exploration of patient-level data for physicians and researchers. PatientExploreR can incorporate EHR data from any institution that employs the CDM for users with approved access. The software code is free and open source under the MIT license, enabling institutions to install and users to expand and modify the application for their own purposes. AVAILABILITY AND IMPLEMENTATION: PatientExploreR can be freely obtained from GitHub: https://github.com/BenGlicksberg/PatientExploreR. We provide instructions for how researchers with approved access to their institutional EHR can use this package. We also release an open sandbox server of synthesized patient data for users without EHR access to explore: http://patientexplorer.ucsf.edu. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Caspase signalling in the absence of apoptosis drives Jnk-dependent invasion.

Tumours evolve several mechanisms to evade apoptosis, yet many resected carcinomas show significantly elevated caspase activity. Moreover, caspase activity is positively correlated with tumour aggression and adverse patient outcome. These observations indicate that caspases might have a functional role in promoting tumour invasion and metastasis. Using a Drosophila model of invasion, we show that precise effector caspase activity drives cell invasion without initiating apoptosis. Affected cells express the matrix metalloprotinase Mmp1 and invade by activating Jnk. Our results link Jnk and effector caspase signalling during the invasive process and suggest that tumours under apoptotic stresses from treatment, immune surveillance or intrinsic signals might be induced further along the metastatic cascade.

Drosophila cancer models.

Cancer is driven by complex genetic and cellular mechanisms. Recently, the Drosophila community has become increasingly interested in exploring cancer issues. The Drosophila field has made seminal contributions to many of the mechanisms that are fundamental to the cancer process; several of these mechanisms have already been validated in vertebrates. Less well known are the Drosophila field's early direct contributions to the cancer field: some of the earliest tumor suppressors were identified in flies. In this review, we identify major contributions that Drosophila studies have made toward dissecting the pathways and mechanisms underlying tumor progression. We also highlight areas, such as drug discovery, where we expect Drosophila studies to make a major scientific impact in the future.