Approach to machine learning for extraction of real-world data variables from electronic health records.

Background: As artificial intelligence (AI) continues to advance with breakthroughs in natural language processing (NLP) and machine learning (ML), such as the development of models like OpenAI's ChatGPT, new opportunities are emerging for efficient curation of electronic health records (EHR) into real-world data (RWD) for evidence generation in oncology. Our objective is to describe the research and development of industry methods to promote transparency and explainability. Methods: We applied NLP with ML techniques to train, validate, and test the extraction of information from unstructured documents (e.g., clinician notes, radiology reports, lab reports, etc.) to output a set of structured variables required for RWD analysis. This research used a nationwide electronic health record (EHR)-derived database. Models were selected based on performance. Variables curated with an approach using ML extraction are those where the value is determined solely based on an ML model (i.e. not confirmed by abstraction), which identifies key information from visit notes and documents. These models do not predict future events or infer missing information. Results: We developed an approach using NLP and ML for extraction of clinically meaningful information from unstructured EHR documents and found high performance of output variables compared with variables curated by manually abstracted data. These extraction methods resulted in research-ready variables including initial cancer diagnosis with date, advanced/metastatic diagnosis with date, disease stage, histology, smoking status, surgery status with date, biomarker test results with dates, and oral treatments with dates. Conclusion: NLP and ML enable the extraction of retrospective clinical data in EHR with speed and scalability to help researchers learn from the experience of every person with cancer.

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning.

Meaningful real-world evidence (RWE) generation requires unstructured data found in electronic health records (EHRs) which are often missing from administrative claims; however, obtaining relevant data from unstructured EHR sources is resource-intensive. In response, researchers are using natural language processing (NLP) with machine learning (ML) techniques (i.e., ML extraction) to extract real-world data (RWD) at scale. This study assessed the quality and fitness-for-use of EHR-derived oncology data curated using NLP with ML as compared to the reference standard of expert abstraction. Using a sample of 186,313 patients with lung cancer from a nationwide EHR-derived de-identified database, we performed a series of replication analyses demonstrating some common analyses conducted in retrospective observational research with complex EHR-derived data to generate evidence. Eligible patients were selected into biomarker- and treatment-defined cohorts, first with expert-abstracted then with ML-extracted data. We utilized the biomarker- and treatment-defined cohorts to perform analyses related to biomarker-associated survival and treatment comparative effectiveness, respectively. Across all analyses, the results differed by less than 8% between the data curation methods, and similar conclusions were reached. These results highlight that high-performance ML-extracted variables trained on expert-abstracted data can achieve similar results as when using abstracted data, unlocking the ability to perform oncology research at scale.

Considerations for the Use of Machine Learning Extracted Real-World Data to Support Evidence Generation: A Research-Centric Evaluation Framework.

A vast amount of real-world data, such as pathology reports and clinical notes, are captured as unstructured text in electronic health records (EHRs). However, this information is both difficult and costly to extract through human abstraction, especially when scaling to large datasets is needed. Fortunately, Natural Language Processing (NLP) and Machine Learning (ML) techniques provide promising solutions for a variety of information extraction tasks such as identifying a group of patients who have a specific diagnosis, share common characteristics, or show progression of a disease. However, using these ML-extracted data for research still introduces unique challenges in assessing validity and generalizability to different cohorts of interest. In order to enable effective and accurate use of ML-extracted real-world data (RWD) to support research and real-world evidence generation, we propose a research-centric evaluation framework for model developers, ML-extracted data users and other RWD stakeholders. This framework covers the fundamentals of evaluating RWD produced using ML methods to maximize the use of EHR data for research purposes.

Weekly dengue forecasts in Iquitos, Peru; San Juan, Puerto Rico; and Singapore.

BACKGROUND: Predictive models can serve as early warning systems and can be used to forecast future risk of various infectious diseases. Conventionally, regression and time series models are used to forecast dengue incidence, using dengue surveillance (e.g., case counts) and weather data. However, these models may be limited in terms of model assumptions and the number of predictors that can be included. Machine learning (ML) methods are designed to work with a large number of predictors and thus offer an appealing alternative. Here, we compared the performance of ML algorithms with that of regression models in predicting dengue cases and outbreaks from 4 to up to 12 weeks in advance. Many countries lack sufficient health surveillance infrastructure, as such we evaluated the contribution of dengue surveillance and weather data on the predictive power of these models. METHODS: We developed ML, regression, and time series models to forecast weekly dengue case counts and outbreaks in Iquitos, Peru; San Juan, Puerto Rico; and Singapore from 1990-2016. Forecasts were generated using available weekly dengue surveillance, and weather data. We evaluated the agreement between model forecasts and actual dengue observations using Mean Absolute Error and Matthew's Correlation Coefficient (MCC). RESULTS: For near term predictions of weekly case counts and when using surveillance data, ML models had 21% and 33% less error than regression and time series models respectively. However, using weather data only, ML models did not demonstrate a practical advantage. When forecasting weekly dengue outbreaks 12 weeks in advance, ML models achieved a maximum MCC of 0.61. CONCLUSIONS: Our results identified 2 scenarios when ML models are advantageous over regression model: 1) predicting dengue weekly case counts 4 weeks ahead when dengue surveillance data are available and 2) predicting weekly dengue outbreaks 12 weeks ahead when dengue surveillance data are unavailable. Given the advantages of ML models, dengue early warning systems may be improved by the inclusion of these models.

Periconceptional folic acid and risk for neural tube defects among higher risk pregnancies.

BACKGROUND: Women with a previous neural tube defect (NTD)-affected pregnancy are recommended to consume 4,000 µg daily folic acid (FA) for prevention (10 times the general-population recommendation). Protection from doses between 400 and 4,000 µg for this and other higher risk groups is unclear. METHODS: In the case-control Slone Birth Defects Study (1988-2015), we examined the associations between periconceptional FA doses and NTDs among four higher risk groups: NTD family history, periconceptional antiepileptic drug exposure (AED), pregestational diabetes, and prepregnancy obesity. Mothers completed standardized interviews about pregnancy events and exposures. FA categorizations were based on (a) supplements only and (b) supplements and diet ("total folate"). We estimated odds ratios (ORs) and 95% confidence intervals (CIs) (adjusted for age and study center) using logistic regression. RESULTS: Cases and controls included: 45 and 119 with family history, 25 and 108 with AED exposure, 12 and 63 with pregestational diabetes, 111 and 1,243 with obesity. Daily FA supplementation was associated with lower NTD risk compared to no supplementation (adjusted ORs were 0.33 [95% CI 0.13, 0.76] for family history, 0.31 [0.09, 0.95] for AED exposure, 0.25 [0.04, 1.05] for pregestational diabetes, 0.65 [0.40, 1.04] for obesity). Though estimates were imprecise, as total folate increased stronger point estimates were observed, notably among family history. No mothers with a prior NTD-affected pregnancy supplemented with 4,000 µg. CONCLUSIONS: Our findings reinforce that all women of childbearing potential should consume at least 400 µg FA/day to protect against NTDs. Higher risk groups may benefit from higher doses.

The association between autoimmune disease and 30-day mortality among sepsis ICU patients: a cohort study.

INTRODUCTION: Sepsis results from a dysregulated host response to an infection that is associated with an imbalance between pro- and anti-inflammatory cytokines. This imbalance is hypothesized to be a driver of patient mortality. Certain autoimmune diseases modulate the expression of cytokines involved in the pathophysiology of sepsis. However, the outcomes of patients with autoimmune disease who develop sepsis have not been studied in detail. The objective of this study is to determine whether patients with autoimmune diseases have different sepsis outcomes than patients without these comorbidities. METHODS: Using the Multiparameter Intelligent Monitoring in Intensive Care III database (v. 1.4) which contains retrospective clinical data for over 50,000 adult ICU stays, we compared 30-day mortality risk for sepsis patients with and without autoimmune disease. We used logistic regression models to control for known confounders, including demographics, disease severity, and immunomodulation medications. We used mediation analysis to evaluate how the chronic use of immunomodulation medications affects the relationship between autoimmune disease and 30-day mortality. RESULTS: Our study found a statistically significant 27.00% reduction in the 30-day mortality risk associated with autoimmune disease presence. This association was found to be the strongest (OR 0.71, 95% CI 0.54-0.93, P = 0.014) among patients with septic shock. The autoimmune disease-30-day mortality association was not mediated through the chronic use of immunomodulation medications (indirect effect OR 1.07, 95% CI 1.01-1.13, P = 0.020). CONCLUSIONS: We demonstrated that autoimmune diseases are associated with a lower 30-day mortality risk in sepsis. Our findings suggest that autoimmune diseases affect 30-day mortality through a mechanism unrelated to the chronic use of immunomodulation medications. Since this study was conducted within a single study center, research using data from other medical centers will provide further validation.

Statistical modeling of the effect of rainfall flushing on dengue transmission in Singapore.

BACKGROUND: Rainfall patterns are one of the main drivers of dengue transmission as mosquitoes require standing water to reproduce. However, excess rainfall can be disruptive to the Aedes reproductive cycle by "flushing out" aquatic stages from breeding sites. We developed models to predict the occurrence of such "flushing" events from rainfall data and to evaluate the effect of flushing on dengue outbreak risk in Singapore between 2000 and 2016. METHODS: We used machine learning and regression models to predict days with "flushing" in the dataset based on entomological and corresponding rainfall observations collected in Singapore. We used a distributed lag nonlinear logistic regression model to estimate the association between the number of flushing events per week and the risk of a dengue outbreak. RESULTS: Days with flushing were identified through the developed logistic regression model based on entomological data (test set accuracy = 92%). Predictions were based upon the aggregate number of thresholds indicating unusually rainy conditions over multiple weeks. We observed a statistically significant reduction in dengue outbreak risk one to six weeks after flushing events occurred. For weeks with five or more flushing events, compared with weeks with no flushing events, the risk of a dengue outbreak in the subsequent weeks was reduced by 16% to 70%. CONCLUSIONS: We have developed a high accuracy predictive model associating temporal rainfall patterns with flushing conditions. Using predicted flushing events, we have demonstrated a statistically significant reduction in dengue outbreak risk following flushing, with the time lag well aligned with time of mosquito development from larvae and infection transmission. Vector control programs should consider the effects of hydrological conditions in endemic areas on dengue transmission.

Association of Clomiphene and Assisted Reproductive Technologies With the Risk of Neural Tube Defects.

Clomiphene and assisted reproductive technologies (ART) are methods used to help subfertile couples become pregnant. ART has been reported to be associated with neural tube defects (NTDs) in offspring. To evaluate these associations, we studied mothers of 219 cases and 4,262 controls from the Slone Epidemiology Center Birth Defects Study (1993-2012) who were interviewed within 6 months after delivery about pregnancy events, including use of fertility treatments. We considered exposures to clomiphene (without ART) and ART during the periconceptional period. Logistic regression models were used to calculate adjusted odds ratios and 95% confidence intervals, controlling for education and study center. We observed elevated adjusted odds ratios of 2.1 (95% confidence interval: 0.9, 4.8) and 2.0 (95% confidence interval: 1.1, 3.6) for clomiphene and ART exposure, respectively. We performed a mediation analysis to assess whether the observed elevated NTD risk was mediated through multiple births. For clomiphene exposure without ART use, the direct effect estimate of the adjusted odds ratio (aORDE) was 1.7 and the indirect effect estimate (aORIE) was 1.4. Conversely, for ART exposure, the aORDE was 0.9 and the aORIE was 2.5. Our findings suggest that relatively little of the clomiphene-NTD association is mediated through the pathway of multiple births, while the ART-NTD association was explained by the multiple-births pathway.

Impact of Periconceptional Use of Nitrosatable Drugs on the Risk of Neural Tube Defects.

Nitrosatable drugs (NSDs) can, in the presence of nitrosating agents and highly acidic conditions, form N-nitroso compounds that have been found to be teratogenic in animal models. Using data from the Slone Epidemiology Center Birth Defects Study collected from 1998 to 2012, we compared maternal periconceptional NSD use between 334 neural tube defect cases and 7,619 nonmalformed controls. We categorized NSDs according to their functional group (secondary amine, tertiary amine, and amide). With logistic regression models, we estimated adjusted odds ratios and 95% confidence intervals. Neural tube defect risk was associated with maternal periconceptional use of secondary (adjusted odds ratio (aOR) = 1.7, 95% confidence interval (CI): 1.1, 2.4) and tertiary (aOR = 1.7, 95% CI: 1.2, 2.5) amines; an association was observed for amides, but the 95% confidence interval included the null (aOR = 1.4, 95% CI: 0.7, 2.5). Within the secondary amine group, elevated adjusted odds ratios were observed for 3 drugs but were null for the remaining medications. Increases in risk were observed for both strata of folic acid intake (<400 µg/day, ≥400 µg/day), with a slightly higher risk in the ≥400-µg/day stratum. Our findings support previously reported positive associations between neural tube defects and periconceptional exposure to NSDs containing a secondary or tertiary amine or amide.

Drivers of Emerging Infectious Disease Events as a Framework for Digital Detection.

The growing field of digital disease detection, or epidemic intelligence, attempts to improve timely detection and awareness of infectious disease (ID) events. Early detection remains an important priority; thus, the next frontier for ID surveillance is to improve the recognition and monitoring of drivers (antecedent conditions) of ID emergence for signals that precede disease events. These data could help alert public health officials to indicators of elevated ID risk, thereby triggering targeted active surveillance and interventions. We believe that ID emergence risks can be anticipated through surveillance of their drivers, just as successful warning systems of climate-based, meteorologically sensitive diseases are supported by improved temperature and precipitation data. We present approaches to driver surveillance, gaps in the current literature, and a scientific framework for the creation of a digital warning system. Fulfilling the promise of driver surveillance will require concerted action to expand the collection of appropriate digital driver data.

Risk of spina bifida and maternal cigarette, alcohol, and coffee use during the first month of pregnancy.

This study was conducted to assess the association between the risks of spina bifida (SB) in relation to cigarette, alcohol, and caffeine consumption by women during the first month of pregnancy. Between 1988-2012, this multi-center case-control study interviewed mothers of 776 SB cases and 8,756 controls about pregnancy events and exposures. We evaluated cigarette smoking, frequency of alcohol drinking, and caffeine intake during the first lunar month of pregnancy in relation to SB risk. Logistic regression models were used to calculate adjusted odds ratios and 95% confidence intervals. Levels of cigarette smoking (1-9 and ≥10/day), alcohol intake (average ≥4 drinks/day) and caffeine intake (<1, 1, and ≥2 cups/day) were not likely to be associated with increased risk of SB. Further, results were similar among women who ingested less than the recommended amount of folic acid (400 µg/day).