Automated Versus Semi-Automated Lab Value Extraction for the VA Cardiac Surgical Quality Improvement Program.

INTRODUCTION: The Veterans Affairs Surgical Quality Improvement Program (VASQIP) trains surgical quality nurses (SQNs) at each Veterans Affairs (VA) hospital to extract or verify 187 variables from the medical record for all cardiac surgical cases. For ten preoperative laboratory values, VASQIP has a semiautomated (SA) system in which local lab values are automatically extracted, verified by SQNs, and lab values recorded at other VA facilities are manually extracted. The objective of this study was to develop and validate a method to automate the extraction of these ten preoperative laboratory values and compare results with the current SA method. MATERIALS AND METHODS: We developed methods to extract ten preoperative laboratory values and measurement dates from the VA Corporate Data Warehouse using Logical Observation Identifiers Names and Codes. Automated (A) versus SA information extraction was compared in terms of agreement, conformance to data definitions, proximity to surgery, and missingness. RESULTS: For surgeries with both A and SA lab values, the intraclass correlation coefficients for the ten variables ranged from 0.90 to 0.98. For several variables, the A method resulted in much lower rates of missing data (e.g., 2.4% versus 22.5% missing data for high-density lipoprotein) and eliminated out-of-date-range entries. CONCLUSIONS: Although SQN-extracted data are widely considered the gold standard within National Surgical Quality Improvement Programs, there may be advantages to fully automating extraction of lab values, including high congruence with SA SQN-extracted or verified values and lower rates of missingness and out-of-date-range data.

Evaluating accuracy and fairness of clinical decision support algorithms when health care resources are limited.

OBJECTIVE: Guidance on how to evaluate accuracy and algorithmic fairness across subgroups is missing for clinical models that flag patients for an intervention but when health care resources to administer that intervention are limited. We aimed to propose a framework of metrics that would fit this specific use case. METHODS: We evaluated the following metrics and applied them to a Veterans Health Administration clinical model that flags patients for intervention who are at risk of overdose or a suicidal event among outpatients who were prescribed opioids (N = 405,817): Receiver - Operating Characteristic and area under the curve, precision - recall curve, calibration - reliability curve, false positive rate, false negative rate, and false omission rate. In addition, we developed a new approach to visualize false positives and false negatives that we named 'per true positive bars.' We demonstrate the utility of these metrics to our use case for three cohorts of patients at the highest risk (top 0.5 %, 1.0 %, and 5.0 %) by evaluating algorithmic fairness across the following age groups: <=30, 31-50, 51-65, and >65 years old. RESULTS: Metrics that allowed us to assess group differences more clearly were the false positive rate, false negative rate, false omission rate, and the new 'per true positive bars'. Metrics with limited utility to our use case were the Receiver - Operating Characteristic and area under the curve, the calibration - reliability curve, and the precision - recall curve. CONCLUSION: There is no "one size fits all" approach to model performance monitoring and bias analysis. Our work informs future researchers and clinicians who seek to evaluate accuracy and fairness of predictive models that identify patients to intervene on in the context of limited health care resources. In terms of ease of interpretation and utility for our use case, the new 'per true positive bars' may be the most intuitive to a range of stakeholders and facilitates choosing a threshold that allows weighing false positives against false negatives, which is especially important when predicting severe adverse events.

Disparities in Cancer Mortality among Disaggregated Asian American Subpopulations, 2018-2021.

Federal, state, and institutional data collection practices and analyses involving Asian Americans as a single, aggregated group obscure critical health disparities among the vast diversity of Asian American subpopulations. Using from the Centers for Disease Control and Prevention Wide-Ranging Online Data for Epidemiologic Research (CDC WONDER) Underlying Causes of Death database, we conducted a cross-sectional study using data on disaggregated Asian American subgroups (Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, other Asians) between 2018 and 2021. We examine deaths from 22 cancer types and in situ, benign neoplasms, identified using ICD-10 codes C00-C97 and D00-D48. Overall, our study comprised 327,311 Asian American decedents, with a mean age of death at 70.57 years (SD=2.79), wherein females accounted for approximately half of the sample (n=36,596/73,207; 49.99%). Notably, compared to the aggregated Asian American reference group, we found higher proportions of deaths from total cancers among Chinese (25.99% vs. 22.37% [ref]), Korean (25.29% vs. 22.37% [ref]), and Vietnamese (24.98% vs. 22.37% [ref]) subgroups. In contrast, total cancer deaths were less prevalent among Asian Indians (17.49% vs. 22.37% [ref]), Japanese (18.90% vs. 22.37% [ref]), and other Asians (20.37% vs. 22.37% [ref]). We identified further disparities by cancer type, sex, and age. Disaggregated data collection and analyses are imperative to understanding differences in cancer mortality among Asian American subgroups, illustrating at-risk populations with greater granularity. Future studies should aim to describe the association between these trends and social, demographic, and environmental risk factors.

Question-answering system extracts information on injection drug use from clinical notes.

BACKGROUND: Injection drug use (IDU) can increase mortality and morbidity. Therefore, identifying IDU early and initiating harm reduction interventions can benefit individuals at risk. However, extracting IDU behaviors from patients' electronic health records (EHR) is difficult because there is no other structured data available, such as International Classification of Disease (ICD) codes, and IDU is most often documented in unstructured free-text clinical notes. Although natural language processing can efficiently extract this information from unstructured data, there are no validated tools. METHODS: To address this gap in clinical information, we design a question-answering (QA) framework to extract information on IDU from clinical notes for use in clinical operations. Our framework involves two main steps: (1) generating a gold-standard QA dataset and (2) developing and testing the QA model. We use 2323 clinical notes of 1145 patients curated from the US Department of Veterans Affairs (VA) Corporate Data Warehouse to construct the gold-standard dataset for developing and evaluating the QA model. We also demonstrate the QA model's ability to extract IDU-related information from temporally out-of-distribution data. RESULTS: Here, we show that for a strict match between gold-standard and predicted answers, the QA model achieves a 51.65% F1 score. For a relaxed match between the gold-standard and predicted answers, the QA model obtains a 78.03% F1 score, along with 85.38% Precision and 79.02% Recall scores. Moreover, the QA model demonstrates consistent performance when subjected to temporally out-of-distribution data. CONCLUSIONS: Our study introduces a QA framework designed to extract IDU information from clinical notes, aiming to enhance the accurate and efficient detection of people who inject drugs, extract relevant information, and ultimately facilitate informed patient care.

There are many health risks associated with injection drug use (IDU). Identifying people who inject drugs early can reduce the likelihood of these issues arising. However, extracting information about any possible IDU from a person's electronic health records can be difficult because the information is often in text-based general clinical notes rather than provided in a particular section of the record or as numerical data. Manually extracting information from these notes is time-consuming and inefficient. We used a computational method to train computer software to be able to extract IDU details. Potentially, this approach could be used by healthcare providers to more efficiently and accurately identify people who inject drugs, and therefore provide better advice and medical care.

Disaggregating Asian-American Mortality in Drug-Related Overdoses and Behavioral Disorders: A Cross-Sectional Study.

Asian Americans have been historically underrepresented in the national drug overdose discourse due to their lower substance use and overdose rates compared to other racial/ethnic groups. However, aggregated analyses fail to capture the vast diversity among Asian-American subgroups, obscuring critical disparities. We conducted a cross-sectional study between 2018 and 2021 examining Asian-American individuals within the CDC WONDER database with drug overdoses as the underlying cause of death (n = 3195; ICD-10 codes X40-X44, X60-X64, X85, and Y10-Y14) or psychoactive substance-related mental and behavioral disorders as one of multiple causes of death (n = 15,513; ICD-10 codes F10-F19). Proportional mortality ratios were calculated, comparing disaggregated Asian-American subgroups to the reference group (Asian Americans as a single aggregate group). Z-tests identified significant differences between subgroups. Compared to the reference group (0.99%), drug overdose deaths were less prevalent among Japanese (0.46%; p < 0.001), Chinese (0.47%; p < 0.001), and Filipino (0.82%; p < 0.001) subgroups, contrasting with a higher prevalence among Asian Indian (1.20%; p < 0.001), Vietnamese (1.35%; p < 0.001), Korean (1.36%; p < 0.001), and other Asian (1.79%; p < 0.001) subgroups. Similarly, compared to the reference group (4.80%), deaths from mental and behavioral disorders were less prevalent among Chinese (3.18%; p < 0.001), Filipino (4.52%; p < 0.001), and Asian Indian (4.56%; p < 0.001) subgroups, while more prevalent among Korean (5.60%; p < 0.001), Vietnamese (5.64%; p < 0.001), Japanese (5.81%; p < 0.001), and other Asian (6.14%; p < 0.001) subgroups. Disaggregated data also revealed substantial geographical variations in these deaths obscured by aggregated analyses. Our findings revealed pronounced intra-racial disparities, underscoring the importance of data disaggregation to inform targeted clinical and public health interventions.

High dimensional predictions of suicide risk in 4.2 million US Veterans using ensemble transfer learning.

We present an ensemble transfer learning method to predict suicide from Veterans Affairs (VA) electronic medical records (EMR). A diverse set of base models was trained to predict a binary outcome constructed from reported suicide, suicide attempt, and overdose diagnoses with varying choices of study design and prediction methodology. Each model used twenty cross-sectional and 190 longitudinal variables observed in eight time intervals covering 7.5 years prior to the time of prediction. Ensembles of seven base models were created and fine-tuned with ten variables expected to change with study design and outcome definition in order to predict suicide and combined outcome in a prospective cohort. The ensemble models achieved c-statistics of 0.73 on 2-year suicide risk and 0.83 on the combined outcome when predicting on a prospective cohort of [Formula: see text] 4.2 M veterans. The ensembles rely on nonlinear base models trained using a matched retrospective nested case-control (Rcc) study cohort and show good calibration across a diversity of subgroups, including risk strata, age, sex, race, and level of healthcare utilization. In addition, a linear Rcc base model provided a rich set of biological predictors, including indicators of suicide, substance use disorder, mental health diagnoses and treatments, hypoxia and vascular damage, and demographics.

Development of a 3-Step theory of suicide ontology to facilitate 3ST factor extraction from clinical progress notes.

OBJECTIVE: Suicide risk prediction algorithms at the Veterans Health Administration (VHA) do not include predictors based on the 3-Step Theory of suicide (3ST), which builds on hopelessness, psychological pain, connectedness, and capacity for suicide. These four factors are not available from structured fields in VHA electronic health records, but they are found in unstructured clinical text. An ontology and controlled vocabulary that maps psychosocial and behavioral terms to these factors does not exist. The objectives of this study were 1) to develop an ontology with a controlled vocabulary of terms that map onto classes that represent the 3ST factors as identified within electronic clinical progress notes, and 2) to determine the accuracy of automated extractions based on terms in the controlled vocabulary. METHODS: A team of four annotators did linguistic annotation of 30,000 clinical progress notes from 231 Veterans in VHA electronic health records who attempted suicide or who died by suicide for terms relating to the 3ST factors. Annotation involved manually assigning a label to words or phrases that indicated presence or absence of the factor (polarity). These words and phrases were entered into a controlled vocabulary that was then used by our computational system to tag 14 million clinical progress notes from Veterans who attempted or died by suicide after 2013. Tagged text was extracted and machine-labelled for presence or absence of the 3ST factors. Accuracy of these machine-labels was determined for 1000 randomly selected extractions for each factor against a ground truth created by our annotators. RESULTS: Linguistic annotation identified 8486 terms that related to 33 subclasses across the four factors and polarities. Precision of machine-labeled extractions ranged from 0.73 to 1.00 for most factor-polarity combinations, whereas recall was somewhat lower 0.65-0.91. CONCLUSION: The ontology that was developed consists of classes that represent each of the four 3ST factors, subclasses, relationships, and terms that map onto those classes which are stored in a controlled vocabulary (https://bioportal.bioontology.org/ontologies/THREE-ST). The use case that we present shows how scores based on clinical notes tagged for terms in the controlled vocabulary capture meaningful change in the 3ST factors during weeks preceding a suicidal event.

Deploying a national clinical text processing infrastructure.

OBJECTIVES: Clinical text processing offers a promising avenue for improving multiple aspects of healthcare, though operational deployment remains a substantial challenge. This case report details the implementation of a national clinical text processing infrastructure within the Department of Veterans Affairs (VA). METHODS: Two foundational use cases, cancer case management and suicide and overdose prevention, illustrate how text processing can be practically implemented at scale for diverse clinical applications using shared services. RESULTS: Insights from these use cases underline both commonalities and differences, providing a replicable model for future text processing applications. CONCLUSIONS: This project enables more efficient initiation, testing, and future deployment of text processing models, streamlining the integration of these use cases into healthcare operations. This project implementation is in a large integrated health delivery system in the United States, but we expect the lessons learned to be relevant to any health system, including smaller local and regional health systems in the United States.

The Problem of Pain in Rheumatology: Variations in Case Definitions Derived From Chronic Pain Phenotyping Algorithms Using Electronic Health Records.

OBJECTIVE: The aim of this study was to investigate and compare different case definitions for chronic pain to provide estimates of possible misclassification when researchers are limited by available electronic health record and administrative claims data, allowing for greater precision in case definitions. METHODS: We compared the prevalence of different case definitions for chronic pain (N = 3042) in patients with autoimmune rheumatic diseases. We estimated the prevalence of chronic pain based on 15 unique combinations of pain scores, diagnostic codes, analgesic medications, and pain interventions. RESULTS: Chronic pain prevalence was lowest in unimodal pain phenotyping algorithms: 15% using analgesic medications, 18% using pain scores, 21% using pain diagnostic codes, and 22% using pain interventions. In comparison, the prevalence using a well-validated phenotyping algorithm was 37%. The prevalence of chronic pain also increased with the increasing number (bimodal to quadrimodal) of phenotyping algorithms that comprised the multimodal phenotyping algorithms. The highest estimated chronic pain prevalence (47%) was the multimodal phenotyping algorithm that combined pain scores, diagnostic codes, analgesic medications, and pain interventions. However, this quadrimodal phenotyping algorithm yielded a 10% overestimation of chronic pain compared to the well-validated algorithm. CONCLUSION: This is the first empirical study to our knowledge that shows that established common modes of phenotyping chronic pain can lead to substantially varying estimates of the number of patients with chronic pain. These findings can be a reference for biases in case definitions for chronic pain and could be used to estimate the extent of possible misclassifications or corrections in using datasets that cannot include specific data elements.

Eliminating Algorithmic Racial Bias in Clinical Decision Support Algorithms: Use Cases from the Veterans Health Administration.

The Veterans Health Administration uses equity- and evidence-based principles to examine, correct, and eliminate use of potentially biased clinical equations and predictive models. We discuss the processes, successes, challenges, and next steps in four examples. We detail elimination of the race modifier for estimated kidney function and discuss steps to achieve more equitable pulmonary function testing measurement. We detail the use of equity lenses in two predictive clinical modeling tools: Stratification Tool for Opioid Risk Mitigation (STORM) and Care Assessment Need (CAN) predictive models. We conclude with consideration of ways to advance racial health equity in clinical decision support algorithms.

A call for better validation of opioid overdose risk algorithms.

Clinical decision support (CDS) systems powered by predictive models have the potential to improve the accuracy and efficiency of clinical decision-making. However, without sufficient validation, these systems have the potential to mislead clinicians and harm patients. This is especially true for CDS systems used by opioid prescribers and dispensers, where a flawed prediction can directly harm patients. To prevent these harms, regulators and researchers have proposed guidance for validating predictive models and CDS systems. However, this guidance is not universally followed and is not required by law. We call on CDS developers, deployers, and users to hold these systems to higher standards of clinical and technical validation. We provide a case study on two CDS systems deployed on a national scale in the United States for predicting a patient's risk of adverse opioid-related events: the Stratification Tool for Opioid Risk Mitigation (STORM), used by the Veterans Health Administration, and NarxCare, a commercial system.

Sarcoidosis rates in BCG-vaccinated and unvaccinated young adults: A natural experiment using Danish registers.

OBJECTIVES: Sarcoidosis may have an infectious trigger, including Mycobacterium spp. The Bacille Calmette-Guérin (BCG) vaccine provides partial protection against tuberculosis and induces trained immunity. We examined the incidence rate (IR) of sarcoidosis in Danish individuals born during high BCG vaccine uptake (born before 1976) compared with individuals born during low BCG vaccine uptake (born in or after 1976). METHODS: We performed a quasi-randomized registry-based incidence study using data from the Danish Civil Registration System and the Danish National Patient Registry between 1995 and 2016. We included individuals aged 25-35 years old and born between 1970 and 1981. Using Poisson regression models, we calculated the incidence rate ratio (IRR) of sarcoidosis in individuals born during low BCG vaccine uptake versus high BCG vaccine uptake, adjusting for age and calendar year (separately for men and women). RESULTS: The IR of sarcoidosis was increased for individuals born during low BCG vaccine uptake compared with individuals born during high BCG vaccine uptake, which was largely attributed to men. The IRR of sarcoidosis for men born during low BCG vaccine uptake versus high BCG vaccine uptake was 1.22 (95% confidence interval [CI] 1.02-1.45). In women, the IRR was 1.08 (95% CI 0.88-1.31). CONCLUSION: In this quasi-experimental study that minimizes confounding, the time period with high BCG vaccine uptake was associated with a lower incidence rate of sarcoidosis in men, with a similar effect seen in women that did not reach significance. Our findings support a potential protective effect of BCG vaccination against the development of sarcoidosis. Future interventional studies for high-risk individuals could be considered.

Practical Considerations for Developing Clinical Natural Language Processing Systems for Population Health Management and Measurement.

Experts have noted a concerning gap between clinical natural language processing (NLP) research and real-world applications, such as clinical decision support. To help address this gap, in this viewpoint, we enumerate a set of practical considerations for developing an NLP system to support real-world clinical needs and improve health outcomes. They include determining (1) the readiness of the data and compute resources for NLP, (2) the organizational incentives to use and maintain the NLP systems, and (3) the feasibility of implementation and continued monitoring. These considerations are intended to benefit the design of future clinical NLP projects and can be applied across a variety of settings, including large health systems or smaller clinical practices that have adopted electronic medical records in the United States and globally.

Revelations from a Machine Learning Analysis of the Most Downloaded Articles Published in Journal of Palliative Medicine 1999-2018.

The Journal of Palliative Medicine (JPM) is globally recognized as a leading interdisciplinary peer-reviewed palliative care journal providing balanced information that informs and improves the practice of palliative care. JPM shapes the values, integrity, and standards of the subspecialty of palliative medicine by what it chooses to publish. The global JPM readership chooses to download the articles that are of most relevance and utility to them. Utilizing machine learning methods, the top 100 most downloaded articles in JPM were analyzed to gain a better understanding of any latent trends and patterns in the topics between 1999 and 2018. The top five topic themes identified in the first decade were different from the ones identified in the second decade of publication. There is evidence of differentiation and maturation of the field in the context of comprehensive health care. Although noncancer serious illnesses have still not risen to the same prominence as cancer palliation, there is a directional quality to the emerging evidence as it pertains to cardiac, respiratory, neurological, renal, and other etiologies. Across both decades under study, there was persistent evidence of the importance of understanding and managing the mental health care needs of seriously ill patients and their families. A cause for concern is that the word "spirituality" was prominent in the first decade and was lacking in the second. Future palliative care clinical and research initiatives should focus on its development as an essential interprofessional and medical subspecialty germane to all types of serious illnesses and across all venues.

Development of a Natural Language Processing System for Extracting Rheumatoid Arthritis Outcomes From Clinical Notes Using the National Rheumatology Informatics System for Effectiveness Registry.

OBJECTIVE: To accelerate the use of outcome measures in rheumatology, we developed and evaluated a natural language processing (NLP) pipeline for extracting these measures from free-text outpatient rheumatology notes within the American College of Rheumatology's Rheumatology Informatics System for Effectiveness (RISE) registry. METHODS: We included all patients in RISE (2015-2018). The NLP pipeline extracted scores corresponding to 8 measures of rheumatoid arthritis (RA) disease activity (DA) and functional status (FS) documented in outpatient rheumatology notes. Score extraction performance was evaluated by chart review, and we assessed agreement with scores documented in structured data. We conducted an external validation of our NLP pipeline using data from rheumatology notes from an academic medical center that is not included in the RISE registry. RESULTS: We processed over 34 million notes from 854,628 patients, 158 practices, and 24 electronic health record (EHR) systems from RISE. Manual chart review revealed a sensitivity, positive predictive value (PPV), and F1 score of 95%, 87%, and 91%, respectively. Substantial agreement was observed between scores extracted from RISE notes and scores derived from structured data (κ = 0.43-0.68 among DA and 0.86-0.98 among FS measures). In the external validation, we found a sensitivity, PPV, and F1 score of 92%, 69%, and 79%, respectively. CONCLUSION: We developed an NLP pipeline to extract RA outcome measures from a national registry of notes from multiple EHR systems and found it to have good internal and external validity. This pipeline can facilitate measurement of clinical- and patient-reported outcomes for use in research and quality measurement.

Development and validation of MedDRA Tagger: a tool for extraction and structuring medical information from clinical notes.

Rapid and automated extraction of clinical information from patients' notes is a desirable though difficult task. Natural language processing (NLP) and machine learning have great potential to automate and accelerate such applications, but developing such models can require a large amount of labeled clinical text, which can be a slow and laborious process. To address this gap, we propose the MedDRA tagger, a fast annotation tool that makes use of industrial level libraries such as spaCy, biomedical ontologies and weak supervision to annotate and extract clinical concepts at scale. The tool can be used to annotate clinical text and obtain labels for training machine learning models and further refine the clinical concept extraction performance, or to extract clinical concepts for observational study purposes. To demonstrate the usability and versatility of our tool, we present three different use cases: we use the tagger to determine patients with a primary brain cancer diagnosis, we show evidence of rising mental health symptoms at the population level and our last use case shows the evolution of COVID-19 symptomatology throughout three waves between February 2020 and October 2021. The validation of our tool showed good performance on both specific annotations from our development set (F1 score 0.81) and open source annotated data set (F1 score 0.79). We successfully demonstrate the versatility of our pipeline with three different use cases. Finally, we note that the modular nature of our tool allows for a straightforward adaptation to another biomedical ontology. We also show that our tool is independent of EHR system, and as such generalizable.

Sarcoidosis incidence after mTOR inhibitor treatment.

OBJECTIVE: Mechanistic target of rapamycin (mTOR) inhibitors are effective in animal models of granulomatous disease, but their benefit in sarcoidosis patients is unknown. We evaluated the incidence of sarcoidosis in patients treated with mTOR inhibitors versus calcineurin inhibitors. METHODS: This was a cohort study using the Optum Clinformatics® Data Mart (CDM) Database (2003-2019), IBM® MarketScan® Research Database (2006-2016), and Danish health and administrative registries (1996-2018). Patients aged ≥18 years with ≥1 year continuous enrollment before and after kidney, liver, heart, or lung transplant treated with an mTOR inhibitor or calcineurin inhibitor were included. Patients diagnosed with sarcoidosis before, or up to 90 days after, transplant were excluded. The incidence of sarcoidosis by treatment group was calculated. RESULTS: In the Optum CDM/IBM MarketScan cohort, 1,898 patients were treated with an mTOR inhibitor (mean age 49 years; 34% female) and 9,894 patients were treated with a calcineurin inhibitor (mean age 50 years; 37% female). The mean follow-up in the mTOR inhibitor group was 1.1 years, with no incident sarcoidosis diagnosed. In the calcineurin inhibitor group, the mean follow-up was 2.2 years, with 12 incident sarcoidosis cases diagnosed. In the Danish cohort, 230 patients were treated with an mTOR inhibitor (mean age 49; 45% female), with no incident sarcoidosis diagnosed. There were 3,411 patients treated with a calcineurin inhibitor (mean age 45; 40% female), with 10 incident cases of sarcoidosis diagnosed. CONCLUSIONS: This study indicates a potential protective effect of mTOR inhibitor treatment compared with calcineurin inhibitor treatment against the development of sarcoidosis.