Using Natural Language Processing and Machine Learning to Identify Internal Medicine-Pediatrics Residency Values in Applications.

PROBLEM: Although holistic review has been used successfully in some residency programs to decrease bias, such review is time-consuming and unsustainable for many programs without initial prescreening. The unstructured qualitative data in residency applications, including notable experiences, letters of recommendation, personal statement, and medical student performance evaluations, require extensive time, resources, and metrics to evaluate; therefore, previous applicant screening relied heavily on quantitative metrics, which can be socioeconomically and racially biased. APPROACH: Using residency applications to the University of Utah internal medicine-pediatrics program from 2015 to 2019, the authors extracted relevant snippets of text from the narrative sections of applications. Expert reviewers annotated these snippets into specific values (academic strength; intellectual curiosity; compassion; communication; work ethic; teamwork; leadership; self-awareness; diversity, equity, and inclusion; professionalism; and adaptability) previously identified as associated with resident success. The authors prospectively applied a machine learning model (MLM) to snippets from applications from 2023, and output was compared with a manual holistic review performed without knowledge of MLM results. OUTCOMES: Overall, the MLM had a sensitivity of 0.64, specificity of 0.97, positive predictive value of 0.62, negative predictive value of 0.97, and F1 score of 0.63. The mean (SD) total number of annotations per application was significantly correlated with invited for interview status (invited: 208.6 [59.1]; not invited: 145.2 [57.2]; P < .001). In addition, 8 of the 10 individual values were significantly predictive of an applicant's invited for interview status. NEXT STEPS: The authors created an MLM that can identify several values important for resident success in internal medicine-pediatrics programs with moderate sensitivity and high specificity. The authors will continue to refine the MLM by increasing the number of annotations, exploring parameter tuning and feature engineering options, and identifying which application sections have the highest correlation with invited for interview status.

Natural language processing for the surveillance of postoperative venous thromboembolism.

BACKGROUND: The objective of this study was to develop a portal natural language processing approach to aid in the identification of postoperative venous thromboembolism events from free-text clinical notes. METHODS: We abstracted clinical notes from 25,494 operative events from 2 independent health care systems. A venous thromboembolism detected as part of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) was used as the reference standard. A natural language processing engine, easy clinical information extractor-pulmonary embolism/deep vein thrombosis (EasyCIE-PEDVT), was trained to detect pulmonary embolism and deep vein thrombosis from clinical notes. International Classification of Diseases (ICD) discharge diagnosis codes for venous thromboembolism were used as baseline comparators. The classification performance of EasyCIE-PEDVT was compared with International Classification of Diseases codes using sensitivity, specificity, area under the receiver operating characteristic curve, using an internal and external validation cohort. RESULTS: To detect pulmonary embolism, EasyCIE-PEDVT had a sensitivity of 0.714 and 0.815 in internal and external validation, respectively. To detect deep vein thrombosis, EasyCIE-PEDVT had a sensitivity of 0.846 and 0.849 in internal and external validation, respectively. EasyCIE-PEDVT had significantly higher discrimination for deep vein thrombosis compared with International Classification of Diseases codes in internal validation (area under the receiver operating characteristic curve: 0.920 vs 0.761; P < .001) and external validation (area under the receiver operating characteristic curve: 0.921 vs 0.794; P < .001). There was no significant difference in the discrimination for pulmonary embolism between EasyCIE-PEDVT and ICD codes. CONCLUSION: Accurate surveillance of postoperative venous thromboembolism may be achieved using natural language processing on clinical notes in 2 independent health care systems. These findings suggest natural language processing may augment manual chart abstraction for large registries such as NSQIP.

Portable Automated Surveillance of Surgical Site Infections Using Natural Language Processing: Development and Validation.

OBJECTIVES: We present the development and validation of a portable NLP approach for automated surveillance of SSIs. SUMMARY OF BACKGROUND DATA: The surveillance of SSIs is labor-intensive limiting the generalizability and scalability of surgical quality surveillance programs. METHODS: We abstracted patient clinical text notes after surgical procedures from 2 independent healthcare systems using different electronic healthcare records. An SSI detected as part of the American College of Surgeons' National Surgical Quality Improvement Program was used as the reference standard. We developed a rules-based NLP system (Easy Clinical Information Extractor [CIE]-SSI) for operative event-level detection of SSIs using an training cohort (4574 operative events) from 1 healthcare system and then conducted internal validation on a blind cohort from the same healthcare system (1850 operative events) and external validation on a blind cohort from the second healthcare system (15,360 operative events). EasyCIE-SSI performance was measured using sensitivity, specificity, and area under the receiver-operating-curve (AUC). RESULTS: The prevalence of SSI was 4% and 5% in the internal and external validation corpora. In internal validation, EasyCIE-SSI had a sensitivity, specificity, AUC of 94%, 88%, 0.912 for the detection of SSI, respectively. In external validation, EasyCIE-SSI had sensitivity, specificity, AUC of 79%, 92%, 0.852 for the detection of SSI, respectively. The sensitivity of EasyCIE-SSI decreased in clean, skin/subcutaneous, and outpatient procedures in the external validation compared to internal validation. CONCLUSION: Automated surveillance of SSIs can be achieved using NLP of clinical notes with high sensitivity and specificity.

A two-site survey of medical center personnel's willingness to share clinical data for research: implications for reproducible health NLP research.

BACKGROUND: A shareable repository of clinical notes is critical for advancing natural language processing (NLP) research, and therefore a goal of many NLP researchers is to create a shareable repository of clinical notes, that has breadth (from multiple institutions) as well as depth (as much individual data as possible). METHODS: We aimed to assess the degree to which individuals would be willing to contribute their health data to such a repository. A compact e-survey probed willingness to share demographic and clinical data categories. Participants were faculty, staff, and students in two geographically diverse major medical centers (Utah and New York). Such a sample could be expected to respond like a typical potential participant from the general public who is given complete and fully informed consent about the pros and cons of participating in a research study. RESULTS: Two thousand one hundred forty respondents completed the surveys. 56% of respondents were "somewhat/definitely willing" to share clinical data with identifiers, while 89% of respondents were "somewhat (17%)/definitely willing (72%)" to share without identifiers. Results were consistent across gender, age, and education, but there were some differences by geographical region. Individuals were most reluctant (50-74%) sharing mental health, substance abuse, and domestic violence data. CONCLUSIONS: We conclude that a substantial fraction of potential patient participants, once educated about risks and benefits, would be willing to donate de-identified clinical data to a shared research repository. A slight majority even would be willing to share absent de-identification, suggesting that perceptions about data misuse are not a major concern. Such a repository of clinical notes should be invaluable for clinical NLP research and advancement.

The Grocery Purchase Quality Index-2016 Performs Similarly to the Healthy Eating Index-2015 in a National Survey of Household Food Purchases.

BACKGROUND: Household food purchases are potential indicators of the quality of the home food environment, and grocery purchase behavior is a main focus of US Department of Agriculture (USDA) nutrition education programs; therefore, objective measures of grocery purchases are needed. OBJECTIVE: The objective of the study was to evaluate the Grocery Purchase Quality Index-2016 (GPQI-2016) as a tool for assessing grocery food purchase quality by using the Healthy Eating Index-2015 (HEI-2015) as the reference standard. DESIGN: In 2012, the USDA Economic Research Service conducted the National Household Food Acquisition and Purchase Survey. Members of participating households recorded all foods acquired for a week. Foods purchased at stores were mapped to the 29 food categories used in USDA Food Plans, expenditure shares were estimated, and GPQI-2016 scores were calculated. USDA food codes, provided in the survey database, were used to calculate the HEI-2015. PARTICIPANTS/SETTING: All households in the 48 coterminous states were eligible for the survey. The analytic sample size was 4,276 households. MAIN OUTCOME MEASURES: GPQI-2016 and HEI-2015 scores were compared. STATISTICAL ANALYSES PERFORMED: Correlation of scores was assessed using Spearman's correlation coefficient. Linear regression models with fixed effects were used to determine differences among various subgroups of households. RESULTS: The correlation coefficient for the total GPQI-2016 score and the total HEI-2015 score was 0.70. For the component scores, the strongest correlations were for Total and Whole Fruit (0.89 to 0.90); the weakest were for Dairy (0.67), Refined Grains (0.66), and Sweets and Sodas/Added Sugars (0.65) (all, P<0.01). Both the GPQI-2016 and HEI-2015 were significantly different among subgroups in expected directions. CONCLUSIONS: Overall, the GPQI-2016, estimated from a national survey of households, performed similarly to the HEI-2015. The tool has potential for evaluating nutrition education programs and retail-oriented interventions when the nutrient content and gram weights of foods purchased are not available.

Trie-based rule processing for clinical NLP: A use-case study of n-trie, making the ConText algorithm more efficient and scalable.

OBJECTIVE: To develop and evaluate an efficient Trie structure for large-scale, rule-based clinical natural language processing (NLP), which we call n-trie. BACKGROUND: Despite the popularity of machine learning techniques in natural language processing, rule-based systems boast important advantages: distinctive transparency, ease of incorporating external knowledge, and less demanding annotation requirements. However, processing efficiency remains a major obstacle for adopting standard rule-base NLP solutions in big data analyses. METHODS: We developed n-trie to specifically address the token-based nature of context detection, an important facet of clinical NLP that is known to slow down NLP pipelines. N-trie, a new rule processing engine using a revised Trie structure, allows fast execution of lexicon-based NLP rules. To determine its applicability and evaluate its performance, we applied the n-trie engine in an implementation (called FastContext) of the ConText algorithm and compared its processing speed and accuracy with JavaConText and GeneralConText, two widely used Java ConText implementations, as well as with a standalone machine learning NegEx implementation, NegScope. RESULTS: The n-trie engine ran two orders of magnitude faster and was far less sensitive to rule set size than the comparison implementations, and it proved faster than the best machine learning negation detector. Additionally, the engine consistently gained accuracy improvement as the rule set increased (the desired outcome of adding new rules), while the other implementations did not. CONCLUSIONS: The n-trie engine is an efficient, scalable engine to support NLP rule processing and shows the potential for application in other NLP tasks beyond context detection.

An Innovative Method for Monitoring Food Quality and the Healthfulness of Consumers' Grocery Purchases.

This study presents a method laying the groundwork for systematically monitoring food quality and the healthfulness of consumers' point-of-sale grocery purchases. The method automates the process of identifying United States Department of Agriculture (USDA) Food Patterns Equivalent Database (FPED) components of grocery food items. The input to the process is the compact abbreviated descriptions of food items that are similar to those appearing on the point-of-sale sales receipts of most food retailers. The FPED components of grocery food items are identified using Natural Language Processing techniques combined with a collection of food concept maps and relationships that are manually built using the USDA Food and Nutrient Database for Dietary Studies, the USDA National Nutrient Database for Standard Reference, the What We Eat In America food categories, and the hierarchical organization of food items used by many grocery stores. We have established the construct validity of the method using data from the National Health and Nutrition Examination Survey, but further evaluation of validity and reliability will require a large-scale reference standard with known grocery food quality measures. Here we evaluate the method's utility in identifying the FPED components of grocery food items available in a large sample of retail grocery sales data (~190 million transaction records).

Extractive text summarization system to aid data extraction from full text in systematic review development.

OBJECTIVES: Extracting data from publication reports is a standard process in systematic review (SR) development. However, the data extraction process still relies too much on manual effort which is slow, costly, and subject to human error. In this study, we developed a text summarization system aimed at enhancing productivity and reducing errors in the traditional data extraction process. METHODS: We developed a computer system that used machine learning and natural language processing approaches to automatically generate summaries of full-text scientific publications. The summaries at the sentence and fragment levels were evaluated in finding common clinical SR data elements such as sample size, group size, and PICO values. We compared the computer-generated summaries with human written summaries (title and abstract) in terms of the presence of necessary information for the data extraction as presented in the Cochrane review's study characteristics tables. RESULTS: At the sentence level, the computer-generated summaries covered more information than humans do for systematic reviews (recall 91.2% vs. 83.8%, p<0.001). They also had a better density of relevant sentences (precision 59% vs. 39%, p<0.001). At the fragment level, the ensemble approach combining rule-based, concept mapping, and dictionary-based methods performed better than individual methods alone, achieving an 84.7% F-measure. CONCLUSION: Computer-generated summaries are potential alternative information sources for data extraction in systematic review development. Machine learning and natural language processing are promising approaches to the development of such an extractive summarization system.

An Analysis of Information Technology Adoption by IRBs of Large Academic Medical Centers in the United States.

The clinical research landscape has changed dramatically in recent years in terms of both volume and complexity. This poses new challenges for Institutional Review Boards' (IRBs) review efficiency and quality, especially at large academic medical centers. This article discusses the technical facets of IRB modernization. We analyzed the information technology used by IRBs in large academic institutions across the United States. We found that large academic medical centers have a high electronic IRB adoption rate; however, the capabilities of electronic IRB systems vary greatly. We discuss potential use-cases of a fully exploited electronic IRB system that promise to streamline the clinical research work flow. The key to that approach utilizes a structured and standardized information model for the IRB application.

Towards Measuring the Food Quality of Grocery Purchases: an Estimation Model of the Healthy Eating Index-2010 Using only Food Item Counts.

Measuring the quality of food consumed by individuals or groups in the U.S. is essential to informed public health surveillance efforts and sound nutrition policymaking. For example, the Healthy Eating Index-2010 (HEI) is an ideal metric to assess the food quality of households, but the traditional methods of collecting the data required to calculate the HEI are expensive and burdensome. We evaluated an alternative source: rather than measuring the quality of the foods consumers eat, we want to estimate the quality of the foods consumers buy. To accomplish that we need a way of estimating the HEI based solely on the count of food items. We developed an estimation model of the HEI, using an augmented set of the What We Eat In America (WWEIA) food categories. Then we mapped ~92,000 grocery food items to it. The model uses an inverse Cumulative Distribution Function sampling technique. Here we describe the model and report reliability metrics based on NHANES data from 2003-2010.

POETenceph - Automatic identification of clinical notes indicating encephalopathy using a realist ontology.

Identifying inpatients with encephalopathy is important. The disorder is prevalent, often missed, and puts patients at risk. We describe POETenceph, natural language processing pipeline, which ranks clinical notes on the extent to which they indicate the patient had encephalopathy. We use a realist ontology of the entities and relationships indicative of encephalopathy in clinical notes. POETenceph includes a passage rank algorithm, which takes identified disorders; matches them to the ontology; calculates the diffuseness, centrality, and length of the matched entry; adds the scores; and returns the ranked documents. We evaluate it against a corpus of clinical documents annotated for evidence of delirium. Higher POETenceph are associated with increasing numbers of reviewer annotations. Detailed examination found that 65% of the bottom scoring documents contained little or no evidence and 70% of the top contained good evidence. POETenceph can effectively rank clinical documents for their evidence of encephalopathy as characterized by delirium.

A Study of Concept Extraction Across Different Types of Clinical Notes.

Our research investigates methods for creating effective concept extractors for specialty clinical notes. First, we present three new "specialty area" datasets consisting of Cardiology, Neurology, and Orthopedics clinical notes manually annotated with medical concepts. We analyze the medical concepts in each dataset and compare with the widely used i2b2 2010 corpus. Second, we create several types of concept extraction models and examine the effects of training supervised learners with specialty area data versus i2b2 data. We find substantial differences in performance across the datasets, and obtain the best results for all three specialty areas by training with both i2b2 and specialty data. Third, we explore strategies to improve concept extraction on specialty notes with ensemble methods. We compare two types of ensemble methods (Voting/Stacking) and a domain adaptation model, and show that a Stacked ensemble of classifiers trained with i2b2 and specialty data yields the best performance.

Nutrition Informatics Applications in Clinical Practice: a Systematic Review.

Nutrition care and metabolic control contribute to clinical patient outcomes. Biomedical informatics applications represent a way to potentially improve quality and efficiency of nutrition management. We performed a systematic literature review to identify clinical decision support and computerized provider order entry systems used to manage nutrition care. Online research databases were searched using a specific set of keywords. Additionally, bibliographies were referenced for supplemental citations. Four independent reviewers selected sixteen studies out of 364 for review. These papers described adult and neonatal nutrition support applications, blood glucose management applications, and other nutrition applications. Overall, results indicated that computerized interventions could contribute to improved patient outcomes and provider performance. Specifically, computer systems in the clinical setting improved nutrient delivery, rates of malnutrition, weight loss, blood glucose values, clinician efficiency, and error rates. In conclusion, further investigation of informatics applications on nutritional and performance outcomes utilizing rigorous study designs is recommended.

A domain analysis model for eIRB systems: addressing the weak link in clinical research informatics.

Institutional Review Boards (IRBs) are a critical component of clinical research and can become a significant bottleneck due to the dramatic increase, in both volume and complexity of clinical research. Despite the interest in developing clinical research informatics (CRI) systems and supporting data standards to increase clinical research efficiency and interoperability, informatics research in the IRB domain has not attracted much attention in the scientific community. The lack of standardized and structured application forms across different IRBs causes inefficient and inconsistent proposal reviews and cumbersome workflows. These issues are even more prominent in multi-institutional clinical research that is rapidly becoming the norm. This paper proposes and evaluates a domain analysis model for electronic IRB (eIRB) systems, paving the way for streamlined clinical research workflow via integration with other CRI systems and improved IRB application throughput via computer-assisted decision support.

Development of a HIPAA-compliant environment for translational research data and analytics.

High-performance computing centers (HPC) traditionally have far less restrictive privacy management policies than those encountered in healthcare. We show how an HPC can be re-engineered to accommodate clinical data while retaining its utility in computationally intensive tasks such as data mining, machine learning, and statistics. We also discuss deploying protected virtual machines. A critical planning step was to engage the university's information security operations and the information security and privacy office. Access to the environment requires a double authentication mechanism. The first level of authentication requires access to the university's virtual private network and the second requires that the users be listed in the HPC network information service directory. The physical hardware resides in a data center with controlled room access. All employees of the HPC and its users take the university's local Health Insurance Portability and Accountability Act training series. In the first 3 years, researcher count has increased from 6 to 58.

Proposal for a data publication and citation framework when sharing biomedical research resources.

Research data and biospecimen repositories are valuable resources for biomedical investigators. Sharing these resources has great potential benefits including efficient use of resources, avoiding duplicate experiments, gathering adequate sample sizes, and promoting collaboration. However, concerns from data producers about difficulties of getting proper acknowledgement for their data contributions are increasingly becoming obstacles for efficient and large-scale data sharing in reality. In this research project we analyzed the inadequacy of current policy-based solution for promoting data sharing. The recommendations in this paper emphasize data publication and citation. This project aims to promote the acknowledgement of data contributors with realizable informatics tools that augment informal policy-level strategies, and do so in a way that promotes data sharing.

Identifying clinical/translational research cohorts: ascertainment via querying an integrated multi-source database.

BACKGROUND: Ascertainment of potential subjects has been a longstanding problem in clinical research. Various methods have been proposed, including using data in electronic health records. However, these methods typically suffer from scaling effects-some methods work well for large cohorts; others work for small cohorts only. OBJECTIVE: We propose a method that provides a simple identification of pre-research cohorts and relies on data available in most states in the USA: merged public health data sources. MATERIALS AND METHODS: The Utah Population Database Limited query tool allows users to build complex queries that may span several types of health records, such as cancer registries, inpatient hospital discharges, and death certificates; in addition, these can be combined with family history information. The architectural approach incorporates several coding systems for medical information. It provides a front-end graphical user interface and enables researchers to build and run queries and view aggregate results. Multiple strategies have been incorporated to maintain confidentiality. RESULTS: This tool was rapidly adopted; since its release, 241 users representing a wide range of disciplines from 17 institutions have signed the user agreement and used the query tool. Three examples are discussed: pregnancy complications co-occurring with cardiovascular disease; spondyloarthritis; and breast cancer. DISCUSSION AND CONCLUSIONS: This query tool was designed to provide results as pre-research so that institutional review board approval would not be required. This architecture uses well-described technologies that should be within the reach of most institutions.

qDIET: toward an automated, self-sustaining knowledge base to facilitate linking point-of-sale grocery items to nutritional content.

The United States, indeed the world, struggles with a serious obesity epidemic. The costs of this epidemic in terms of healthcare dollar expenditures and human morbidity/mortality are staggering. Surprisingly, clinicians are ill-equipped in general to advise patients on effective, longitudinal weight loss strategies. We argue that one factor hindering clinicians and patients in effective shared decision-making about weight loss is the absence of a metric that can be reasoned about and monitored over time, as clinicians do routinely with, say, serum lipid levels or HgA1C. We propose that a dietary quality measure championed by the USDA and NCI, the HEI-2005/2010, is an ideal metric for this purpose. We describe a new tool, the quality Dietary Information Extraction Tool (qDIET), which is a step toward an automated, self-sustaining process that can link retail grocery purchase data to the appropriate USDA databases to permit the calculation of the HEI-2005/2010.

Text summarization as a decision support aid.

BACKGROUND: PubMed data potentially can provide decision support information, but PubMed was not exclusively designed to be a point-of-care tool. Natural language processing applications that summarize PubMed citations hold promise for extracting decision support information. The objective of this study was to evaluate the efficiency of a text summarization application called Semantic MEDLINE, enhanced with a novel dynamic summarization method, in identifying decision support data. METHODS: We downloaded PubMed citations addressing the prevention and drug treatment of four disease topics. We then processed the citations with Semantic MEDLINE, enhanced with the dynamic summarization method. We also processed the citations with a conventional summarization method, as well as with a baseline procedure. We evaluated the results using clinician-vetted reference standards built from recommendations in a commercial decision support product, DynaMed. RESULTS: For the drug treatment data, Semantic MEDLINE enhanced with dynamic summarization achieved average recall and precision scores of 0.848 and 0.377, while conventional summarization produced 0.583 average recall and 0.712 average precision, and the baseline method yielded average recall and precision values of 0.252 and 0.277. For the prevention data, Semantic MEDLINE enhanced with dynamic summarization achieved average recall and precision scores of 0.655 and 0.329. The baseline technique resulted in recall and precision scores of 0.269 and 0.247. No conventional Semantic MEDLINE method accommodating summarization for prevention exists. CONCLUSION: Semantic MEDLINE with dynamic summarization outperformed conventional summarization in terms of recall, and outperformed the baseline method in both recall and precision. This new approach to text summarization demonstrates potential in identifying decision support data for multiple needs.