Insights into the nutritional prevention of macular degeneration based on a comparative topic modeling approach.

Topic modeling and text mining are subsets of natural language processing (NLP) with relevance for conducting meta-analysis (MA) and systematic review (SR). For evidence synthesis, the above NLP methods are conventionally used for topic-specific literature searches or extracting values from reports to automate essential phases of SR and MA. Instead, this work proposes a comparative topic modeling approach to analyze reports of contradictory results on the same general research question. Specifically, the objective is to identify topics exhibiting distinct associations with significant results for an outcome of interest by ranking them according to their proportional occurrence in (and consistency of distribution across) reports of significant effects. Macular degeneration (MD) is a disease that affects millions of people annually, causing vision loss. Augmenting evidence synthesis to provide insight into MD prevention is therefore of central interest in this article. The proposed method was tested on broad-scope studies addressing whether supplemental nutritional compounds significantly benefit macular degeneration. Six compounds were identified as having a particular association with reports of significant results for benefiting MD. Four of these were further supported in terms of effectiveness upon conducting a follow-up literature search for validation (omega-3 fatty acids, copper, zeaxanthin, and nitrates). The two not supported by the follow-up literature search (niacin and molybdenum) also had scores in the lowest range under the proposed scoring system. Results therefore suggest that the proposed method's score for a given topic may be a viable proxy for its degree of association with the outcome of interest, and can be helpful in the systematic search for potentially causal relationships. Further, the compounds identified by the proposed method were not simultaneously captured as salient topics by state-of-the-art topic models that leverage document and word embeddings (Top2Vec) and transformer models (BERTopic). These results underpin the proposed method's potential to add specificity in understanding effects from broad-scope reports, elucidate topics of interest for future research, and guide evidence synthesis in a scalable way. All of this is accomplished while yielding valuable and actionable insights into the prevention of MD.

Unlocking human-like conversations: Scoping review of automation techniques for personalized healthcare interventions using conversational agents.

BACKGROUND: Conversational agents (CAs) offer a sustainable approach to deliver personalized interventions and improve health outcomes. OBJECTIVES: To review how human-like communication and automation techniques of CAs in personalized healthcare interventions have been implemented. It is intended for designers and developers, computational scientists, behavior scientists, and biomedical engineers who aim at developing CAs for healthcare interventions. METHODOLOGY: A scoping review was conducted in accordance with PRISMA Extension for Scoping Review. A search was performed in May 2023 in Web of Science, Pubmed, Scopus and IEEE databases. Search results were extracted, duplicates removed, and the remaining results were screened. Studies that contained personalized and automated CAs within the healthcare domain were included. Information regarding study characterization, and human-like communication and automation techniques was extracted from articles that met the eligibility criteria. RESULTS: Twenty-three studies were selected. These articles described the development of CAs designed for patients to either self-manage their diseases (such as diabetes, mental health issues, cancer, asthma, COVID-19, and other chronic conditions) or to enhance healthy habits. The human-like communication characteristics studied encompassed aspects like system flexibility, personalization, and affective characteristics. Seven studies used rule-based models, eleven applied retrieval-based techniques for content delivery, five used AI models, and six integrated affective computing. CONCLUSIONS: The increasing interest in employing CAs for personalized healthcare interventions is noteworthy. The adaptability of dialogue structures and personalization features is still limited. Unlocking human-like conversations may encompass the use of affective computing and generative AI to help improve user engagement. Future research should focus on the integration of holistic methods to describe the end-user, and the safe use of generative models.

Consore: A Powerful Federated Data Mining Tool Driving a French Research Network to Accelerate Cancer Research.

BACKGROUND: Real-world data (RWD) related to the health status and care of cancer patients reflect the ongoing medical practice, and their analysis yields essential real-world evidence. Advanced information technologies are vital for their collection, qualification, and reuse in research projects. METHODS: UNICANCER, the French federation of comprehensive cancer centres, has innovated a unique research network: Consore. This potent federated tool enables the analysis of data from millions of cancer patients across eleven French hospitals. RESULTS: Currently operational within eleven French cancer centres, Consore employs natural language processing to structure the therapeutic management data of approximately 1.3 million cancer patients. These data originate from their electronic medical records, encompassing about 65 million medical records. Thanks to the structured data, which are harmonized within a common data model, and its federated search tool, Consore can create patient cohorts based on patient or tumor characteristics, and treatment modalities. This ability to derive larger cohorts is particularly attractive when studying rare cancers. CONCLUSIONS: Consore serves as a tremendous data mining instrument that propels French cancer centres into the big data era. With its federated technical architecture and unique shared data model, Consore facilitates compliance with regulations and acceleration of cancer research projects.

Enhancing traditional Chinese medicine diagnostics: Integrating ontological knowledge for multi-label symptom entity classification.

In traditional Chinese medicine (TCM), artificial intelligence (AI)-assisted syndrome differentiation and disease diagnoses primarily confront the challenges of accurate symptom identification and classification. This study introduces a multi-label entity extraction model grounded in TCM symptom ontology, specifically designed to address the limitations of existing entity recognition models characterized by limited label spaces and an insufficient integration of domain knowledge. This model synergizes a knowledge graph with the TCM symptom ontology framework to facilitate a standardized symptom classification system and enrich it with domain-specific knowledge. It innovatively merges the conventional bidirectional encoder representations from transformers (BERT) + bidirectional long short-term memory (Bi-LSTM) + conditional random fields (CRF) entity recognition methodology with a multi-label classification strategy, thereby adeptly navigating the intricate label interdependencies in the textual data. Introducing a multi-associative feature fusion module is a significant advancement, thereby enabling the extraction of pivotal entity features while discerning the interrelations among diverse categorical labels. The experimental outcomes affirm the model's superior performance in multi-label symptom extraction and substantially elevates the efficiency and accuracy. This advancement robustly underpins research in TCM syndrome differentiation and disease diagnoses.

Classification method of traditional Chinese medicine compound decoction duration based on multi-dimensional feature weighted fusion.

This paper extends a text classification method utilizing natural language processing (NLP) into the field of traditional Chinese medicine (TCM) compound decoction to effectively and scientifically extend the TCM compound decoction duration. Specifically, a TCM compound decoction duration classification named TCM-TextCNN is proposed to fuse multi-dimensional herb features and improve TextCNN. Indeed, first, we utilize word vector technology to construct feature vectors of herb names and medicinal parts, aiming to describe the herb characteristics comprehensively. Second, considering the impact of different herb features on the decoction duration, we use an improved Term Frequency-Inverse Word Frequency (TF-IWF) algorithm to weigh the feature vectors of herb names and medicinal parts. These weighted feature vectors are then concatenated to obtain a multi-dimensional herb feature vector, allowing for a more comprehensive representation. Finally, the feature vector is input into the improved TextCNN, which uses k-max pooling to reduce information loss rather than max pooling. Three fully connected layers are added to generate higher-level feature representations, followed by softmax to obtain the final results. Experimental results on a dataset of TCM compound decoction duration demonstrate that TCM-TextCNN improves accuracy, recall, and F1 score by 5.31%, 5.63%, and 5.22%, respectively, compared to methods solely rely on herb name features, thereby confirming our method's effectiveness in classifying TCM compound decoction duration.

Revolutionizing Breast Cancer Care: AI-Enhanced Diagnosis and Patient History.

Breast cancer poses a significant global health challenge, demanding enhanced diagnostic accuracy and streamlined medical history documentation. This study presents a holistic approach that harnesses the power of artificial intelligence (AI) and machine learning (ML) to address these pressing needs. This study presents a comprehensive methodology for breast cancer diagnosis and medical history generation, integrating data collection, feature extraction, machine learning, and AI-driven history-taking. The research employs a systematic approach to ensure accurate diagnosis and efficient history collection. Data preprocessing merges similar attributes to streamline analysis. Three key algorithms, Support Vector Machine (SVM), K-Nearest Neighbours (KNN), and Fuzzy Logic, are applied. Fuzzy Logic shows exceptional accuracy in handling uncertain data. Deep learning models enhance predictive accuracy, emphasizing the synergy between traditional and deep learning approaches. The AI-driven history collection simplifies the patient history-taking process, adapting questions dynamically based on patient responses. Comprehensive medical history reports summarize patient data, facilitating informed healthcare decisions. The research prioritizes ethical compliance and data privacy. OpenAI has integrated GPT-3.5 to generate automated patient reports, offering structured overviews of patient health history. The study's results indicate the potential for enhanced disease prediction accuracy and streamlined medical history collection, contributing to more reliable healthcare assessments and patient care. Machine learning, deep learning, and AI-driven approaches hold promise for a wide range of applications, particularly in healthcare and beyond.

O2 supplementation disambiguation in clinical narratives to support retrospective COVID-19 studies.

BACKGROUND: Oxygen saturation, a key indicator of COVID-19 severity, poses challenges, especially in cases of silent hypoxemia. Electronic health records (EHRs) often contain supplemental oxygen information within clinical narratives. Streamlining patient identification based on oxygen levels is crucial for COVID-19 research, underscoring the need for automated classifiers in discharge summaries to ease the manual review burden on physicians. METHOD: We analysed text lines extracted from anonymised COVID-19 patient discharge summaries in German to perform a binary classification task, differentiating patients who received oxygen supplementation and those who did not. Various machine learning (ML) algorithms, including classical ML to deep learning (DL) models, were compared. Classifier decisions were explained using Local Interpretable Model-agnostic Explanations (LIME), which visualize the model decisions. RESULT: Classical ML to DL models achieved comparable performance in classification, with an F-measure varying between 0.942 and 0.955, whereas the classical ML approaches were faster. Visualisation of embedding representation of input data reveals notable variations in the encoding patterns between classic and DL encoders. Furthermore, LIME explanations provide insights into the most relevant features at token level that contribute to these observed differences. CONCLUSION: Despite a general tendency towards deep learning, these use cases show that classical approaches yield comparable results at lower computational cost. Model prediction explanations using LIME in textual and visual layouts provided a qualitative explanation for the model performance.

Assessing ChatGPT's Mastery of Bloom's Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study.

BACKGROUND: Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to "hallucinations" (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom's taxonomy. OBJECTIVE: This study aims to explore how GPT-4 performs in terms of Bloom's taxonomy using psychosomatic medicine exam questions. METHODS: We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom's taxonomy. RESULTS: GPT-4's performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4's lowest exam performance was 78.9% (15/19), thereby always surpassing the "pass" threshold. Our qualitative analysis of incorrect answers, based on Bloom's taxonomy, showed that errors were primarily in the "remember" (29/68) and "understand" (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines. CONCLUSIONS: GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom's taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.

Identification of recurrent atrial fibrillation using natural language processing applied to electronic health records.

AIMS: This study aimed to develop and apply natural language processing (NLP) algorithms to identify recurrent atrial fibrillation (AF) episodes following rhythm control therapy initiation using electronic health records (EHRs). METHODS AND RESULTS: We included adults with new-onset AF who initiated rhythm control therapies (ablation, cardioversion, or antiarrhythmic medication) within two US integrated healthcare delivery systems. A code-based algorithm identified potential AF recurrence using diagnosis and procedure codes. An automated NLP algorithm was developed and validated to capture AF recurrence from electrocardiograms, cardiac monitor reports, and clinical notes. Compared with the reference standard cases confirmed by physicians' adjudication, the F-scores, sensitivity, and specificity were all above 0.90 for the NLP algorithms at both sites. We applied the NLP and code-based algorithms to patients with incident AF (n = 22 970) during the 12 months after initiating rhythm control therapy. Applying the NLP algorithms, the percentages of patients with AF recurrence for sites 1 and 2 were 60.7% and 69.9% (ablation), 64.5% and 73.7% (cardioversion), and 49.6% and 55.5% (antiarrhythmic medication), respectively. In comparison, the percentages of patients with code-identified AF recurrence for sites 1 and 2 were 20.2% and 23.7% for ablation, 25.6% and 28.4% for cardioversion, and 20.0% and 27.5% for antiarrhythmic medication, respectively. CONCLUSION: When compared with a code-based approach alone, this study's high-performing automated NLP method identified significantly more patients with recurrent AF. The NLP algorithms could enable efficient evaluation of treatment effectiveness of AF therapies in large populations and help develop tailored interventions.

Deploying a national clinical text processing infrastructure.

OBJECTIVES: Clinical text processing offers a promising avenue for improving multiple aspects of healthcare, though operational deployment remains a substantial challenge. This case report details the implementation of a national clinical text processing infrastructure within the Department of Veterans Affairs (VA). METHODS: Two foundational use cases, cancer case management and suicide and overdose prevention, illustrate how text processing can be practically implemented at scale for diverse clinical applications using shared services. RESULTS: Insights from these use cases underline both commonalities and differences, providing a replicable model for future text processing applications. CONCLUSIONS: This project enables more efficient initiation, testing, and future deployment of text processing models, streamlining the integration of these use cases into healthcare operations. This project implementation is in a large integrated health delivery system in the United States, but we expect the lessons learned to be relevant to any health system, including smaller local and regional health systems in the United States.

Evaluation of the Needs and Experiences of Patients with Hypertriglyceridemia: Social Media Listening Infosurveillance Study.

BACKGROUND: Hypertriglyceridemia is a risk factor for cardiovascular diseases. Internet usage in China is increasing, giving rise to large-scale data sources, especially to access, disseminate, and discuss medical information. Social media listening (SML) is a new approach to analyze and monitor online discussions related to various health-related topics in diverse diseases, which can generate insights into users' experiences and expectations. However, to date, no studies have evaluated the utility of SML to understand patients' cognizance and expectations pertaining to the management of hypertriglyceridemia. OBJECTIVE: The aim of this study was to utilize SML to explore the disease cognition level of patients with hypertriglyceridemia, choice of intervention measures, and the status quo of online consultations and question-and-answer (Q&A) search platforms. METHODS: An infosurveillance study was conducted wherein a disease-specific comprehensive search was performed between 2004 and 2020 in Q&A search and online consultation platforms. Predefined single and combined keywords related to hypertriglyceridemia were used in the search, including disease, symptoms, diagnosis, and treatment indicators; lifestyle interventions; and therapeutic agents. The search output was aggregated using an aggregator tool and evaluated. RESULTS: Disease-specific consultation data (n=69,845) and corresponding response data (n=111,763) were analyzed from 20 data sources (6 Q&A search platforms and 14 online consultation platforms). Doctors from inland areas had relatively high voice volumes and appear to exert a substantial influence on these platforms. Patients with hypertriglyceridemia engaging on the internet have an average level of cognition about the disease and its intervention measures. However, a strong demand for the concept of the disease and "how to treat it" was observed. More emphasis on the persistence of the disease and the safety of medications was observed. Young patients have a lower willingness for drug interventions, whereas patients with severe hypertriglyceridemia have a clearer intention to use drug intervention and few patients have a strong willingness for the use of traditional Chinese medicine. CONCLUSIONS: Findings from this disease-specific SML study revealed that patients with hypertriglyceridemia in China actively seek information from both online Q&A search and consultation platforms. However, the integrity of internet doctors' suggestions on lifestyle interventions and the accuracy of drug intervention recommendations still need to be improved. Further, a combined prospective qualitative study with SML is required for added rigor and confirmation of the relevance of the findings.

Application of natural language processing to identify social needs from patient medical notes: development and assessment of a scalable, performant, and rule-based model in an integrated healthcare delivery system.

Objectives: To develop and test a scalable, performant, and rule-based model for identifying 3 major domains of social needs (residential instability, food insecurity, and transportation issues) from the unstructured data in electronic health records (EHRs). Materials and Methods: We included patients aged 18 years or older who received care at the Johns Hopkins Health System (JHHS) between July 2016 and June 2021 and had at least 1 unstructured (free-text) note in their EHR during the study period. We used a combination of manual lexicon curation and semiautomated lexicon creation for feature development. We developed an initial rules-based pipeline (Match Pipeline) using 2 keyword sets for each social needs domain. We performed rule-based keyword matching for distinct lexicons and tested the algorithm using an annotated dataset comprising 192 patients. Starting with a set of expert-identified keywords, we tested the adjustments by evaluating false positives and negatives identified in the labeled dataset. We assessed the performance of the algorithm using measures of precision, recall, and F1 score. Results: The algorithm for identifying residential instability had the best overall performance, with a weighted average for precision, recall, and F1 score of 0.92, 0.84, and 0.92 for identifying patients with homelessness and 0.84, 0.82, and 0.79 for identifying patients with housing insecurity. Metrics for the food insecurity algorithm were high but the transportation issues algorithm was the lowest overall performing metric. Discussion: The NLP algorithm in identifying social needs at JHHS performed relatively well and would provide the opportunity for implementation in a healthcare system. Conclusion: The NLP approach developed in this project could be adapted and potentially operationalized in the routine data processes of a healthcare system.

Exploring Political Mistrust in Pandemic Risk Communication: Mixed-Method Study Using Social Media Data Analysis.

BACKGROUND: This research extends prior studies by the Finnish Institute for Health and Welfare on pandemic-related risk perception, concentrating on the role of trust in health authorities and its impact on public health outcomes. OBJECTIVE: The paper aims to investigate variations in trust levels over time and across social media platforms, as well as to further explore 12 subcategories of political mistrust. It seeks to understand the dynamics of political trust, including mistrust accumulation, fluctuations over time, and changes in topic relevance. Additionally, the study aims to compare qualitative research findings with those obtained through computational methods. METHODS: Data were gathered from a large-scale data set consisting of 13,629 Twitter and Facebook posts from 2020 to 2023 related to COVID-19. For analysis, a fine-tuned FinBERT model with an 80% accuracy rate was used for predicting political mistrust. The BERTopic model was also used for superior topic modeling performance. RESULTS: Our preliminary analysis identifies 43 mistrust-related topics categorized into 9 major themes. The most salient topics include COVID-19 mortality, coping strategies, polymerase chain reaction testing, and vaccine efficacy. Discourse related to mistrust in authority is associated with perceptions of disease severity, willingness to adopt health measures, and information-seeking behavior. Our findings highlight that the distinct user engagement mechanisms and platform features of Facebook and Twitter contributed to varying patterns of mistrust and susceptibility to misinformation during the pandemic. CONCLUSIONS: The study highlights the effectiveness of computational methods like natural language processing in managing large-scale engagement and misinformation. It underscores the critical role of trust in health authorities for effective risk communication and public compliance. The findings also emphasize the necessity for transparent communication from authorities, concluding that a holistic approach to public health communication is integral for managing health crises effectively.

Using natural language processing to characterize and predict homeopathic product-associated adverse events in consumer reviews: comparison to reports to FDA Adverse Event Reporting System (FAERS).

OBJECTIVE: Apply natural language processing (NLP) to Amazon consumer reviews to identify adverse events (AEs) associated with unapproved over the counter (OTC) homeopathic drugs and compare findings with reports to the US Food and Drug Administration Adverse Event Reporting System (FAERS). MATERIALS AND METHODS: Data were extracted from publicly available Amazon reviews and analyzed using JMP 16 Pro Text Explorer. Topic modeling identified themes. Sentiment analysis (SA) explored consumer perceptions. A machine learning model optimized prediction of AEs in reviews. Reports for the same time interval and product class were obtained from the FAERS public dashboard and analyzed. RESULTS: Homeopathic cough/cold products were the largest category common to both data sources (Amazon = 616, FAERS = 445) and were analyzed further. Oral symptoms and unpleasant taste were described in both datasets. Amazon reviews describing an AE had lower Amazon ratings (X2 = 224.28, P < .0001). The optimal model for predicting AEs was Neural Boosted 5-fold combining topic modeling and Amazon ratings as predictors (mean AUC = 0.927). DISCUSSION: Topic modeling and SA of Amazon reviews provided information about consumers' perceptions and opinions of homeopathic OTC cough and cold products. Amazon ratings appear to be a good indicator of the presence or absence of AEs, and identified events were similar to FAERS. CONCLUSION: Amazon reviews may complement traditional data sources to identify AEs associated with unapproved OTC homeopathic products. This study is the first to use NLP in this context and lays the groundwork for future larger scale efforts.

Using natural language processing to identify the status of homelessness and housing instability among serious illness patients from clinical notes in an integrated healthcare system.

Background: Efficiently identifying the social risks of patients with serious illnesses (SIs) is the critical first step in providing patient-centered and value-driven care for this medically vulnerable population. Objective: To apply and further hone an existing natural language process (NLP) algorithm that identifies patients who are homeless/at risk of homeless to a SI population. Methods: Patients diagnosed with SI between 2019 and 2020 were identified using an adapted list of diagnosis codes from the Center for Advance Palliative Care from the Kaiser Permanente Southern California electronic health record. Clinical notes associated with medical encounters within 6 months before and after the diagnosis date were processed by a previously developed NLP algorithm to identify patients who were homeless/at risk of homelessness. To improve the generalizability to the SI population, the algorithm was refined by multiple iterations of chart review and adjudication. The updated algorithm was then applied to the SI population. Results: Among 206 993 patients with a SI diagnosis, 1737 (0.84%) were identified as homeless/at risk of homelessness. These patients were more likely to be male (51.1%), age among 45-64 years (44.7%), and have one or more emergency visit (65.8%) within a year of their diagnosis date. Validation of the updated algorithm yielded a sensitivity of 100.0% and a positive predictive value of 93.8%. Conclusions: The improved NLP algorithm effectively identified patients with SI who were homeless/at risk of homelessness and can be used to target interventions for this vulnerable group.

Extracting Complementary and Integrative Health Approaches in Electronic Health Records.

Complementary and Integrative Health (CIH) has gained increasing popularity in the past decades. While the evidence bases to support them are growing, there is still a gap in understanding their effects and potential adverse events using real-world data. The overall goal of this study is to represent information pertinent to both psychological and physical CIH approaches (specifically, using examples of music therapy, chiropractic, and aquatic exercise in this study) in an electronic health record (EHR) system. We also aim to evaluate the ability of existing natural language processing (NLP) systems to identify CIH approaches. A total of 300 notes were randomly selected and manually annotated. Annotations were made for status, symptom, and frequency of each approach. This set of annotations was used as a gold standard to evaluate the performance of NLP systems used in this study (specifically BioMedICUS, MetaMap, and cTAKES) for extracting CIH concepts. Venn diagram was used to investigate the consistency of medical records searching by Current Procedural Terminology (CPT) codes and CIH approaches keywords in SQL. Since CPT codes usually do not have specific mentions of CIH approaches, the Venn diagram had less overlap with those found in clinical notes for all three CIH therapies. The three NLP systems achieved 0.41 in average lenient match F1-score in all three CIH approaches, respectively. BioMedICUS achieved the best performance in aquatic exercise with an F1-score of 0.66. This study contributes to the overall representation of CIH in clinical note and lays a foundation for using EHR for clinical research for CIH approaches.

ChatGPT and Bard exhibit spontaneous citation fabrication during psychiatry literature search.

ChatGPT (Generative Pre-Trained Transformer) is a large language model (LLM), which comprises a neural network that has learned information and patterns of language use from large amounts of text on the internet. ChatGPT, introduced by OpenAI, responds to human queries in a conversational manner. Here, we aimed to assess whether ChatGPT could reliably produce accurate references to supplement the literature search process. We describe our March 2023 exchange with ChatGPT, which generated thirty-five citations, two of which were real. 12 citations were similar to actual manuscripts (e.g., titles with incorrect author lists, journals, or publication years) and the remaining 21, while plausible, were in fact a pastiche of multiple existent manuscripts. In June 2023, we re-tested ChatGPT's performance and compared it to that of Google's GPT counterpart, Bard 2.0. We investigated performance in English, as well as in Spanish and Italian. Fabrications made by LLMs, including erroneous citations, have been called "hallucinations"; we discuss reasons for which this is a misnomer. Furthermore, we describe potential explanations for citation fabrication by GPTs, as well as measures being taken to remedy this issue, including reinforcement learning. Our results underscore that output from conversational LLMs should be verified.

Development of a novel drug information provision system for Kampo medicine using natural language processing technology.

BACKGROUND: Kampo medicine is widely used in Japan; however, most physicians and pharmacists have insufficient knowledge and experience in it. Although a chatbot-style system using machine learning and natural language processing has been used in some clinical settings and proven useful, the system developed specifically for the Japanese language using this method has not been validated by research. The purpose of this study is to develop a novel drug information provision system for Kampo medicines using a natural language classifier® (NLC®) based on IBM Watson. METHODS: The target Kampo formulas were 33 formulas listed in the 17th revision of the Japanese Pharmacopoeia. The information included in the system comes from the package inserts of Kampo medicines, Manuals for Management of Individual Serious Adverse Drug Reactions, and data on off-label usage. The system developed in this study classifies questions about the drug information of Kampo formulas input by natural language into preset questions and outputs preset answers for the questions. The system uses morphological analysis, synonym conversion by thesaurus, and NLC®. We fine-tuned the information registered into NLC® and increased the thesaurus. To validate the system, 900 validation questions were provided by six pharmacists who were classified into high or low levels of knowledge and experience of Kampo medicines and three pharmacy students. RESULTS: The precision, recall, and F-measure of the system performance were 0.986, 0.915, and 0.949, respectively. The results were stable even with differences in the amount of expertise of the question authors. CONCLUSIONS: We developed a system using natural language classification that can give appropriate answers to most of the validation questions.

Patient Dietary Supplements Use: Do Results from Natural Language Processing of Clinical Notes Agree with Survey Data?

There is widespread use of dietary supplements, some prescribed but many taken without a physician's guidance. There are many potential interactions between supplements and both over-the-counter and prescription medications in ways that are unknown to patients. Structured medical records do not adequately document supplement use; however, unstructured clinical notes often contain extra information on supplements. We studied a group of 377 patients from three healthcare facilities and developed a natural language processing (NLP) tool to detect supplement use. Using surveys of these patients, we investigated the correlation between self-reported supplement use and NLP extractions from the clinical notes. Our model achieved an F1 score of 0.914 for detecting all supplements. Individual supplement detection had a variable correlation with survey responses, ranging from an F1 of 0.83 for calcium to an F1 of 0.39 for folic acid. Our study demonstrated good NLP performance while also finding that self-reported supplement use is not always consistent with the documented use in clinical records.

Holistic Evaluation of Language Models.

Language models (LMs) like GPT-3, PaLM, and ChatGPT are the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of LMs. LMs can serve many purposes and their behavior should satisfy many desiderata. To navigate the vast space of potential scenarios and metrics, we taxonomize the space and select representative subsets. We evaluate models on 16 core scenarios and 7 metrics, exposing important trade-offs. We supplement our core evaluation with seven targeted evaluations to deeply analyze specific aspects (including world knowledge, reasoning, regurgitation of copyrighted content, and generation of disinformation). We benchmark 30 LMs, from OpenAI, Microsoft, Google, Meta, Cohere, AI21 Labs, and others. Prior to HELM, models were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: all 30 models are now benchmarked under the same standardized conditions. Our evaluation surfaces 25 top-level findings. For full transparency, we release all raw model prompts and completions publicly. HELM is a living benchmark for the community, continuously updated with new scenarios, metrics, and models https://crfm.stanford.edu/helm/latest/.