Artificial intelligence chatbot performance in triage of ophthalmic conditions.

BACKGROUND: Timely access to human expertise for affordable and efficient triage of ophthalmic conditions is inconsistent. With recent advancements in publicly available artificial intelligence (AI) chatbots, the lay public may turn to these tools for triage of ophthalmic complaints. Validation studies are necessary to evaluate the performance of AI chatbots as triage tools and inform the public regarding their safety. OBJECTIVE: To evaluate the triage performance of AI chatbots for ophthalmic conditions. DESIGN: Cross-sectional study. SETTING: Single centre. PARTICIPANTS: Ophthalmology trainees, OpenAI ChatGPT (GPT-4), Bing Chat, and WebMD Symptom Checker. METHODS: Forty-four clinical vignettes representing common ophthalmic complaints were developed, and a standardized pathway of prompts was presented to each tool in March 2023. Primary outcomes were proportion of responses with the correct diagnosis listed in the top 3 possible diagnoses and proportion with correct triage urgency. Ancillary outcomes included presence of grossly inaccurate statements, mean reading grade level, mean response word count, proportion with attribution, and most common sources cited. RESULTS: The ophthalmologists in training, ChatGPT, Bing Chat, and the WebMD Symptom Checker listed the appropriate diagnosis among the top 3 suggestions in 42 (95%), 41 (93%), 34 (77%), and 8 (33%) cases, respectively. Triage urgency was appropriate in 38 (86%), 43 (98%), and 37 (84%) cases for ophthalmology trainees, ChatGPT, and Bing Chat, correspondingly. CONCLUSIONS: ChatGPT using the GPT-4 model offered high diagnostic and triage accuracy that was comparable with that of ophthalmology trainees with no grossly inaccurate statements. Bing Chat had lower accuracy and a tendency to overestimate triage urgency.

Predicting Kidney Transplant Recipient Cohorts' 30-Day Rehospitalization Using Clinical Notes and Electronic Health Care Record Data.

Introduction: Rehospitalization after kidney transplant is costly to patients and health care systems and is associated with poor outcomes. Few prediction model studies have examined whether inclusion of clinical notes data from the electronic medical record (EMR) enhances prediction of rehospitalization. Methods: In a retrospective, observational study of first-time, adult kidney transplant recipients at a large, urban hospital in southeastern United States (2005-2015), we examined 30-day rehospitalization (30DR) using structured EMR and unstructured (i.e., clinical notes) data. We used natural language processing (NLP) methods on 8 types of clinical notes and included terms in predictive models using unsupervised machine learning approaches. Both the area under the receiver operating curve and precision-recall curve (ROC and PRC, respectively) were used to determine and compare model accuracy, and 5-fold cross-validation tested model performance. Results: Among 2060 kidney transplant recipients, 30.7% were readmitted within 30 days. Predictive models using clinical notes did not meaningfully improve performance over previous models using structured data alone (ROC 0.6821; 95% confidence interval [CI]: 0.6644, 0.6998). Predictive models built using solely clinical notes performed worse than models using both clinical notes and structured data. The data that contributed to the top performing models were not identical but both included structured data and progress notes (ROC 0.6902; 95% CI: 0.6699, 0.7105). Conclusions: Including new features from clinical notes in risk prediction models did not substantially increase predictive accuracy for 30DR for kidney transplant recipients. Future research should consider pooling data from multiple institutions to increase sample size and avoid overfitting models.

Development of digital voice biomarkers and associations with cognition, cerebrospinal biomarkers, and neural representation in early Alzheimer's disease.

Introduction: Advances in natural language processing (NLP), speech recognition, and machine learning (ML) allow the exploration of linguistic and acoustic changes previously difficult to measure. We developed processes for deriving lexical-semantic and acoustic measures as Alzheimer's disease (AD) digital voice biomarkers. Methods: We collected connected speech, neuropsychological, neuroimaging, and cerebrospinal fluid (CSF) AD biomarker data from 92 cognitively unimpaired (40 Aß+) and 114 impaired (63 Aß+) participants. Acoustic and lexical-semantic features were derived from audio recordings using ML approaches. Results: Lexical-semantic (area under the curve [AUC] = 0.80) and acoustic (AUC = 0.77) scores demonstrated higher diagnostic performance for detecting MCI compared to Boston Naming Test (AUC = 0.66). Only lexical-semantic scores detected amyloid-ß status (p = 0.0003). Acoustic scores associated with hippocampal volume (p = 0.017) while lexical-semantic scores associated with CSF amyloid-ß (p = 0.007). Both measures were significantly associated with 2-year disease progression. Discussion: These preliminary findings suggest that derived digital biomarkers may identify cognitive impairment in preclinical and prodromal AD, and may predict disease progression. Highlights: This study derived lexical-semantic and acoustics features as Alzheimer's disease (AD) digital biomarkers.These features were derived from audio recordings using machine learning approaches.Voice biomarkers detected cognitive impairment and amyloid-ß status in early stages of AD.Voice biomarkers may predict Alzheimer's disease progression.These markers significantly mapped to functional connectivity in AD-susceptible brain regions.

COVID-19 Pandemic-Associated Changes in the Acuity of Brain MRI Findings: A Secondary Analysis of Reports Using Natural Language Processing.

RATIONALE AND OBJECTIVES: We aimed to assess early COVID-19 pandemic-associated changes in brain MRI examination frequency and acuity of imaging findings acuity. METHODS: Using a natural language processing model, we retrospectively categorized reported findings of 12,346 brain MRI examinations performed during 6-month pre-pandemic and early pandemic time periods across a large metropolitan health system into 3 acuity levels: (1) normal or near normal; (2) incidental or chronic findings not requiring a management change; and (3) new or progressive findings requiring a management change. Brain MRI frequency and imaging finding acuity level were compared over time. RESULTS: Between March and August of 2019 (pre-pandemic) and 2020 (early pandemic), our health system brain MRI examination volumes decreased 17.0% (6745 vs 5601). Comparing calendar-matched 6-month periods, the proportion of higher acuity findings increased significantly (P< 0.001) from pre-pandemic (22.5%, 43.6% and 34.0% in acuity level 1, 2, and 3, respectively) to early pandemic periods (19.1%, 40.9%, and 40.1%). During the second 3 months of the early pandemic period, as MRI volumes recovered to near baseline, the proportion of higher acuity findings remained high (42.6% vs 34.1%) compared with a similar pre-pandemic period. In a multivariable analysis, Black (B coefficient, 0.16) and underinsured population (B coefficient, 0.33) presented with higher acuity findings (P< 0.05). CONCLUSIONS: As the volume of brain MRI examinations decreased during the early COVID-19 pandemic, the relative proportion of examinations with higher acuity findings increased significantly. Pandemic-related changes in patient outcomes related to reduced imaging access merits further attention.

SMAT: An attention-based deep learning solution to the automation of schema matching.

Schema matching aims to identify the correspondences among attributes of database schemas. It is frequently considered as the most challenging and decisive stage existing in many contemporary web semantics and database systems. Low-quality algorithmic matchers fail to provide improvement while manually annotation consumes extensive human efforts. Further complications arise from data privacy in certain domains such as healthcare, where only schema-level matching should be used to prevent data leakage. For this problem, we propose SMAT, a new deep learning model based on state-of-the-art natural language processing techniques to obtain semantic mappings between source and target schemas using only the attribute name and description. SMAT avoids directly encoding domain knowledge about the source and target systems, which allows it to be more easily deployed across different sites. We also introduce a new benchmark dataset, OMAP, based on real-world schema-level mappings from the healthcare domain. Our extensive evaluation of various benchmark datasets demonstrates the potential of SMAT to help automate schema-level matching tasks.

Towards comprehensive syntactic and semantic annotations of the clinical narrative.

OBJECTIVE: To create annotated clinical narratives with layers of syntactic and semantic labels to facilitate advances in clinical natural language processing (NLP). To develop NLP algorithms and open source components. METHODS: Manual annotation of a clinical narrative corpus of 127 606 tokens following the Treebank schema for syntactic information, PropBank schema for predicate-argument structures, and the Unified Medical Language System (UMLS) schema for semantic information. NLP components were developed. RESULTS: The final corpus consists of 13 091 sentences containing 1772 distinct predicate lemmas. Of the 766 newly created PropBank frames, 74 are verbs. There are 28 539 named entity (NE) annotations spread over 15 UMLS semantic groups, one UMLS semantic type, and the Person semantic category. The most frequent annotations belong to the UMLS semantic groups of Procedures (15.71%), Disorders (14.74%), Concepts and Ideas (15.10%), Anatomy (12.80%), Chemicals and Drugs (7.49%), and the UMLS semantic type of Sign or Symptom (12.46%). Inter-annotator agreement results: Treebank (0.926), PropBank (0.891-0.931), NE (0.697-0.750). The part-of-speech tagger, constituency parser, dependency parser, and semantic role labeler are built from the corpus and released open source. A significant limitation uncovered by this project is the need for the NLP community to develop a widely agreed-upon schema for the annotation of clinical concepts and their relations. CONCLUSIONS: This project takes a foundational step towards bringing the field of clinical NLP up to par with NLP in the general domain. The corpus creation and NLP components provide a resource for research and application development that would have been previously impossible.

A corpus of full-text journal articles is a robust evaluation tool for revealing differences in performance of biomedical natural language processing tools.

BACKGROUND: We introduce the linguistic annotation of a corpus of 97 full-text biomedical publications, known as the Colorado Richly Annotated Full Text (CRAFT) corpus. We further assess the performance of existing tools for performing sentence splitting, tokenization, syntactic parsing, and named entity recognition on this corpus. RESULTS: Many biomedical natural language processing systems demonstrated large differences between their previously published results and their performance on the CRAFT corpus when tested with the publicly available models or rule sets. Trainable systems differed widely with respect to their ability to build high-performing models based on this data. CONCLUSIONS: The finding that some systems were able to train high-performing models based on this corpus is additional evidence, beyond high inter-annotator agreement, that the quality of the CRAFT corpus is high. The overall poor performance of various systems indicates that considerable work needs to be done to enable natural language processing systems to work well when the input is full-text journal articles. The CRAFT corpus provides a valuable resource to the biomedical natural language processing community for evaluation and training of new models for biomedical full text publications.