Evaluating a Large Language Model's Ability to Answer Clinicians' Requests for Evidence Summaries.

Objective: This study investigated the performance of a generative artificial intelligence (AI) tool using GPT-4 in answering clinical questions in comparison with medical librarians' gold-standard evidence syntheses. Methods: Questions were extracted from an in-house database of clinical evidence requests previously answered by medical librarians. Questions with multiple parts were subdivided into individual topics. A standardized prompt was developed using the COSTAR framework. Librarians submitted each question into aiChat, an internally-managed chat tool using GPT-4, and recorded the responses. The summaries generated by aiChat were evaluated on whether they contained the critical elements used in the established gold-standard summary of the librarian. A subset of questions was randomly selected for verification of references provided by aiChat. Results: Of the 216 evaluated questions, aiChat's response was assessed as "correct" for 180 (83.3%) questions, "partially correct" for 35 (16.2%) questions, and "incorrect" for 1 (0.5%) question. No significant differences were observed in question ratings by question category (p=0.39). For a subset of 30% (n=66) of questions, 162 references were provided in the aiChat summaries, and 60 (37%) were confirmed as nonfabricated. Conclusions: Overall, the performance of a generative AI tool was promising. However, many included references could not be independently verified, and attempts were not made to assess whether any additional concepts introduced by aiChat were factually accurate. Thus, we envision this being the first of a series of investigations designed to further our understanding of how current and future versions of generative AI can be used and integrated into medical librarians' workflow.

Using a Natural Language Processing Approach to Support Rapid Knowledge Acquisition.

Implementing artificial intelligence to extract insights from large, real-world clinical data sets can supplement and enhance knowledge management efforts for health sciences research and clinical care. At Vanderbilt University Medical Center (VUMC), the in-house developed Word Cloud natural language processing system extracts coded concepts from patient records in VUMC's electronic health record repository using the Unified Medical Language System terminology. Through this process, the Word Cloud extracts the most prominent concepts found in the clinical documentation of a specific patient or population. The Word Cloud provides added value for clinical care decision-making and research. This viewpoint paper describes a use case for how the VUMC Center for Knowledge Management leverages the condition-disease associations represented by the Word Cloud to aid in the knowledge generation needed to inform the interpretation of phenome-wide association studies.

Acute Effects of Ketamine on Intracranial Pressure in Children With Severe Traumatic Brain Injury.

OBJECTIVES: The acute cerebral physiologic effects of ketamine in children have been incompletely described. We assessed the acute effects of ketamine on intracranial pressure (ICP) and cerebral perfusion pressure (CPP) in children with severe traumatic brain injury (TBI). DESIGN: In this retrospective observational study, patients received bolus doses of ketamine for sedation or as a treatment for ICP crisis (ICP > 20 mm Hg for > 5 min). Administration times were synchronized with ICP and CPP recordings at 1-minute intervals logged in an automated database within the electronic health record. ICP and CPP were each averaged in epochs following drug administration and compared with baseline values. Age-based CPP thresholds were subtracted from CPP recordings and compared with baseline values. Trends in ICP and CPP over time were assessed using generalized least squares regression. SETTING: A 30-bed tertiary care children's hospital PICU. PATIENTS: Children with severe TBI who underwent ICP monitoring. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We analyzed data from 33 patients, ages 1 month to 16 years, 22 of whom received bolus doses of ketamine, with 127 doses analyzed. Demographics, patient, and injury characteristics were similar between patients who did versus did not receive ketamine boluses. In analysis of the subset of ketamine doses used only for sedation, there was no significant difference in ICP or CPP from baseline. Eighteen ketamine doses were given during ICP crises in 11 patients. ICP decreased following these doses and threshold-subtracted CPP rose. CONCLUSIONS: In this retrospective, exploratory study, ICP did not increase following ketamine administration. In the setting of a guidelines-based protocol, ketamine was associated with a reduction in ICP during ICP crises. If these findings are reproduced in a larger study, ketamine may warrant consideration as a treatment for intracranial hypertension in children with severe TBI.

Systematic review of international studies evaluating MDRD and CKD-EPI estimated glomerular filtration rate (eGFR) equations in Black adults.

Use of race adjustment in estimating glomerular filtration rate (eGFR) has been challenged given concerns that it may negatively impact the clinical care of Black patients, as it results in Black patients being systematically assigned higher eGFR values than non-Black patients. We conducted a systematic review to assess how well eGFR, with and without race adjustment, estimates measured GFR (mGFR) in Black adults globally. A search across multiple databases for articles published from 1999 to May 2021 that compared eGFR to mGFR and reported outcomes by Black race was performed. We included studies that assessed eGFR using the Modification of Diet in Renal Disease (MDRD) and Chronic Kidney Disease Epidemiology Collaboration (CKD-EPICr) creatinine equations. Risk of study bias and applicability were assessed with the QUality Assessment of Diagnostic Accuracy Studies-2. Of 13,167 citations identified, 12 met the data synthesis criteria (unique patient cohorts in which eGFR was compared to mGFR with and without race adjustment). The studies included patients with and without kidney disease from Africa (n = 6), the United States (n = 3), Europe (n = 2), and Brazil (n = 1). Of 11 CKD-EPI equation studies, all assessed bias, 8 assessed accuracy, 6 assessed precision, and 5 assessed correlation/concordance. Of 7 MDRD equation studies, all assessed bias, 6 assessed accuracy, 5 assessed precision, and 3 assessed correlation/concordance. The majority of studies found that removal of race adjustment improved bias, accuracy, and precision of eGFR equations for Black adults. Risk of study bias was often unclear, but applicability concerns were low. Our systematic review supports the need for future studies to be conducted in diverse populations to assess the possibility of alternative approaches for estimating GFR. This study additionally provides systematic-level evidence for the American Society of Nephrology-National Kidney Foundation Task Force efforts to pursue other options for GFR estimation.

Real-time clinical note monitoring to detect conditions for rapid follow-up: A case study of clinical trial enrollment in drug-induced torsades de pointes and Stevens-Johnson syndrome.

Identifying acute events as they occur is challenging in large hospital systems. Here, we describe an automated method to detect 2 rare adverse drug events (ADEs), drug-induced torsades de pointes and Stevens-Johnson syndrome and toxic epidermal necrolysis, in near real time for participant recruitment into prospective clinical studies. A text processing system searched clinical notes from the electronic health record (EHR) for relevant keywords and alerted study personnel via email of potential patients for chart review or in-person evaluation. Between 2016 and 2018, the automated recruitment system resulted in capture of 138 true cases of drug-induced rare events, improving recall from 43% to 93%. Our focused electronic alert system maintained 2-year enrollment, including across an EHR migration from a bespoke system to Epic. Real-time monitoring of EHR notes may accelerate research for certain conditions less amenable to conventional study recruitment paradigms.

SPI-Hub™: a gateway to scholarly publishing information.

BACKGROUND: Advances in the health sciences rely on sharing research and data through publication. As information professionals are often asked to contribute their knowledge to assist clinicians and researchers in selecting journals for publication, the authors recognized an opportunity to build a decision support tool, SPI-Hub: Scholarly Publishing Information Hub™, to capture the team's collective publishing industry knowledge, while carefully retaining the quality of service. CASE PRESENTATION: SPI-Hub's decision support functionality relies on a data framework that describes journal publication policies and practices through a newly designed metadata structure, the Knowledge Management Journal Record™. Metadata fields are populated through a semi-automated process that uses custom programming to access content from multiple sources. Each record includes 25 metadata fields representing best publishing practices. Currently, the database includes more than 24,000 health sciences journal records. To correctly capture the resources needed for both completion and future maintenance of the project, the team conducted an internal study to assess time requirements for completing records through different stages of automation. CONCLUSIONS: The journal decision support tool, SPI-Hub, provides an opportunity to assess publication practices by compiling data from a variety of sources in a single location. Automated and semi-automated approaches have effectively reduced the time needed for data collection. Through a comprehensive knowledge management framework and the incorporation of multiple quality points specific to each journal, SPI-Hub provides prospective users with both recommendations for publication and holistic assessment of the trustworthiness of journals in which to publish research and acquire trusted knowledge.

A long journey to short abbreviations: developing an open-source framework for clinical abbreviation recognition and disambiguation (CARD).

OBJECTIVE: The goal of this study was to develop a practical framework for recognizing and disambiguating clinical abbreviations, thereby improving current clinical natural language processing (NLP) systems' capability to handle abbreviations in clinical narratives. METHODS: We developed an open-source framework for clinical abbreviation recognition and disambiguation (CARD) that leverages our previously developed methods, including: (1) machine learning based approaches to recognize abbreviations from a clinical corpus, (2) clustering-based semiautomated methods to generate possible senses of abbreviations, and (3) profile-based word sense disambiguation methods for clinical abbreviations. We applied CARD to clinical corpora from Vanderbilt University Medical Center (VUMC) and generated 2 comprehensive sense inventories for abbreviations in discharge summaries and clinic visit notes. Furthermore, we developed a wrapper that integrates CARD with MetaMap, a widely used general clinical NLP system. RESULTS AND CONCLUSION: CARD detected 27 317 and 107 303 distinct abbreviations from discharge summaries and clinic visit notes, respectively. Two sense inventories were constructed for the 1000 most frequent abbreviations in these 2 corpora. Using the sense inventories created from discharge summaries, CARD achieved an F1 score of 0.755 for identifying and disambiguating all abbreviations in a corpus from the VUMC discharge summaries, which is superior to MetaMap and Apache's clinical Text Analysis Knowledge Extraction System (cTAKES). Using additional external corpora, we also demonstrated that the MetaMap-CARD wrapper improved MetaMap's performance in recognizing disorder entities in clinical notes. The CARD framework, 2 sense inventories, and the wrapper for MetaMap are publicly available at https://sbmi.uth.edu/ccb/resources/abbreviation.htm . We believe the CARD framework can be a valuable resource for improving abbreviation identification in clinical NLP systems.

Using Best Practices to Extract, Organize, and Reuse Embedded Decision Support Content Knowledge Rules from Mature Clinical Systems.

Clinical decision support (CDS) knowledge, embedded over time in mature medical systems, presents an interesting and complex opportunity for information organization, maintenance, and reuse. To have a holistic view of all decision support requires an in-depth understanding of each clinical system as well as expert knowledge of the latest evidence. This approach to clinical decision support presents an opportunity to unify and externalize the knowledge within rules-based decision support. Driven by an institutional need to prioritize decision support content for migration to new clinical systems, the Center for Knowledge Management and Health Information Technology teams applied their unique expertise to extract content from individual systems, organize it through a single extensible schema, and present it for discovery and reuse through a newly created Clinical Support Knowledge Acquisition and Archival Tool (CS-KAAT). CS-KAAT can build and maintain the underlying knowledge infrastructure needed by clinical systems.

Code status and resuscitation options in the electronic health record.

AIM: The advance discussion and documentation of code-status is important in preventing undesired cardiopulmonary resuscitation and related end of life interventions. Code-status documentation remains infrequent and paper-based, which limits its usefulness. This study evaluates a tool to document code-status in the electronic health records at a large teaching hospital, and analyzes the corresponding data. METHODS: Encounter data for patients admitted to the Medical Center were collected over a period of 12 months (01-APR-2012-31-MAR-2013) and the code-status attribute was tracked for individual patients. The code-status data were analyzed separately for adult and pediatric patient populations. We considered 131,399 encounters for 83,248 adult patients and 80,778 encounters for 55,656 pediatric patients in this study. RESULTS: 71% of the adult patients and 30% of the pediatric patients studied had a documented code-status. Age and severity of illness influenced the decision to document code-status. Demographics such as gender, race, ethnicity, and proximity of primary residence were also associated with the documentation of code-status. CONCLUSION: Absence of a recorded code-status may result in unnecessary interventions. Code-status in paper charts may be difficult to access in cardiopulmonary arrest situations and may result in unnecessary and unwanted interventions and procedures. Documentation of code-status in electronic records creates a readily available reference for care providers.

Assisted annotation of medical free text using RapTAT.

OBJECTIVE: To determine whether assisted annotation using interactive training can reduce the time required to annotate a clinical document corpus without introducing bias. MATERIALS AND METHODS: A tool, RapTAT, was designed to assist annotation by iteratively pre-annotating probable phrases of interest within a document, presenting the annotations to a reviewer for correction, and then using the corrected annotations for further machine learning-based training before pre-annotating subsequent documents. Annotators reviewed 404 clinical notes either manually or using RapTAT assistance for concepts related to quality of care during heart failure treatment. Notes were divided into 20 batches of 19-21 documents for iterative annotation and training. RESULTS: The number of correct RapTAT pre-annotations increased significantly and annotation time per batch decreased by ~50% over the course of annotation. Annotation rate increased from batch to batch for assisted but not manual reviewers. Pre-annotation F-measure increased from 0.5 to 0.6 to >0.80 (relative to both assisted reviewer and reference annotations) over the first three batches and more slowly thereafter. Overall inter-annotator agreement was significantly higher between RapTAT-assisted reviewers (0.89) than between manual reviewers (0.85). DISCUSSION: The tool reduced workload by decreasing the number of annotations needing to be added and helping reviewers to annotate at an increased rate. Agreement between the pre-annotations and reference standard, and agreement between the pre-annotations and assisted annotations, were similar throughout the annotation process, which suggests that pre-annotation did not introduce bias. CONCLUSIONS: Pre-annotations generated by a tool capable of interactive training can reduce the time required to create an annotated document corpus by up to 50%.

Development and evaluation of RapTAT: a machine learning system for concept mapping of phrases from medical narratives.

Rapid, automated determination of the mapping of free text phrases to pre-defined concepts could assist in the annotation of clinical notes and increase the speed of natural language processing systems. The aim of this study was to design and evaluate a token-order-specific naïve Bayes-based machine learning system (RapTAT) to predict associations between phrases and concepts. Performance was assessed using a reference standard generated from 2860 VA discharge summaries containing 567,520 phrases that had been mapped to 12,056 distinct Systematized Nomenclature of Medicine - Clinical Terms (SNOMED CT) concepts by the MCVS natural language processing system. It was also assessed on the manually annotated, 2010 i2b2 challenge data. Performance was established with regard to precision, recall, and F-measure for each of the concepts within the VA documents using bootstrapping. Within that corpus, concepts identified by MCVS were broadly distributed throughout SNOMED CT, and the token-order-specific language model achieved better performance based on precision, recall, and F-measure (0.95±0.15, 0.96±0.16, and 0.95±0.16, respectively; mean±SD) than the bag-of-words based, naïve Bayes model (0.64±0.45, 0.61±0.46, and 0.60±0.45, respectively) that has previously been used for concept mapping. Precision, recall, and F-measure on the i2b2 test set were 92.9%, 85.9%, and 89.2% respectively, using the token-order-specific model. RapTAT required just 7.2ms to map all phrases within a single discharge summary, and mapping rate did not decrease as the number of processed documents increased. The high performance attained by the tool in terms of both accuracy and speed was encouraging, and the mapping rate should be sufficient to support near-real-time, interactive annotation of medical narratives. These results demonstrate the feasibility of rapidly and accurately mapping phrases to a wide range of medical concepts based on a token-order-specific naïve Bayes model and machine learning.

Analyzing differences between chinese and english clinical text: a cross-institution comparison of discharge summaries in two languages.

Worldwide adoption of Electronic Medical Records (EMRs) databases in health care have generated an unprecedented amount of clinical data available electronically. There has been an increasing trend in US and western institutions towards collaborating with China on medical research using EMR data. However, few studies have investigated characteristics of EMR data in China and their differences with the data in US hospitals. As an initial step towards differentiating EMR data in Chinese and US systems, this study attempts to understand system and cultural differences that may exist between Chinese and English clinical documents. We collected inpatient discharge summaries from one Chinese and from three US institutions and manually analyzed three major clinical components in text: medical problems, tests, and treatments. We reported comparison results at the document level and section level and discussed potential reasons for observed differences. Documenting and understanding differences in clinical reports from the US and China EMRs are important for cross-country collaborations. Our study also provided valuable insights for developing natural language processing tools for Chinese clinical text.

Recommendations for the design, implementation and evaluation of social support in online communities, networks, and groups.

A new model of health care is emerging in which individuals can take charge of their health by connecting to online communities and social networks for personalized support and collective knowledge. Web 2.0 technologies expand the traditional notion of online support groups into a broad and evolving range of informational, emotional, as well as community-based concepts of support. In order to apply these technologies to patient-centered care, it is necessary to incorporate more inclusive conceptual frameworks of social support and community-based research methodologies. This paper introduces a conceptualization of online social support, reviews current challenges in online support research, and outlines six recommendations for the design, evaluation, and implementation of social support in online communities, networks, and groups. The six recommendations are illustrated by CanConnect, an online community for cancer survivors in middle Tennessee. These recommendations address the interdependencies between online and real-world support and emphasize an inclusive framework of interpersonal and community-based support. The applications of these six recommendations are illustrated through a discussion of online support for cancer survivors.

A comparative study of current Clinical Natural Language Processing systems on handling abbreviations in discharge summaries.

Clinical Natural Language Processing (NLP) systems extract clinical information from narrative clinical texts in many settings. Previous research mentions the challenges of handling abbreviations in clinical texts, but provides little insight into how well current NLP systems correctly recognize and interpret abbreviations. In this paper, we compared performance of three existing clinical NLP systems in handling abbreviations: MetaMap, MedLEE, and cTAKES. The evaluation used an expert-annotated gold standard set of clinical documents (derived from from 32 de-identified patient discharge summaries) containing 1,112 abbreviations. The existing NLP systems achieved suboptimal performance in abbreviation identification, with F-scores ranging from 0.165 to 0.601. MedLEE achieved the best F-score of 0.601 for all abbreviations and 0.705 for clinically relevant abbreviations. This study suggested that accurate identification of clinical abbreviations is a challenging task and that more advanced abbreviation recognition modules might improve existing clinical NLP systems.

Detecting abbreviations in discharge summaries using machine learning methods.

Recognition and identification of abbreviations is an important, challenging task in clinical natural language processing (NLP). A comprehensive lexical resource comprised of all common, useful clinical abbreviations would have great applicability. The authors present a corpus-based method to create a lexical resource of clinical abbreviations using machine-learning (ML) methods, and tested its ability to automatically detect abbreviations from hospital discharge summaries. Domain experts manually annotated abbreviations in seventy discharge summaries, which were randomly broken into a training set (40 documents) and a test set (30 documents). We implemented and evaluated several ML algorithms using the training set and a list of pre-defined features. The subsequent evaluation using the test set showed that the Random Forest classifier had the highest F-measure of 94.8% (precision 98.8% and recall of 91.2%). When a voting scheme was used to combine output from various ML classifiers, the system achieved the highest F-measure of 95.7%.

Integrated information systems for electronic chemotherapy medication administration.

INTRODUCTION: Chemotherapy administration is a highly complex and distributed task in both the inpatient and outpatient infusion center settings. The American Society of Clinical Oncology and the Oncology Nursing Society (ASCO/ONS) have developed standards that specify procedures and documentation requirements for safe chemotherapy administration. Yet paper-based approaches to medication administration have several disadvantages and do not provide any decision support for patient safety checks. Electronic medication administration that includes bar coding technology may provide additional safety checks, enable consistent documentation structure, and have additional downstream benefits. METHODS: We describe the specialized configuration of clinical informatics systems for electronic chemotherapy medication administration. The system integrates the patient registration system, the inpatient order entry system, the pharmacy information system, the nursing documentation system, and the electronic health record. RESULTS: We describe the process of deploying this infrastructure in the adult and pediatric inpatient oncology, hematology, and bone marrow transplant wards at Vanderbilt University Medical Center. We have successfully adapted the system for the oncology-specific documentation requirements detailed in the ASCO/ONS guidelines for chemotherapy administration. However, several limitations remain with regard to recording the day of treatment and dose number. CONCLUSION: Overall, the configured systems facilitate compliance with the ASCO/ONS guidelines and improve the consistency of documentation and multidisciplinary team communication. Our success has prompted us to deploy this infrastructure in our outpatient chemotherapy infusion centers, a process that is currently underway and that will require a few unique considerations.

Generating Clinical Notes for Electronic Health Record Systems.

Clinical notes summarize interactions that occur between patients and healthcare providers. With adoption of electronic health record (EHR) and computer-based documentation (CBD) systems, there is a growing emphasis on structuring clinical notes to support reusing data for subsequent tasks. However, clinical documentation remains one of the most challenging areas for EHR system development and adoption. The current manuscript describes the Vanderbilt experience with implementing clinical documentation with an EHR system. Based on their experience rolling out an EHR system that supports multiple methods for clinical documentation, the authors recommend that documentation method selection be made on the basis of clinical workflow, note content standards and usability considerations, rather than on a theoretical need for structured data.

Integrating best evidence into patient care: a process facilitated by a seamless integration with informatics tools.

The Vanderbilt University paper discusses how the Eskind Biomedical Library at Vanderbilt University Medical Center transitioned from a simplistic approach that linked resources to the institutional electronic medical record system, StarPanel, to a value-added service that is designed to deliver highly relevant information. Clinical teams formulate complex patient-specific questions via an evidence-based medicine literature request basket linked to individual patient records. The paper transitions into discussing how the StarPanel approach acted as a springboard for two additional projects that use highly trained knowledge management librarians with informatics expertise to integrate evidence into both order sets and a patient portal, MyHealth@Vanderbilt.

A real-time ventilator management dashboard: toward hardwiring compliance with evidence-based guidelines.

Care of mechanically ventilated patients requires coordination between multiple caregivers, necessitating the availability of accurate and timely information on patient status. Researchers have documented positive effects of several interventions on the rates of developing ventilator associated complications, such as providing regular oral care and elevating the head of the bed. Informatics tools, such as electronic whiteboards, reminders, and alerts have been shown to aid in clinician compliance with guidelines or protocols. The purpose of this project was to design and implement a real-time ventilator management dashboard to show patient status with respect to elements important for ventilator management and infection prevention in the adult Intensive Care Units at Vanderbilt University Medical Center.