Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support.

OBJECTIVES: To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts. MATERIALS AND METHODS: We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness. RESULTS: Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001). CONCLUSION: End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.

Leveraging large language models for generating responses to patient messages-a subjective analysis.

OBJECTIVE: This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal. MATERIALS AND METHODS: Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate fine-tuned models, we used 10 representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness. RESULTS: The dataset consisted of 499 794 pairs of patient messages and corresponding responses from the patient portal, with 5000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider's responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT's responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness. CONCLUSION: This subjective analysis suggests that leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and healthcare providers.

The 21st Century Cures Act Information Blocking Rule in Post-Acute Long-Term Care.

Included as part of the 21st Century Cures Act, the information blocking rule entered the first compliance phase in April 2021. Under this rule, post-acute long-term care (PALTC) facilities must not engage in any activity that interferes with accessing, using, or exchanging electronic health information. In addition, facilities must respond to information requests in a timely fashion and allow records to be readily available to patients and their delegates. Although hospitals have been slow to adapt to these changes, skilled nursing and other PALTC centers have been even slower. With a Final Rule enacted in recent years, awareness of the information-blocking rules became more crucial. We believe this commentary will help our colleagues interpret the rule for the PALTC setting. In addition, we provide points of emphasis to help guide those providers and administrative staff workers toward compliance and avoid potential penalties.

Fact Check: Assessing the Response of ChatGPT to Alzheimer's Disease Statements with Varying Degrees of Misinformation.

Background: There are many myths regarding Alzheimer's disease (AD) that have been circulated on the Internet, each exhibiting varying degrees of accuracy, inaccuracy, and misinformation. Large language models such as ChatGPT, may be a useful tool to help assess these myths for veracity and inaccuracy. However, they can induce misinformation as well. The objective of this study is to assess ChatGPT's ability to identify and address AD myths with reliable information. Methods: We conducted a cross-sectional study of clinicians' evaluation of ChatGPT (GPT 4.0)'s responses to 20 selected AD myths. We prompted ChatGPT to express its opinion on each myth and then requested it to rephrase its explanation using a simplified language that could be more readily understood by individuals with a middle school education. We implemented a survey using Redcap to determine the degree to which clinicians agreed with the accuracy of each ChatGPT's explanation and the degree to which the simplified rewriting was readable and retained the message of the original. We also collected their explanation on any disagreement with ChatGPT's responses. We used five Likert-type scale with a score ranging from -2 to 2 to quantify clinicians' agreement in each aspect of the evaluation. Results: The clinicians (n=11) were generally satisfied with ChatGPT's explanations, with a mean (SD) score of 1.0(±0.3) across the 20 myths. While ChatGPT correctly identified that all the 20 myths were inaccurate, some clinicians disagreed with its explanations on 7 of the myths.Overall, 9 of the 11 professionals either agreed or strongly agreed that ChatGPT has the potential to provide meaningful explanations of certain myths. Conclusions: The majority of surveyed healthcare professionals acknowledged the potential value of ChatGPT in mitigating AD misinformation. However, the need for more refined and detailed explanations of the disease's mechanisms and treatments was highlighted.

Leveraging Large Language Models for Generating Responses to Patient Messages.

Objective: This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal. Methods: Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate the fine-tuned models, we used ten representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness. Results: The dataset consisted of a total of 499,794 pairs of patient messages and corresponding responses from the patient portal, with 5,000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider's responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT's responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness. Conclusion: Leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and primary care providers.

Lessons learned: Development of COVID-19 clinical staging models at a large urban research institution.

Background/Objective: The University of Illinois at Chicago (UIC), along with many academic institutions worldwide, made significant efforts to address the many challenges presented during the COVID-19 pandemic by developing clinical staging and predictive models. Data from patients with a clinical encounter at UIC from July 1, 2019 to March 30, 2022 were abstracted from the electronic health record and stored in the UIC Center for Clinical and Translational Science Clinical Research Data Warehouse, prior to data analysis. While we saw some success, there were many failures along the way. For this paper, we wanted to discuss some of these obstacles and many of the lessons learned from the journey. Methods: Principle investigators, research staff, and other project team members were invited to complete an anonymous Qualtrics survey to reflect on the project. The survey included open-ended questions centering on participants' opinions about the project, including whether project goals were met, project successes, project failures, and areas that could have been improved. We then identified themes among the results. Results: Nine project team members (out of 30 members contacted) completed the survey. The responders were anonymous. The survey responses were grouped into four key themes: Collaboration, Infrastructure, Data Acquisition/Validation, and Model Building. Conclusion: Through our COVID-19 research efforts, the team learned about our strengths and deficiencies. We continue to work to improve our research and data translation capabilities.

Cloud Computing Synthetic Syndromic Surveillance Systems: Opioid Epidemic in Illinois.

With an increasing number of overdose cases yearly, the city of Chicago is facing an opioid epidemic. Many of these overdose cases lead to 911 calls that necessitate timely response from our limited emergency medicine services. This paper demonstrates how data from these calls along with synthetic and geospatial data can help create a syndromic surveillance system to combat this opioid crisis. Chicago EMS data is obtained from the Illinois Department of Public Health with a database structure using the NEMSIS standard. This information is combined with information from the RTI U.S. Household Population database, before being transferred to an Azure Data Lake. Afterwards, the data is integrated with Azure Synapse before being refined in another data lake and filtered with ICD-10 codes. Afterwards, we moved the data to ArcGIS Enterprise to apply spatial statistics and geospatial analytics to create our surveillance system.

An Initial Cost-Effectiveness Analysis of Magnetic Resonance-Guided Laser Interstitial Thermal Therapy in Pediatric Epilepsy Surgery.

INTRODUCTION: Magnetic resonance-guided laser interstitial thermal therapy (MRgLITT) is a new technology that provides a clinically efficacious and minimally invasive alternative to conventional microsurgical resection. However, little data exist on how costs compare to traditional open surgery. The goal of this paper is to investigate the cost-effectiveness of MRgLITT in the treatment of pediatric epilepsy. METHODS: We retrospectively analyzed the medical records of pediatric patients who underwent MRgLITT via the Visualase® thermal therapy system (Medtronic, Inc., Minneapolis, MN, USA) between December 2013 and September 2017. Direct costs associated with preoperative, operative, and follow-up care were extracted. Benefit was calculated in quality-adjusted life years (QALYs), and the cost-effectiveness was derived from the discounted total direct costs over QALY. Sensitivity analysis on 4 variables was utilized to assess the validity of our results. RESULTS: Twelve consecutive pediatric patients with medically refractory epilepsy underwent MRgLITT procedures. At the last postoperative follow-up, 8 patients were seizure free (Engel I, 66.7%), 2 demonstrated significant improvement (Engel II, 16.7%), and 2 patients showed worthwhile improvement (Engel III, 16.7%). The average cumulative discounted QALY was 2.11 over the lifetime of a patient. Adjusting for inflation, MRgLITT procedures had a cost-effectiveness of USD 22,211 per QALY. Our sensitivity analysis of cost variables is robust and supports the procedure to be cost--effective. CONCLUSION: Our data suggests that MRgLITT may be a cost-effective alternative to traditional surgical resection in pediatric epilepsy surgery.

Is the use of Stereotactic Electroencephalography Safe and Effective in Children? A Meta-Analysis of the use of Stereotactic Electroencephalography in Comparison to Subdural Grids for Invasive Epilepsy Monitoring in Pediatric Subjects.

BACKGROUND: Stereoelectroencephalography (SEEG) is an alternative addition to subdural grids (SDG) in invasive extra-operative monitoring for medically refractory epilepsy. Few studies exist on the clinical efficacy and safety of these techniques in pediatric populations. OBJECTIVE: To provide a comparative quantitative summary of surgical complications and postoperative seizure freedom associated with invasive extra-operative presurgical techniques in pediatric patients. METHODS: The systematic review was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). A literature search was conducted utilizing Ovid Medline, Embase, Pubmed, and the Cochrane database. RESULTS: Fourteen papers with a total of 697 pediatric patients undergoing invasive SDG monitoring and 9 papers with a total of 277 pediatric patients undergoing SEEG monitoring were utilized in the systemic review. Cerebral spinal fluid (CSF) leaks were the most common adverse event in the SDG studies (pooled prevalence 11.9% 95% confidence interval [CI] 5.7-23.3). There was one case of CSF leak in the SEEG studies. Intracranial hemorrhages (SDG: 10.7%, 95% CI 5.3-20.3; SEEG: 2.9%, 95% CI -0.7 to 10.8) and infection (SDG: 10.8%, 95% CI 6.7-17) were more common in the SDG studies reviewed. At the time of the last postoperative visit, a greater percentage of pediatric patients achieved seizure freedom in the SEEG studies (SEEG: 66.5%, 95% CI 58.8-73.4; SDG: 52.1%, 95% CI 43.0-61.1). CONCLUSION: SEEG is a safe alternative to SDG and should be considered on an individual basis for selected pediatric patients.

Taxes, bankruptcy costs, and capital structure in for-profit and not-for-profit hospitals.

About 60% of the US hospitals are not-for-profit and it is not clear how traditional theories of capital structure should be adapted to understand the borrowing behavior of not-for-profit hospitals. This paper identifies important determinants of capital structure taken from theories describing for-profit firms as well as prior literature on not-for-profit hospitals. We examine the differential effects these factors have on the capital structure of for-profit and not-for-profit hospitals. Specifically, we use a difference-in-differences regression framework to study how differences in leverage between for-profit and not-for-profit hospitals change in response to key explanatory variables (i.e. tax rates and bankruptcy costs). The sample in this study includes most US short-term general acute hospitals from 2000 to 2012. We find that personal and corporate income taxes and bankruptcy costs have significant and distinct effects on the capital structure of for-profit and not-for-profit hospitals. Specifically, relative to not-for-profit hospitals: (1) higher corporate income tax encourages for-profit hospitals to increase their debt usage; (2) higher personal income tax discourages for-profit hospitals to use debt; and (3) higher expected bankruptcy costs lead for-profit hospitals to use less debt. Over the past decade, the capital structure of for-profit hospitals has been more flexible as compared to that of not-for-profit hospitals. This may suggest that not-for-profit hospitals are more constrained by external financing resources. Particularly, our analysis suggests that not-for-profit hospitals operating in states with high corporate taxes but low personal income taxes may face particular challenges of borrowing funds relative to their for-profit competitors.

An initial cost-effectiveness analysis of intraoperative magnetic resonance imaging (iMRI) in pediatric epilepsy surgery.

PURPOSE: Previous studies have illustrated the clinical utility of the addition of intraoperative magnetic resonance imaging (iMRI) to conventional microsurgical resection. While iMRI requires initial capital cost investment, long-term reduction in costly follow-up management and reoperation costs may prove economically efficacious. The objective of this study is to investigate the cost-effectiveness of the addition of iMRI utilization versus conventional microsurgical techniques in focal cortical dysplasia (FCD) resection in pediatric patients with medically refractory epilepsy. METHODS: We retrospectively reviewed the medical records of pediatric subjects who underwent surgical resection of FCD at the Children's National Health System between March 2005 and April 2015. Patients were assigned to one of three cohorts: iMRI-assisted resection, conventional resection with iMRI-assisted reoperation, or conventional resection. Direct costs included preoperative, operative, postoperative, long-term follow-up, and antiepileptic drug (AED) costs. The cost-effectiveness was calculated as the sum total of all direct medical costs over the quality-adjusted life years (QALYs). We also performed sensitivity analysis on numerous variables to assess the validity of our results. RESULTS: Fifty-six consecutive pediatric patients underwent resective surgery for medically intractable FCD. Ten patients underwent iMRI-assisted resection; 7 underwent conventional resection followed by iMRI-assisted reoperation; 39 patients underwent conventional microsurgical resection. Taken over the lifetime of the patient, the cumulative discounted QALY of patients in the iMRI-assisted resection cohort was about 2.91 years, versus 2.61 years in the conventional resection with iMRI-assisted reoperation cohort, and 1.76 years for the conventional resection cohort. Adjusting for inflation, iMRI-assisted surgeries have a cost-effectiveness ratio of $16,179 per QALY, versus $28,514 per QALY for the conventional resection with iMRI-assisted reoperation cohort, and $49,960 per QALY for the conventional resection cohort. Sensitivity analysis demonstrated that no one single variable significantly altered cost-effectiveness across all three cohorts compared to the baseline results. CONCLUSION: The addition of iMRI to conventional microsurgical techniques for resection of FCD in pediatric patients with intractable epilepsy resulted in increased seizure freedom and reduction in long-term direct medical costs compared to conventional surgeries. Our data suggests that iMRI may be a cost-effective addition to the surgical armamentarium for epilepsy surgery.

Implicit statistical learning in language processing: word predictability is the key.

Fundamental learning abilities related to the implicit encoding of sequential structure have been postulated to underlie language acquisition and processing. However, there is very little direct evidence to date supporting such a link between implicit statistical learning and language. In three experiments using novel methods of assessing implicit learning and language abilities, we show that sensitivity to sequential structure - as measured by improvements to immediate memory span for structurally-consistent input sequences - is significantly correlated with the ability to use knowledge of word predictability to aid speech perception under degraded listening conditions. Importantly, the association remained even after controlling for participant performance on other cognitive tasks, including short-term and working memory, intelligence, attention and inhibition, and vocabulary knowledge. Thus, the evidence suggests that implicit learning abilities are essential for acquiring long-term knowledge of the sequential structure of language - i.e., knowledge of word predictability - and that individual differences on such abilities impact speech perception in everyday situations. These findings provide a new theoretical rationale linking basic learning phenomena to specific aspects of spoken language processing in adults, and may furthermore indicate new fruitful directions for investigating both typical and atypical language development.