Fellows of the American Medical Informatics Association (FAMIA): Looking Back and Looking Ahead.

BACKGROUND: Over the past 30 years, the American Medical Informatics Association (AMIA) has played a pivotal role in fostering a collaborative community for professionals in biomedical and health informatics. As an interdisciplinary association, AMIA brings together individuals with clinical, research, and computer expertise and emphasizes the use of data to enhance biomedical research and clinical work. The need for a recognition program within AMIA, acknowledging applied informatics skills by members, led to the establishment of the Fellows of AMIA (FAMIA) Recognition Program in 2018. OBJECTIVES: To outline the evolution of the FAMIA program and shed light on its origins, development, and impact. This report explores factors that led to the establishment of FAMIA, considerations affecting its development, and the objectives FAMIA seeks to achieve within the broader context of AMIA. METHODS: The development of FAMIA is examined through a historical lens, encompassing key milestones, discussions, and decisions that shaped the program. Insights into the formation of FAMIA were gathered through discussions within AMIA membership and leadership, including proposals, board-level discussions, and the involvement of key stakeholders. Additionally, the report outlines criteria for FAMIA eligibility and the pathways available for recognition, namely the Certification Pathway and the Long-Term Experience Pathway. RESULTS: The FAMIA program has inducted five classes, totaling 602 fellows. An overview of disciplines, roles, and application pathways for FAMIA members is provided. A comparative analysis with other fellow recognition programs in related fields showcases the unique features and contributions of FAMIA in acknowledging applied informatics. CONCLUSION: Now in its sixth year, FAMIA acknowledges the growing influence of applied informatics within health information professionals, recognizing individuals with experience, training, and a commitment to the highest level of applied informatics and the science associated with it.

Using large language model to guide patients to create efficient and comprehensive clinical care message.

OBJECTIVE: This study aims to investigate the feasibility of using Large Language Models (LLMs) to engage with patients at the time they are drafting a question to their healthcare providers, and generate pertinent follow-up questions that the patient can answer before sending their message, with the goal of ensuring that their healthcare provider receives all the information they need to safely and accurately answer the patient's question, eliminating back-and-forth messaging, and the associated delays and frustrations. METHODS: We collected a dataset of patient messages sent between January 1, 2022 to March 7, 2023 at Vanderbilt University Medical Center. Two internal medicine physicians identified 7 common scenarios. We used 3 LLMs to generate follow-up questions: (1) Comprehensive LLM Artificial Intelligence Responder (CLAIR): a locally fine-tuned LLM, (2) GPT4 with a simple prompt, and (3) GPT4 with a complex prompt. Five physicians rated them with the actual follow-ups written by healthcare providers on clarity, completeness, conciseness, and utility. RESULTS: For five scenarios, our CLAIR model had the best performance. The GPT4 model received higher scores for utility and completeness but lower scores for clarity and conciseness. CLAIR generated follow-up questions with similar clarity and conciseness as the actual follow-ups written by healthcare providers, with higher utility than healthcare providers and GPT4, and lower completeness than GPT4, but better than healthcare providers. CONCLUSION: LLMs can generate follow-up patient messages designed to clarify a medical question that compares favorably to those generated by healthcare providers.

Accuracy and Bias in Artificial Intelligence Chatbot Recommendations for Oculoplastic Surgeons.

Purpose The purpose of this study is to assess the accuracy of and bias in recommendations for oculoplastic surgeons from three artificial intelligence (AI) chatbot systems. Methods ChatGPT, Microsoft Bing Balanced, and Google Bard were asked for recommendations for oculoplastic surgeons practicing in 20 cities with the highest population in the United States. Three prompts were used: "can you help me find (an oculoplastic surgeon)/(a doctor who does eyelid lifts)/(an oculofacial plastic surgeon) in (city)." Results A total of 672 suggestions were made between (oculoplastic surgeon; doctor who does eyelid lifts; oculofacial plastic surgeon); 19.8% suggestions were excluded, leaving 539 suggested physicians. Of these, 64.1% were oculoplastics specialists (of which 70.1% were American Society of Ophthalmic Plastic and Reconstructive Surgery (ASOPRS) members); 16.1% were general plastic surgery trained, 9.0% were ENT trained, 8.8% were ophthalmology but not oculoplastics trained, and 1.9% were trained in another specialty. 27.7% of recommendations across all AI systems were female. Conclusions Among the chatbot systems tested, there were high rates of inaccuracy: up to 38% of recommended surgeons were nonexistent or not practicing in the city requested, and 35.9% of those recommended as oculoplastic/oculofacial plastic surgeons were not oculoplastics specialists. Choice of prompt affected the result, with requests for "a doctor who does eyelid lifts" resulting in more plastic surgeons and ENTs and fewer oculoplastic surgeons. It is important to identify inaccuracies and biases in recommendations provided by AI systems as more patients may start using them to choose a surgeon.

Development and Validation of an Automated, Real-Time Predictive Model for Postpartum Hemorrhage.

OBJECTIVE: To develop and validate a predictive model for postpartum hemorrhage that can be deployed in clinical care using automated, real-time electronic health record (EHR) data and to compare performance of the model with a nationally published risk prediction tool. METHODS: A multivariable logistic regression model was developed from retrospective EHR data from 21,108 patients delivering at a quaternary medical center between January 1, 2018, and April 30, 2022. Deliveries were divided into derivation and validation sets based on an 80/20 split by date of delivery. Postpartum hemorrhage was defined as blood loss of 1,000 mL or more in addition to postpartum transfusion of 1 or more units of packed red blood cells. Model performance was evaluated by the area under the receiver operating characteristic curve (AUC) and was compared with a postpartum hemorrhage risk assessment tool published by the CMQCC (California Maternal Quality Care Collaborative). The model was then programmed into the EHR and again validated with prospectively collected data from 928 patients between November 7, 2023, and January 31, 2024. RESULTS: Postpartum hemorrhage occurred in 235 of 16,862 patients (1.4%) in the derivation cohort. The predictive model included 21 risk factors and demonstrated an AUC of 0.81 (95% CI, 0.79-0.84) and calibration slope of 1.0 (Brier score 0.013). During external temporal validation, the model maintained discrimination (AUC 0.80, 95% CI, 0.72-0.84) and calibration (calibration slope 0.95, Brier score 0.014). This was superior to the CMQCC tool (AUC 0.69 [95% CI, 0.67-0.70], P <.001). The model maintained performance in prospective, automated data collected with the predictive model in real time (AUC 0.82 [95% CI, 0.73-0.91]). CONCLUSION: We created and temporally validated a postpartum hemorrhage prediction model, demonstrated its superior performance over a commonly used risk prediction tool, successfully coded the model into the EHR, and prospectively validated the model using risk factor data collected in real time. Future work should evaluate external generalizability and effects on patient outcomes; to facilitate this work, we have included the model coefficients and examples of EHR integration in the article.

Implementable Prediction of Pressure Injuries in Hospitalized Adults: Model Development and Validation.

Background: Numerous pressure injury prediction models have been developed using electronic health record data, yet hospital-acquired pressure injuries (HAPIs) are increasing, which demonstrates the critical challenge of implementing these models in routine care. Objective: To help bridge the gap between development and implementation, we sought to create a model that was feasible, broadly applicable, dynamic, actionable, and rigorously validated and then compare its performance to usual care (ie, the Braden scale). Methods: We extracted electronic health record data from 197,991 adult hospital admissions with 51 candidate features. For risk prediction and feature selection, we used logistic regression with a least absolute shrinkage and selection operator (LASSO) approach. To compare the model with usual care, we used the area under the receiver operating curve (AUC), Brier score, slope, intercept, and integrated calibration index. The model was validated using a temporally staggered cohort. Results: A total of 5458 HAPIs were identified between January 2018 and July 2022. We determined 22 features were necessary to achieve a parsimonious and highly accurate model. The top 5 features included tracheostomy, edema, central line, first albumin measure, and age. Our model achieved higher discrimination than the Braden scale (AUC 0.897, 95% CI 0.893-0.901 vs AUC 0.798, 95% CI 0.791-0.803). Conclusions: We developed and validated an accurate prediction model for HAPIs that surpassed the standard-of-care risk assessment and fulfilled necessary elements for implementation. Future work includes a pragmatic randomized trial to assess whether our model improves patient outcomes.

Surgically-relevant quality of life thresholds for the Short Inflammatory Bowel Disease Questionnaire in Crohn's disease.

BACKGROUND: Despite growing interest in patient-reported outcome measures to track the progression of Crohn's disease, frameworks to apply these questionnaires in the preoperative setting are lacking. Using the Short Inflammatory Bowel Disease Questionnaire (sIBDQ), this study aimed to describe the interpretable quality of life thresholds and examine potential associations with future bowel resection in Crohn's disease. METHODS: Adult patients with Crohn's disease completing an sIBDQ at a clinic visit between 2020 and 2022 were eligible. A stoplight framework was adopted for sIBDQ scores, including a "Resection Red" zone suggesting poor quality of life that may benefit from discussions about surgery as well as a "Nonoperative Green" zone. Thresholds were identified with both anchor- and distribution-based methods using receiver operating characteristic curve analysis and subgroup percentile scores, respectively. To quantify associations between sIBDQ scores and subsequent bowel resection, multivariable logistic regression models were fit with covariates of age, sex assigned at birth, body mass index, medications, disease pattern and location, resection history, and the Harvey Bradshaw Index. The incremental discriminatory value of the sIBDQ beyond clinical factors was assessed through the area under the receiver operating characteristics curve (AUC) with an internal validation through bootstrap resampling. RESULTS: Of the 2003 included patients, 102 underwent Crohn's-related bowel resection. The sIBDQ Nonoperative Green zone threshold ranged from 61 to 64 and the Resection Red zone from 36 to 38. When adjusting for clinical covariates, a worse sIBDQ score was associated with greater odds of subsequent 90-day bowel resection when considered as a 1-point (odds ratio [OR] [95% CI], 1.05 [1.03-1.07]) or 5-point change (OR [95% CI], 1.27 [1.14-1.41]). Inclusion of the sIBDQ modestly improved discriminative performance (AUC [95% CI], 0.85 [0.85-0.86]) relative to models that included only demographics (0.57 [0.57-0.58]) or demographics with clinical covariates (0.83 [0.83-0.84]). CONCLUSION: In the decision-making process for bowel resection, disease-specific patient-reported outcome measures may be useful to identify patients with Crohn's disease with poor quality of life and promote a shared understanding of personalized burden.

Identifying antinuclear antibody positive individuals at risk for developing systemic autoimmune disease: development and validation of a real-time risk model.

Objective: Positive antinuclear antibodies (ANAs) cause diagnostic dilemmas for clinicians. Currently, no tools exist to help clinicians interpret the significance of a positive ANA in individuals without diagnosed autoimmune diseases. We developed and validated a risk model to predict risk of developing autoimmune disease in positive ANA individuals. Methods: Using a de-identified electronic health record (EHR), we randomly chart reviewed 2,000 positive ANA individuals to determine if a systemic autoimmune disease was diagnosed by a rheumatologist. A priori, we considered demographics, billing codes for autoimmune disease-related symptoms, and laboratory values as variables for the risk model. We performed logistic regression and machine learning models using training and validation samples. Results: We assembled training (n = 1030) and validation (n = 449) sets. Positive ANA individuals who were younger, female, had a higher titer ANA, higher platelet count, disease-specific autoantibodies, and more billing codes related to symptoms of autoimmune diseases were all more likely to develop autoimmune diseases. The most important variables included having a disease-specific autoantibody, number of billing codes for autoimmune disease-related symptoms, and platelet count. In the logistic regression model, AUC was 0.83 (95% CI 0.79-0.86) in the training set and 0.75 (95% CI 0.68-0.81) in the validation set. Conclusion: We developed and validated a risk model that predicts risk for developing systemic autoimmune diseases and can be deployed easily within the EHR. The model can risk stratify positive ANA individuals to ensure high-risk individuals receive urgent rheumatology referrals while reassuring low-risk individuals and reducing unnecessary referrals.

Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support.

OBJECTIVES: To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts. MATERIALS AND METHODS: We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness. RESULTS: Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001). CONCLUSION: End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.

Leveraging large language models for generating responses to patient messages-a subjective analysis.

OBJECTIVE: This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal. MATERIALS AND METHODS: Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate fine-tuned models, we used 10 representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness. RESULTS: The dataset consisted of 499 794 pairs of patient messages and corresponding responses from the patient portal, with 5000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider's responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT's responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness. CONCLUSION: This subjective analysis suggests that leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and healthcare providers.

Clinical Decision Support for Hypertension Management in Chronic Kidney Disease: A Randomized Clinical Trial.

Importance: Chronic kidney disease (CKD) affects 37 million adults in the United States, and for patients with CKD, hypertension is a key risk factor for adverse outcomes, such as kidney failure, cardiovascular events, and death. Objective: To evaluate a computerized clinical decision support (CDS) system for the management of uncontrolled hypertension in patients with CKD. Design, Setting, and Participants: This multiclinic, randomized clinical trial randomized primary care practitioners (PCPs) at a primary care network, including 15 hospital-based, ambulatory, and community health center-based clinics, through a stratified, matched-pair randomization approach February 2021 to February 2022. All adult patients with a visit to a PCP in the last 2 years were eligible and those with evidence of CKD and hypertension were included. Intervention: The intervention consisted of a CDS system based on behavioral economic principles and human-centered design methods that delivered tailored, evidence-based recommendations, including initiation or titration of renin-angiotensin-aldosterone system inhibitors. The patients in the control group received usual care from PCPs with the CDS system operating in silent mode. Main Outcomes and Measures: The primary outcome was the change in mean systolic blood pressure (SBP) between baseline and 180 days compared between groups. The primary analysis was a repeated measures linear mixed model, using SBP at baseline, 90 days, and 180 days in an intention-to-treat repeated measures model to account for missing data. Secondary outcomes included blood pressure (BP) control and outcomes such as percentage of patients who received an action that aligned with the CDS recommendations. Results: The study included 174 PCPs and 2026 patients (mean [SD] age, 75.3 [0.3] years; 1223 [60.4%] female; mean [SD] SBP at baseline, 154.0 [14.3] mm Hg), with 87 PCPs and 1029 patients randomized to the intervention and 87 PCPs and 997 patients randomized to usual care. Overall, 1714 patients (84.6%) were treated for hypertension at baseline. There were 1623 patients (80.1%) with an SBP measurement at 180 days. From the linear mixed model, there was a statistically significant difference in mean SBP change in the intervention group compared with the usual care group (change, -14.6 [95% CI, -13.1 to -16.0] mm Hg vs -11.7 [-10.2 to -13.1] mm Hg; P = .005). There was no difference in the percentage of patients who achieved BP control in the intervention group compared with the control group (50.4% [95% CI, 46.5% to 54.3%] vs 47.1% [95% CI, 43.3% to 51.0%]). More patients received an action aligned with the CDS recommendations in the intervention group than in the usual care group (49.9% [95% CI, 45.1% to 54.8%] vs 34.6% [95% CI, 29.8% to 39.4%]; P < .001). Conclusions and Relevance: These findings suggest that implementing this computerized CDS system could lead to improved management of uncontrolled hypertension and potentially improved clinical outcomes at the population level for patients with CKD. Trial Registration: ClinicalTrials.gov Identifier: NCT03679247.

Leveraging explainable artificial intelligence to optimize clinical decision support.

OBJECTIVE: To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. METHODS: We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. RESULTS: The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. CONCLUSION: We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.