Potential of Large Language Models in Health Care: Delphi Study.

BACKGROUND: A large language model (LLM) is a machine learning model inferred from text data that captures subtle patterns of language use in context. Modern LLMs are based on neural network architectures that incorporate transformer methods. They allow the model to relate words together through attention to multiple words in a text sequence. LLMs have been shown to be highly effective for a range of tasks in natural language processing (NLP), including classification and information extraction tasks and generative applications. OBJECTIVE: The aim of this adapted Delphi study was to collect researchers' opinions on how LLMs might influence health care and on the strengths, weaknesses, opportunities, and threats of LLM use in health care. METHODS: We invited researchers in the fields of health informatics, nursing informatics, and medical NLP to share their opinions on LLM use in health care. We started the first round with open questions based on our strengths, weaknesses, opportunities, and threats framework. In the second and third round, the participants scored these items. RESULTS: The first, second, and third rounds had 28, 23, and 21 participants, respectively. Almost all participants (26/28, 93% in round 1 and 20/21, 95% in round 3) were affiliated with academic institutions. Agreement was reached on 103 items related to use cases, benefits, risks, reliability, adoption aspects, and the future of LLMs in health care. Participants offered several use cases, including supporting clinical tasks, documentation tasks, and medical research and education, and agreed that LLM-based systems will act as health assistants for patient education. The agreed-upon benefits included increased efficiency in data handling and extraction, improved automation of processes, improved quality of health care services and overall health outcomes, provision of personalized care, accelerated diagnosis and treatment processes, and improved interaction between patients and health care professionals. In total, 5 risks to health care in general were identified: cybersecurity breaches, the potential for patient misinformation, ethical concerns, the likelihood of biased decision-making, and the risk associated with inaccurate communication. Overconfidence in LLM-based systems was recognized as a risk to the medical profession. The 6 agreed-upon privacy risks included the use of unregulated cloud services that compromise data security, exposure of sensitive patient data, breaches of confidentiality, fraudulent use of information, vulnerabilities in data storage and communication, and inappropriate access or use of patient data. CONCLUSIONS: Future research related to LLMs should not only focus on testing their possibilities for NLP-related tasks but also consider the workflows the models could contribute to and the requirements regarding quality, integration, and regulations needed for successful implementation in practice.

Will Generative AI Tools Improve Access to Reliable Health Information?

This Medical News article is an interview with JAMA Editor in Chief Kirsten Bibbins-Domingo and Virologist Davey Smith, head of the Division of Infectious Diseases and Global Public Health at the University of California, San Diego.

Visualizing Study Profiles Using the GMDS Competency Catalog for Medical Informatics Programs.

BACKGROUND: Medical informatics programs cover a variety of topics. OBJECTIVES: To test the utility of the GMDS medical informatics competency catalog in comparing programs by developing study profiles. METHODS: Coverage of 234 competencies is recorded and visualized in a spider diagram. RESULTS: Spider diagrams allow visualizing various study profiles. CONCLUSION: The GMDS catalog seems useful for comparing medical informatics study programs, e.g., for interested students, employers, or accreditation reviewers.

Understanding IT in Healthcare: Relevance and Training Needs of IT in Private Medical Practice.

BACKGROUND: The integration of Information Technology (IT) into private medical practice is crucial in modern healthcare. Physicians managing office-related IT without proper knowledge risk operational inefficiencies and security. OBJECTIVES: This study determines the relevance of specific IT topics in medical practice and identifies the training needs of physicians for enhancing IT competencies in healthcare. METHODS: In March 2023 a cross-sectional online survey was conducted with physicians comprising nine IT-related topics in Tyrol, Austria. RESULTS: The survey results highlighted a strong perceived relevance and high demand for IT education among physicians working in their medical practice, especially in areas of core medical IT and security. The majority of responses indicated high relevance (76.7%) and high demand (69.7%) for IT topics in medical practice. CONCLUSION: The findings underscore a significant need for targeted IT training and support in medical practices, particularly in areas related to the medical practice and security. Addressing these needs could lead to improved healthcare delivery and better management of technological resources in the healthcare sector.

Designing for caregiving networks: a case study of primary caregivers of children with medical complexity.

OBJECTIVE: The study aimed to characterize the experiences of primary caregivers of children with medical complexity (CMC) in engaging with other members of the child's caregiving network, thereby informing the design of health information technology (IT) for the caregiving network. Caregiving networks include friends, family, community members, and other trusted individuals who provide resources, information, health, or childcare. MATERIALS AND METHODS: We performed a secondary analysis of two qualitative studies. Primary studies conducted semi-structured interviews (n = 50) with family caregivers of CMC. Interviews were held in the Midwest (n = 30) and the mid-Atlantic region (n = 20). Interviews were transcribed verbatim for thematic analysis. Emergent themes were mapped to implications for the design of future health IT. RESULTS: Thematic analysis identified 8 themes characterizing a wide range of primary caregivers' experiences in constructing, managing, and ensuring high-quality care delivery across the caregiving network. DISCUSSION: Findings evidence a critical need to create flexible and customizable tools designed to support hiring/training processes, coordinating daily care across the caregiving network, communicating changing needs and care updates across the caregiving network, and creating contingency plans for instances where caregivers are unavailable to provide care to the CMC. Informaticists should additionally design accessible platforms that allow primary caregivers to connect with and learn from other caregivers while minimizing exposure to sensitive or emotional content as indicated by the user. CONCLUSION: This article contributes to the design of health IT for CMC caregiving networks by uncovering previously underrecognized needs and experiences of CMC primary caregivers and drawing direct connections to design implications.

Sociotechnical feasibility of natural language processing-driven tools in clinical trial eligibility prescreening for Alzheimer's disease and related dementias.

BACKGROUND: Alzheimer's disease and related dementias (ADRD) affect over 55 million globally. Current clinical trials suffer from low recruitment rates, a challenge potentially addressable via natural language processing (NLP) technologies for researchers to effectively identify eligible clinical trial participants. OBJECTIVE: This study investigates the sociotechnical feasibility of NLP-driven tools for ADRD research prescreening and analyzes the tools' cognitive complexity's effect on usability to identify cognitive support strategies. METHODS: A randomized experiment was conducted with 60 clinical research staff using three prescreening tools (Criteria2Query, Informatics for Integrating Biology and the Bedside [i2b2], and Leaf). Cognitive task analysis was employed to analyze the usability of each tool using the Health Information Technology Usability Evaluation Scale. Data analysis involved calculating descriptive statistics, interrater agreement via intraclass correlation coefficient, cognitive complexity, and Generalized Estimating Equations models. RESULTS: Leaf scored highest for usability followed by Criteria2Query and i2b2. Cognitive complexity was found to be affected by age, computer literacy, and number of criteria, but was not significantly associated with usability. DISCUSSION: Adopting NLP for ADRD prescreening demands careful task delegation, comprehensive training, precise translation of eligibility criteria, and increased research accessibility. The study highlights the relevance of these factors in enhancing NLP-driven tools' usability and efficacy in clinical research prescreening. CONCLUSION: User-modifiable NLP-driven prescreening tools were favorably received, with system type, evaluation sequence, and user's computer literacy influencing usability more than cognitive complexity. The study emphasizes NLP's potential in improving recruitment for clinical trials, endorsing a mixed-methods approach for future system evaluation and enhancements.

Incorporating ChatGPT in Medical Informatics Education: Mixed Methods Study on Student Perceptions and Experiential Integration Proposals.

BACKGROUND: The integration of artificial intelligence (AI) technologies, such as ChatGPT, in the educational landscape has the potential to enhance the learning experience of medical informatics students and prepare them for using AI in professional settings. The incorporation of AI in classes aims to develop critical thinking by encouraging students to interact with ChatGPT and critically analyze the responses generated by the chatbot. This approach also helps students develop important skills in the field of biomedical and health informatics to enhance their interaction with AI tools. OBJECTIVE: The aim of the study is to explore the perceptions of students regarding the use of ChatGPT as a learning tool in their educational context and provide professors with examples of prompts for incorporating ChatGPT into their teaching and learning activities, thereby enhancing the educational experience for students in medical informatics courses. METHODS: This study used a mixed methods approach to gain insights from students regarding the use of ChatGPT in education. To accomplish this, a structured questionnaire was applied to evaluate students' familiarity with ChatGPT, gauge their perceptions of its use, and understand their attitudes toward its use in academic and learning tasks. Learning outcomes of 2 courses were analyzed to propose ChatGPT's incorporation in master's programs in medicine and medical informatics. RESULTS: The majority of students expressed satisfaction with the use of ChatGPT in education, finding it beneficial for various purposes, including generating academic content, brainstorming ideas, and rewriting text. While some participants raised concerns about potential biases and the need for informed use, the overall perception was positive. Additionally, the study proposed integrating ChatGPT into 2 specific courses in the master's programs in medicine and medical informatics. The incorporation of ChatGPT was envisioned to enhance student learning experiences and assist in project planning, programming code generation, examination preparation, workflow exploration, and technical interview preparation, thus advancing medical informatics education. In medical teaching, it will be used as an assistant for simplifying the explanation of concepts and solving complex problems, as well as for generating clinical narratives and patient simulators. CONCLUSIONS: The study's valuable insights into medical faculty students' perspectives and integration proposals for ChatGPT serve as an informative guide for professors aiming to enhance medical informatics education. The research delves into the potential of ChatGPT, emphasizes the necessity of collaboration in academic environments, identifies subject areas with discernible benefits, and underscores its transformative role in fostering innovative and engaging learning experiences. The envisaged proposals hold promise in empowering future health care professionals to work in the rapidly evolving era of digital health care.

Perceptions of students and faculty on NCAAA-accredited health informatics programs in Saudi Arabia: an evaluative study.

BACKGROUND: As the healthcare sector becomes increasingly reliant on technology, it is crucial for universities to offer bachelor's degrees in health informatics (HI). HI professionals bridge the gap between IT and healthcare, ensuring that technology complements patient care and clinical workflows; they promote enhanced patient outcomes, support clinical research, and uphold data security and privacy standards. This study aims to evaluate accredited HI academic programs in Saudi Arabia. METHODS: This study employed a quantitative, descriptive, cross-sectional design utilising a self-reported electronic questionnaire consisting of predetermined items and response alternatives. Probability-stratified random sampling was also performed. RESULT: The responses rates were 39% (n = 241) for students and 62% (n = 53) for faculty members. While the participants expressed different opinions regarding the eight variables being examined, the faculty members and students generally exhibited a strong level of consensus on many variables. A notable association was observed between facilities and various other characteristics, including student engagement, research activities, admission processes, and curriculum. Similarly, a notable correlation exists between student engagement and the curriculum in connection to research, attrition, the function of faculty members, and academic outcomes. CONCLUSION: While faculty members and students hold similar views about the institution and its offerings, certain areas of divergence highlight the distinct perspectives and priorities of each group. The perception disparity between students and faculty in areas such as admission, faculty roles, and internships sheds light on areas of improvement and alignment for universities.

Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network.

BACKGROUND: Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability. METHODS: Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts. RESULTS: Model development included 41 929 patients, 14.6% with POU. The external validation included 31 932 (UK), 23 100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD = 0.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P < .05). CONCLUSIONS: Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.

Job search strategies and early careers of clinical informatics fellowship alumni (2016-2022).

OBJECTIVE: To report on clinical informatics (CI) fellows' job search and early careers. MATERIALS AND METHODS: In the summer of 2022, we performed a voluntary and anonymous survey of 242 known clinical informatics fellowship alumni from 2016 to 2022. The survey included questions about their initial job search process; first job, salary, and informatics time after training; and early career progression over the first 1-6 years after fellowship. RESULTS: Nearly half (101, 41.7%) responded to the survey. Median informatics time was 50%; most were compensated similar/better than a purely clinical position. Most reported CI fellowship significantly impacted their career, were satisfied with their first and current job after training, and provided advice for current fellows and CI education leaders. Graduates in 2022 had a median job search of 5 months, beginning 3-15 months before graduation; most had a position created for them. Nearly all graduates from 2016-2021 (61, 93.8%) had at least one change in roles/benefits since finishing training, with a trend for increased informatics time and salary. DISCUSSION: There was a wide variety of roles, salary, and funding sources for CI positions. This highlights some of the unique challenges CI fellows face and the importance of networking. These results will help CI education leaders, fellows, alumni, and prospective fellowship applicants. CONCLUSION: Graduates felt that CI fellowship had a significant impact on their career, were pleased with their first jobs and early career trajectory. Continued follow-up of the experience of new graduates and alumni is needed to assess emerging patterns over time.

Conceptual design of a generic data harmonization process for OMOP common data model.

BACKGROUND: To gain insight into the real-life care of patients in the healthcare system, data from hospital information systems and insurance systems are required. Consequently, linking clinical data with claims data is necessary. To ensure their syntactic and semantic interoperability, the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) from the Observational Health Data Sciences and Informatics (OHDSI) community was chosen. However, there is no detailed guide that would allow researchers to follow a generic process for data harmonization, i.e. the transformation of local source data into the standardized OMOP CDM format. Thus, the aim of this paper is to conceptualize a generic data harmonization process for OMOP CDM. METHODS: For this purpose, we conducted a literature review focusing on publications that address the harmonization of clinical or claims data in OMOP CDM. Subsequently, the process steps used and their chronological order as well as applied OHDSI tools were extracted for each included publication. The results were then compared to derive a generic sequence of the process steps. RESULTS: From 23 publications included, a generic data harmonization process for OMOP CDM was conceptualized, consisting of nine process steps: dataset specification, data profiling, vocabulary identification, coverage analysis of vocabularies, semantic mapping, structural mapping, extract-transform-load-process, qualitative and quantitative data quality analysis. Furthermore, we identified seven OHDSI tools which supported five of the process steps. CONCLUSIONS: The generic data harmonization process can be used as a step-by-step guide to assist other researchers in harmonizing source data in OMOP CDM.

Comparing "Meaningful Use" of Health Information Technology in Pennsylvania: Electronic Prescription Rates of Metropolitan and Rural Counties.

The Health Information Technology for Economic and Clinical Health Act incentivized the adoption of electronic health records (EHRs). Health systems looked to leverage technology to assist in serving populations in health professional shortage areas. Qualitative research points to EHR usability as a source of health inequities in rural settings, making the challenges of EHR usage a subject of interest. Pennsylvania offers a model for investigating rural health infrastructure with it having the third largest rural population in the United States. This study analyzed the adoption of Electronic Prescribing in the 67 Pennsylvania (PA) counties. Physician adoption and usage data for PA and the United States were compared using a t-test to establish a basis for comparison. PA counties were categorized using the United States Department of Agriculture (USDA)'s Rural-Urban Commuting Areas (RUCAs) system. Surescript use percentages were plotted against the RUCA scores of each PA county to create a polynomial regression model. PA office-based physicians, on average, utilize e-prescription tools at the same rate as the national average with 59% of practices utilizing Surescripts as of 2013. There was no significant correlation between Surescript usage and the rural/urban classification of counties in Pennsylvania (R-squared value of 0.06). Pennsylvania was able to adopt health information technology (HIT) infrastructure at the same rate as the national average. Rural and metropolitan definitions do not correlate to meaningful use of HIT, thus usability of HIT cannot be tied to health outcomes. Future studies looking at specific forms of HIT and their ability to decrease the burden of administrative work for clinicians.