Prompt Engineering Paradigms for Medical Applications: Scoping Review.

BACKGROUND: Prompt engineering, focusing on crafting effective prompts to large language models (LLMs), has garnered attention for its capabilities at harnessing the potential of LLMs. This is even more crucial in the medical domain due to its specialized terminology and language technicity. Clinical natural language processing applications must navigate complex language and ensure privacy compliance. Prompt engineering offers a novel approach by designing tailored prompts to guide models in exploiting clinically relevant information from complex medical texts. Despite its promise, the efficacy of prompt engineering in the medical domain remains to be fully explored. OBJECTIVE: The aim of the study is to review research efforts and technical approaches in prompt engineering for medical applications as well as provide an overview of opportunities and challenges for clinical practice. METHODS: Databases indexing the fields of medicine, computer science, and medical informatics were queried in order to identify relevant published papers. Since prompt engineering is an emerging field, preprint databases were also considered. Multiple data were extracted, such as the prompt paradigm, the involved LLMs, the languages of the study, the domain of the topic, the baselines, and several learning, design, and architecture strategies specific to prompt engineering. We include studies that apply prompt engineering-based methods to the medical domain, published between 2022 and 2024, and covering multiple prompt paradigms such as prompt learning (PL), prompt tuning (PT), and prompt design (PD). RESULTS: We included 114 recent prompt engineering studies. Among the 3 prompt paradigms, we have observed that PD is the most prevalent (78 papers). In 12 papers, PD, PL, and PT terms were used interchangeably. While ChatGPT is the most commonly used LLM, we have identified 7 studies using this LLM on a sensitive clinical data set. Chain-of-thought, present in 17 studies, emerges as the most frequent PD technique. While PL and PT papers typically provide a baseline for evaluating prompt-based approaches, 61% (48/78) of the PD studies do not report any nonprompt-related baseline. Finally, we individually examine each of the key prompt engineering-specific information reported across papers and find that many studies neglect to explicitly mention them, posing a challenge for advancing prompt engineering research. CONCLUSIONS: In addition to reporting on trends and the scientific landscape of prompt engineering, we provide reporting guidelines for future studies to help advance research in the medical field. We also disclose tables and figures summarizing medical prompt engineering papers available and hope that future contributions will leverage these existing works to better advance the field.

[Dolodoc: a virtual pain management coach]. / Dolodoc : un coach virtuel pour mieux gérer la douleur chronique.

Dolodoc is a mobile application aimed at improving autonomy and quality of life for individuals living with chronic pain. Designed as a virtual coach, it offers counseling according to 7 important dimensions of quality of life. Activities, pain and fulfillment of the 7 dimensions of quality of life can be recorded in the application. Moreover, a report can be exported to enhance patient monitoring during clinical interactions. Dolodoc was developed with a user-centered approach and is based on scientific evidence related to the self-management of chronic pain. Indeed, counseling by the coach is based on a multimodal strategy, incorporating elements of physical activity, pacing, positive psychology, and relaxation, among others. Overall, Dolodoc is an innovation that can be used in various clinical settings with an individualized approach.

Dolodoc est une application ayant pour but d'améliorer l'autonomie et la qualité de vie des personnes vivant avec la douleur chronique. Conçue comme un coach virtuel, elle propose des conseils ainsi qu'un suivi d'activités se référant à 7 dimensions importantes pour la qualité de vie. Ces éléments sont consignables dans l'application et un rapport peut être exporté pour agrémenter le suivi du patient. Dolodoc a été développé selon une approche centrée sur l'utilisateur et se base sur des preuves scientifiques en lien avec l'autogestion des douleurs chroniques. En effet, les conseils sont multimodaux et intègrent, entre autres, l'activité physique, le pacing, la psychologie positive et la relaxation. Disponible gratuitement, Dolodoc est une innovation dont l'utilisation individualisée peut s'adapter à différents contextes cliniques.

MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French.

Generative Large Language Models (LLMs) have become ubiquitous in various fields, including healthcare and medicine. Consequently, there is growing interest in leveraging LLMs for medical applications, leading to the emergence of novel models daily. However, evaluation and benchmarking frameworks for LLMs are scarce, particularly those tailored for medical French. To address this gap, we introduce a minimal benchmark consisting of 114 open questions designed to assess the medical capabilities of LLMs in French. The proposed benchmark encompasses a wide range of medical domains, reflecting real-world clinical scenarios' complexity. A preliminary validation involved testing seven widely used LLMs with a parameter size of 7 billion. Results revealed significant variability in performance, emphasizing the importance of rigorous evaluation before deploying LLMs in medical settings. In conclusion, we present a novel and valuable resource for rapidly evaluating LLMs in medical French. By promoting greater accountability and standardization, this benchmark has the potential to enhance trustworthiness and utility in harnessing LLMs for medical applications.

SIMpat: A Synthetic Benchmark for Similarity Metrics on Patient Representations.

Similarity and clustering tasks based on data extracted from electronic health records on the patient level suffer from the curse of dimensionality and the lack of inter-patient data comparability. Indeed, for many health institutions, there are many more variables, and ways of expressing those variables to represent patients than patients sharing the same set of data. To lower redundancy and increase interoperability one strategy is to map data to semantic-driven representations through medical knowledge graphs such as SNOMED-CT. However, patient similarity metrics based on this knowledge-graph information lack quantitative evaluation and comparisons with pure data-driven methods. The reasons are twofold, firstly, it is hard to conceptually assess and formalize a gold-standard similarity between patients resulting in poor inter-annotator agreement in qualitative evaluations. Secondly, the community has been lacking a clear benchmark to compare existing metrics developed by scientific communities coming from various fields such as ontology, data science, and medical informatics. This study proposes to leverage the known challenges of evaluating patient similarities by proposing SIMpat, a synthetic benchmark to quantitatively evaluate available metrics, based on controlled cohorts, which could later be used to assess their sensibility regarding aspects such as the sparsity of variables or specificities of patient disease patterns.

Efficient Clinical Information Extraction from Breast Radiology Reports in French.

Radiology reports contain crucial patient information, in addition to images, that can be automatically extracted for secondary uses such as clinical support and research for diagnosis. We tested several classifiers to classify 1,218 breast MRI reports in French from two Swiss clinical centers. Logistic regression performed better for both internal (accuracy > 0.95 and macro-F1 > 0.86) and external data (accuracy > 0.81 and macro-F1 > 0.41). Automating this task will facilitate efficient extraction of targeted clinical parameters and provide a good basis for future annotation processes through automatic pre-annotation.

Scalar Values in SNOMED CT: A Proposed Extension.

Representing numeric values such as scalars holds great importance for accurately depicting clinical data. While the result value itself will always be represented using an integer, decimal, or other scalar format, it needs to be linked to its corresponding data element. In SNOMED CT, as in most other terminology systems, this is done through an attribute relationship. While some scalar values are already included in this way, they only represent a small fraction of possibilities. Our intention is to expand the scope of scalar representation by validating new attributes using a previously established method. The result is a list of five attributes validated for local representation of scalar values, improving semantic representation and interoperability.

Unsupervised Extraction of Body-Text from Clinical PDF Documents.

Automatic extraction of body-text within clinical PDF documents is necessary to enhance downstream NLP tasks but remains a challenge. This study presents an unsupervised algorithm designed to extract body-text leveraging large volume of data. Using DBSCAN clustering over aggregate pages, our method extracts and organize text blocks using their content and coordinates. Evaluation results demonstrate precision scores ranging from 0.82 to 0.98, recall scores from 0.62 to 0.94, and F1-scores from 0.71 to 0.96 across various medical specialty sources. Future work includes dynamic parameter adjustments for improved accuracy and using larger datasets.

An eHealth Coaching Solution to Improve Transitional Care of Seniors with Heart Failure: Long-Field Trial.

Heart failure is the leading reason for seniors being admitted to hospitals. Over half of the elderly individuals diagnosed with heart failure find themselves readmitted to hospitals within a span of six months. This recurrence is associated with inadequate adherence to medical treatment and recommendations, underscoring the necessity for support systems that aid seniors in better adhering to post-hospitalization instructions. The objective of this study is to evaluate the usability, usefulness and added value of the core functionalities within the H2HCare Ambient Assisted Living developed system that was evaluated with 11 participants over a long field trial of three months. Our assessment encompassed the examination of their Quality of Life as well as the usability and efficacy of the system. Overall, participants reported finding the system user-friendly, beneficial, and conducive to enhanced disease management. Improvements include tailoring the alarm system to patient standards and using a questionnaire to assess situation urgency.

Automatic Classification of Conclusions from Multi-Tracer Reports of PET Brain Imaging in Cognitive Impairment.

The goal of this paper is to build an automatic way to interpret conclusions from brain molecular imaging reports performed for investigation of cognitive disturbances (FDG, Amyloid and Tau PET) by comparing several traditional machine learning (ML) techniques-based text classification methods. Two purposes are defined: to identify positive or negative results in all three modalities, and to extract diagnostic impressions for Alzheimer's Disease (AD), Fronto-Temporal Dementia (FTD), Lewy Bodies Dementia (LBD) based on metabolism of perfusion patterns. A dataset was created by manual parallel annotation of 1668 conclusions of reports from the Nuclear Medicine and Molecular Imaging Division of Geneva University Hospitals. The 6 Machine Learning (ML) algorithms (Support Vector Machine (Linear and Radial Basis function), Naive Bayes, Logistic Regression, Random Forrest, and K-Nearest Neighbors) were trained and evaluated with a 5-fold cross-validation scheme to assess their performance and generalizability. The best classifier was SVM showing the following accuracies: FDG (0.97), Tau (0.94), Amyloid (0.98), Oriented Diagnostic (0.87 for a diagnosis among AD, FTD, LBD, undetermined, other), paving the way for a paradigm shift in the field of data handling in nuclear medicine research.

Beyond Tokens: Fair Evaluation of French Large Language Models for Clinical Named Entity Recognition.

Named Entity Recognition (NER) models based on Transformers have gained prominence for their impressive performance in various languages and domains. This work delves into the often-overlooked aspect of entity-level metrics and exposes significant discrepancies between token and entity-level evaluations. The study utilizes a corpus of synthetic French oncological reports annotated with entities representing oncological morphologies. Four different French BERT-based models are fine-tuned for token classification, and their performance is rigorously assessed at both token and entity-level. In addition to fine-tuning, we evaluate ChatGPT's ability to perform NER through prompt engineering techniques. The findings reveal a notable disparity in model effectiveness when transitioning from token to entity-level metrics, highlighting the importance of comprehensive evaluation methodologies in NER tasks. Furthermore, in comparison to BERT, ChatGPT remains limited when it comes to detecting advanced entities in French.

A Novel Method and Python Library for ECG Signal Quality Assessment.

Electrocardiogram (ECG) is one of the reference cardiovascular diagnostic exams. However, the ECG signal is very prone to being distorted through different sources of artifacts that can later interfere with the diagnostic. For this reason, signal quality assessment (SQA) methods that identify corrupted signals are critical to improve the robustness of automatic ECG diagnostic methods. This work presents a review and open-source implementation of different available indices for SQA as well as introducing an index that considers the ECG as a dynamical system. These indices are then used to develop machine learning models which evaluate the quality of the signals. The proposed index along the designed ML models are shown to improve SQA for ECG signals.

Consent as a compositional act - a framework that provides clarity for the retention and use of data.

BACKGROUND: Informed consent is one of the key principles of conducting research involving humans. When research participants give consent, they perform an act in which they utter, write or otherwise provide an authorisation to somebody to do something. This paper proposes a new understanding of the informed consent as a compositional act. This conceptualisation departs from a modular conceptualisation of informed consent procedures. METHODS: This paper is a conceptual analysis that explores what consent is and what it does or does not do. It presents a framework that explores the basic elements of consent and breaks it down into its component parts. It analyses the consent act by first identifying its basic elements, namely: a) data subjects or legal representative that provides the authorisation of consent; b) a specific thing that is being consented to; and c) specific agent(s) to whom the consent is given. RESULTS: This paper presents a framework that explores the basic elements of consent and breaks it down into its component parts. It goes beyond only providing choices to potential research participants; it explains the rationale of those choices or consenting acts that are taking place when speaking or writing an authorisation to do something to somebody. CONCLUSIONS: We argue that by clearly differentiating the goals, the procedures of implementation, and what is being done or undone when one consent, one can better face the challenges of contemporary data-intensive biomedical research, particularly regarding the retention and use of data. Conceptualising consent as a compositional act enhances more efficient communication and accountability and, therefore, could enable more trustworthy acts of consent in biomedical science.

[Applications, techniques, and best practices for using ChatGPT]. / Applications, techniques et bonnes pratiques d'utilisation de ChatGPT.

The future of a machine writing our reports for us could also lead to it carrying out our consultations, a scenario whose relevance is open to debate. Nevertheless, the present offers us new artificial intelligence tools that can support us in our daily activities. The publication in 2017 of Transformers initiated a disruptive revolution by enabling the emergence of major language models, of which ChatGPT is the best known. In view of their growing adoption, the authors felt it would be useful to offer some pragmatic advice on how to improve the use of these tools. In this article, we first look at how ChatGPT works and its potential applications in medicine, before providing a practical guide to using it to get the best results.

Le futur d'une machine rédigeant nos rapports à notre place pourrait également l'amener à effectuer nos consultations, un scénario dont la pertinence reste à débattre. Le présent nous offre néanmoins de nouveaux instruments d'intelligence artificielle qui peuvent nous soutenir dans nos activités quotidiennes. La publication en 2017 des Transformers a initié une révolution disruptive en permettant l'émergence de grands modèles de langages, dont ChatGPT est le plus connu. Face à leur adoption grandissante, il est apparu utile aux auteurs d'apporter quelques conseils pragmatiques pour améliorer l'utilisation de ces outils. Dans cet article, nous abordons d'abord le fonctionnement de ChatGPT, ses applications potentielles en médecine avant de fournir un guide pratique d'utilisation pour en tirer les meilleurs résultats.

Barriers and facilitators of artificial intelligence conception and implementation for breast imaging diagnosis in clinical practice: a scoping review.

OBJECTIVE: Although artificial intelligence (AI) has demonstrated promise in enhancing breast cancer diagnosis, the implementation of AI algorithms in clinical practice encounters various barriers. This scoping review aims to identify these barriers and facilitators to highlight key considerations for developing and implementing AI solutions in breast cancer imaging. METHOD: A literature search was conducted from 2012 to 2022 in six databases (PubMed, Web of Science, CINHAL, Embase, IEEE, and ArXiv). The articles were included if some barriers and/or facilitators in the conception or implementation of AI in breast clinical imaging were described. We excluded research only focusing on performance, or with data not acquired in a clinical radiology setup and not involving real patients. RESULTS: A total of 107 articles were included. We identified six major barriers related to data (B1), black box and trust (B2), algorithms and conception (B3), evaluation and validation (B4), legal, ethical, and economic issues (B5), and education (B6), and five major facilitators covering data (F1), clinical impact (F2), algorithms and conception (F3), evaluation and validation (F4), and education (F5). CONCLUSION: This scoping review highlighted the need to carefully design, deploy, and evaluate AI solutions in clinical practice, involving all stakeholders to yield improvement in healthcare. CLINICAL RELEVANCE STATEMENT: The identification of barriers and facilitators with suggested solutions can guide and inform future research, and stakeholders to improve the design and implementation of AI for breast cancer detection in clinical practice. KEY POINTS: • Six major identified barriers were related to data; black-box and trust; algorithms and conception; evaluation and validation; legal, ethical, and economic issues; and education. • Five major identified facilitators were related to data, clinical impact, algorithms and conception, evaluation and validation, and education. • Coordinated implication of all stakeholders is required to improve breast cancer diagnosis with AI.

Introducing the "AI Language Models in Health Care" Section: Actionable Strategies for Targeted and Wide-Scale Deployment.

The realm of health care is on the cusp of a significant technological leap, courtesy of the advancements in artificial intelligence (AI) language models, but ensuring the ethical design, deployment, and use of these technologies is imperative to truly realize their potential in improving health care delivery and promoting human well-being and safety. Indeed, these models have demonstrated remarkable prowess in generating humanlike text, evidenced by a growing body of research and real-world applications. This capability paves the way for enhanced patient engagement, clinical decision support, and a plethora of other applications that were once considered beyond reach. However, the journey from potential to real-world application is laden with challenges ranging from ensuring reliability and transparency to navigating a complex regulatory landscape. There is still a need for comprehensive evaluation and rigorous validation to ensure that these models are reliable, transparent, and ethically sound. This editorial introduces the new section, titled "AI Language Models in Health Care." This section seeks to create a platform for academics, practitioners, and innovators to share their insights, research findings, and real-world applications of AI language models in health care. The aim is to foster a community that is not only excited about the possibilities but also critically engaged with the ethical, practical, and regulatory challenges that lie ahead.

Patient Information Summarization in Clinical Settings: Scoping Review.

BACKGROUND: Information overflow, a common problem in the present clinical environment, can be mitigated by summarizing clinical data. Although there are several solutions for clinical summarization, there is a lack of a complete overview of the research relevant to this field. OBJECTIVE: This study aims to identify state-of-the-art solutions for clinical summarization, to analyze their capabilities, and to identify their properties. METHODS: A scoping review of articles published between 2005 and 2022 was conducted. With a clinical focus, PubMed and Web of Science were queried to find an initial set of reports, later extended by articles found through a chain of citations. The included reports were analyzed to answer the questions of where, what, and how medical information is summarized; whether summarization conserves temporality, uncertainty, and medical pertinence; and how the propositions are evaluated and deployed. To answer how information is summarized, methods were compared through a new framework "collect-synthesize-communicate" referring to information gathering from data, its synthesis, and communication to the end user. RESULTS: Overall, 128 articles were included, representing various medical fields. Exclusively structured data were used as input in 46.1% (59/128) of papers, text in 41.4% (53/128) of articles, and both in 10.2% (13/128) of papers. Using the proposed framework, 42.2% (54/128) of the records contributed to information collection, 27.3% (35/128) contributed to information synthesis, and 46.1% (59/128) presented solutions for summary communication. Numerous summarization approaches have been presented, including extractive (n=13) and abstractive summarization (n=19); topic modeling (n=5); summary specification (n=11); concept and relation extraction (n=30); visual design considerations (n=59); and complete pipelines (n=7) using information extraction, synthesis, and communication. Graphical displays (n=53), short texts (n=41), static reports (n=7), and problem-oriented views (n=7) were the most common types in terms of summary communication. Although temporality and uncertainty information were usually not conserved in most studies (74/128, 57.8% and 113/128, 88.3%, respectively), some studies presented solutions to treat this information. Overall, 115 (89.8%) articles showed results of an evaluation, and methods included evaluations with human participants (median 15, IQR 24 participants): measurements in experiments with human participants (n=31), real situations (n=8), and usability studies (n=28). Methods without human involvement included intrinsic evaluation (n=24), performance on a proxy (n=10), or domain-specific tasks (n=11). Overall, 11 (8.6%) reports described a system deployed in clinical settings. CONCLUSIONS: The scientific literature contains many propositions for summarizing patient information but reports very few comparisons of these proposals. This work proposes to compare these algorithms through how they conserve essential aspects of clinical information and through the "collect-synthesize-communicate" framework. We found that current propositions usually address these 3 steps only partially. Moreover, they conserve and use temporality, uncertainty, and pertinent medical aspects to varying extents, and solutions are often preliminary.

Multilingual RECIST classification of radiology reports using supervised learning.

Objectives: The objective of this study is the exploration of Artificial Intelligence and Natural Language Processing techniques to support the automatic assignment of the four Response Evaluation Criteria in Solid Tumors (RECIST) scales based on radiology reports. We also aim at evaluating how languages and institutional specificities of Swiss teaching hospitals are likely to affect the quality of the classification in French and German languages. Methods: In our approach, 7 machine learning methods were evaluated to establish a strong baseline. Then, robust models were built, fine-tuned according to the language (French and German), and compared with the expert annotation. Results: The best strategies yield average F1-scores of 90% and 86% respectively for the 2-classes (Progressive/Non-progressive) and the 4-classes (Progressive Disease, Stable Disease, Partial Response, Complete Response) RECIST classification tasks. Conclusions: These results are competitive with the manual labeling as measured by Matthew's correlation coefficient and Cohen's Kappa (79% and 76%). On this basis, we confirm the capacity of specific models to generalize on new unseen data and we assess the impact of using Pre-trained Language Models (PLMs) on the accuracy of the classifiers.

iCHECK-DH: Guidelines and Checklist for the Reporting on Digital Health Implementations.

BACKGROUND: Implementation of digital health technologies has grown rapidly, but many remain limited to pilot studies due to challenges, such as a lack of evidence or barriers to implementation. Overcoming these challenges requires learning from previous implementations and systematically documenting implementation processes to better understand the real-world impact of a technology and identify effective strategies for future implementation. OBJECTIVE: A group of global experts, facilitated by the Geneva Digital Health Hub, developed the Guidelines and Checklist for the Reporting on Digital Health Implementations (iCHECK-DH, pronounced "I checked") to improve the completeness of reporting on digital health implementations. METHODS: A guideline development group was convened to define key considerations and criteria for reporting on digital health implementations. To ensure the practicality and effectiveness of the checklist, it was pilot-tested by applying it to several real-world digital health implementations, and adjustments were made based on the feedback received. The guiding principle for the development of iCHECK-DH was to identify the minimum set of information needed to comprehensively define a digital health implementation, to support the identification of key factors for success and failure, and to enable others to replicate it in different settings. RESULTS: The result was a 20-item checklist with detailed explanations and examples in this paper. The authors anticipate that widespread adoption will standardize the quality of reporting and, indirectly, improve implementation standards and best practices. CONCLUSIONS: Guidelines for reporting on digital health implementations are important to ensure the accuracy, completeness, and consistency of reported information. This allows for meaningful comparison and evaluation of results, transparency, and accountability and informs stakeholder decision-making. i-CHECK-DH facilitates standardization of the way information is collected and reported, improving systematic documentation and knowledge transfer that can lead to the development of more effective digital health interventions and better health outcomes.

Correction: iCHECK-DH: Guidelines and Checklist for the Reporting on Digital Health Implementations.

[This corrects the article DOI: 10.2196/46694.].

"To Err Is Evolution": We Need the Implementation Report to Learn.

JMIR Medical Informatics is pleased to offer implementation reports as a new article type. Implementation reports present real-world accounts of the implementation of health technologies and clinical interventions. This new article type is intended to promote the rapid documentation and dissemination of the perspectives and experiences of those involved in implementing digital health interventions and assessing the effectiveness of digital health projects.