Person-based design and evaluation of MIA, a digital medical interview assistant for radiology.

Introduction: Radiologists frequently lack direct patient contact due to time constraints. Digital medical interview assistants aim to facilitate the collection of health information. In this paper, we propose leveraging conversational agents to realize a medical interview assistant to facilitate medical history taking, while at the same time offering patients the opportunity to ask questions on the examination. Methods: MIA, the digital medical interview assistant, was developed using a person-based design approach, involving patient opinions and expert knowledge during the design and development with a specific use case in collecting information before a mammography examination. MIA consists of two modules: the interview module and the question answering module (Q&A). To ensure interoperability with clinical information systems, we use HL7 FHIR to store and exchange the results collected by MIA during the patient interaction. The system was evaluated according to an existing evaluation framework that covers a broad range of aspects related to the technical quality of a conversational agent including usability, but also accessibility and security. Results: Thirty-six patients recruited from two Swiss hospitals (Lindenhof group and Inselspital, Bern) and two patient organizations conducted the usability test. MIA was favorably received by the participants, who particularly noted the clarity of communication. However, there is room for improvement in the perceived quality of the conversation, the information provided, and the protection of privacy. The Q&A module achieved a precision of 0.51, a recall of 0.87 and an F-Score of 0.64 based on 114 questions asked by the participants. Security and accessibility also require improvements. Conclusion: The applied person-based process described in this paper can provide best practices for future development of medical interview assistants. The application of a standardized evaluation framework helped in saving time and ensures comparability of results.

A scoping review of large language model based approaches for information extraction from radiology reports.

Radiological imaging is a globally prevalent diagnostic method, yet the free text contained in radiology reports is not frequently used for secondary purposes. Natural Language Processing can provide structured data retrieved from these reports. This paper provides a summary of the current state of research on Large Language Model (LLM) based approaches for information extraction (IE) from radiology reports. We conduct a scoping review that follows the PRISMA-ScR guideline. Queries of five databases were conducted on August 1st 2023. Among the 34 studies that met inclusion criteria, only pre-transformer and encoder-based models are described. External validation shows a general performance decrease, although LLMs might improve generalizability of IE approaches. Reports related to CT and MRI examinations, as well as thoracic reports, prevail. Most common challenges reported are missing validation on external data and augmentation of the described methods. Different reporting granularities affect the comparability and transparency of approaches.

Towards a Reporting Guideline for Studies on Information Extraction from Clinical Texts.

BACKGROUND: The rapid technical progress in the domain of clinical Natural Language Processing and information extraction (IE) has resulted in challenges concerning the comparability and replicability of studies. AIM: This paper proposes a reporting guideline to standardize the description of methodologies and outcomes for studies involving IE from clinical texts. METHODS: The guideline is developed based on the experiences gained from data extraction for a previously conducted scoping review on IE from free-text radiology reports including 34 studies. RESULTS: The guideline comprises the five top-level categories information model, architecture, data, annotation, and outcomes. In total, we define 28 aspects to be reported on in IE studies related to these categories. CONCLUSIONS: The proposed guideline is expected to set a standard for reporting in studies describing IE from clinical text and promote uniformity across the research field. Expected future technological advancements may make regular updates of the guideline necessary. In future research, we plan to develop a taxonomy that clearly defines corresponding value sets as well as integrating both this guideline and the taxonomy by following a consensus-based methodology.

Success Factors for Usability Testing of Digital Health Solutions.

BACKGROUND: The design and development of patient-centered digital health solutions requires user involvement, for example through usability testing. Although there are guidelines for conducting usability tests, there is a lack of knowledge about the technical, human, and organizational factors that influence the success of the tests. OBJECTIVE: To summarize the success factors of usability testing in the context of patient-centered digital health solutions. METHOD: We considered three case studies and collected experiences related to time management, relevance of results and challenges encountered. RESULTS: Success factors relate to participant privacy and data protection, test environment setup, device and application readiness, user comfort and accessibility, test tools and procedures, and adaptability to user limitations. CONCLUSIONS: Small organizational and technical details can have a big impact on the outcome of a usability test. Considering the aspects mentioned in this paper will not only save resources but also the trust of the participating patients.

Large Language Model-Based Evaluation of Medical Question Answering Systems: Algorithm Development and Case Study.

BACKGROUND: Healthcare systems are increasingly resource constrained, leaving less time for important patient-provider interactions. Conversational agents (CAs) could be used to support the provision of information and to answer patients' questions. However, information must be accessible to a variety of patient populations, which requires understanding questions expressed at different language levels. METHODS: This study describes the use of Large Language Models (LLMs) to evaluate predefined medical content in CAs across patient populations. These simulated populations are characterized by a range of health literacy. The evaluation framework includes both fully automated and semi-automated procedures to assess the performance of a CA. RESULTS: A case study in the domain of mammography shows that LLMs can simulate questions from different patient populations. However, the accuracy of the answers provided varies depending on the level of health literacy. CONCLUSIONS: Our scalable evaluation framework enables the simulation of patient populations with different health literacy levels and helps to evaluate domain specific CAs, thus promoting their integration into clinical practice. Future research aims to extend the framework to CAs without predefined content and to apply LLMs to adapt medical information to the specific (health) literacy level of the user.

Large language model-based information extraction from free-text radiology reports: a scoping review protocol.

INTRODUCTION: Radiological imaging is one of the most frequently performed diagnostic tests worldwide. The free-text contained in radiology reports is currently only rarely used for secondary use purposes, including research and predictive analysis. However, this data might be made available by means of information extraction (IE), based on natural language processing (NLP). Recently, a new approach to NLP, large language models (LLMs), has gained momentum and continues to improve performance of IE-related tasks. The objective of this scoping review is to show the state of research regarding IE from free-text radiology reports based on LLMs, to investigate applied methods and to guide future research by showing open challenges and limitations of current approaches. To our knowledge, no systematic or scoping review of IE from radiology reports based on LLMs has been published. Existing publications are outdated and do not comprise LLM-based methods. METHODS AND ANALYSIS: This protocol is designed based on the JBI Manual for Evidence Synthesis, chapter 11.2: 'Development of a scoping review protocol'. Inclusion criteria and a search strategy comprising four databases (PubMed, IEEE Xplore, Web of Science Core Collection and ACM Digital Library) are defined. Furthermore, we describe the screening process, data charting, analysis and presentation of extracted data. ETHICS AND DISSEMINATION: This protocol describes the methodology of a scoping literature review and does not comprise research on or with humans, animals or their data. Therefore, no ethical approval is required. After the publication of this protocol and the conduct of the review, its results are going to be published in an open access journal dedicated to biomedical informatics/digital health.

Exploring the Evolution of Social Media in Mental Health Interventions: A Mapping Review.

BACKGROUND: With the rise of social media, social media use for delivering mental health interventions has become increasingly popular. However, there is no comprehensive overview available on how this field developed over time. OBJECTIVES: The objective of this paper is to provide an overview over time of the use of social media for delivering mental health interventions. Specifically, we examine which mental health conditions and target groups have been targeted, and which social media channels or tools have been used since this topic first appeared in research. METHODS: To provide an overview of the use of social media for mental health interventions, we conducted a search for studies in four databases (PubMed; ACM Digital Library; PsycInfo; and CINAHL) and two trial registries (Clinicaltrials.gov; and Cochranelibrary.com). A sample of representative keywords related to mental health and social media was used for that search. Automatic text analysis methods (e.g., BERTopic analysis, word clouds) were applied to identify topics, and to extract target groups and types of social media. RESULTS: A total of 458 studies were included in this review (n=228 articles, and n=230 registries). Anxiety and depression were the most frequently mentioned conditions in titles of both articles and registries. BERTopic analysis identified depression and anxiety as the main topics, as well as several addictions (including gambling, alcohol, and smoking). Mental health and women's research were highlighted as the main targeted topics of these studies. The most frequently targeted groups were "adults" (39.5%) and "parents" (33.4%). Facebook, WhatsApp, messenger platforms in general, Instagram, and forums were the most frequently mentioned tools in these interventions. CONCLUSIONS: We learned that research interest in social media-based interventions in mental health is increasing, particularly in the last two years. A variety of tools have been studied, and trends towards forums and Facebook show that tools allowing for more content are preferred for mental health interventions. Future research should assess which social media tools are best suited in terms of clinical outcomes. Additionally, we conclude that natural language processing tools can help in studying trends in research on a particular topic.

Designing a Digital Medical Interview Assistant for Radiology.

Radiologists rarely interact with the patients whose radiological images they are reviewing due to time and resource constraints. However, relevant information about the patient's medical history could improve reporting performance and quality. In this work, our objective was to collect requirements for a digital medical interview assistant (DMIA) that collects the medical history from patients by means of a conversational agent and structures as well as provides the collected data to radiologists. Requirements were gathered based on a narrative literature review, a patient questionnaire and input from a radiologist. Based on these results, a system architecture for the DMIA was developed. 37 functional and 17 non-functional requirements were identified. The resulting architecture comprises five components, namely Chatbot, Natural language processing (NLP), Administration, Content Definition and Workflow Engine. To be able to quickly adapt the chatbot content according to the information needs of a specific radiological examination, there is a need for developing a sustainable process for the content generation that considers standardized data modelling as well as rewording of clinical language into consumer health vocabulary understandable to a diverse patient user group.

What Do Autistic People Discuss on Twitter? An Approach Using BERTopic Modelling.

Social media provide easy ways to autistic individuals to communicate and to make their voices heard. The objective of this paper is to identify the main themes that are being discussed by autistic people on Twitter. We collected a sample of tweets containing the hashtag #ActuallyAutistic during the period 10/02/2022 and 14/09/2022. To identify the most discussed topics, BERTopic modelling was applied. We manually grouped the detected topics into 6 major themes using inductive content analysis: 1) General aspects of autism and experiences of autistic individuals; 2) Autism awareness, pride and funding; 3) Interventions, mostly related to Applied Behavior Analysis; 4) Reactions and expressions; 5) Everyday life as an autistic (lifelong condition, work, housing…); and 6) Symbols and characteristics. The majority of tweets were presenting general aspects and experiences as autistic individuals; raising awareness; and about their dissatisfaction with some interventions. The identification of autistic individuals' main discussion themes could help to develop meaningful public health agendas and research involving and addressed to autistic individuals.

User-Centered Design of a Speech-Based Application to Support Caregivers.

The shortage of skilled nursing personnel is - among other reasons - due to the low attractiveness of the profession, comprising high workloads and atypical working hours. Studies show that speech-based documentation systems increase documentation efficiency and satisfaction of physicians. This paper describes the development process of a speech-based application to support nurses, according to the user-centered design approach. User requirements were collected based on interviews (n=6) as well as observations (n=6) in three institutions and were evaluated by means of qualitative content analysis. A prototype of the derived system architecture was implemented. Based on a usability test (n=3), further potentials for improvement were determined. The resulting application enables nurses to dictate personal notes, share them with colleagues and transmit notes to the existing documentation system. We conclude that the user-centered approach ensures the extensive consideration of the nursing staff's requirements and shall be continued for further development.

Sentiment analysis of clinical narratives: A scoping review.

A clinical sentiment is a judgment, thought or attitude promoted by an observation with respect to the health of an individual. Sentiment analysis has drawn attention in the healthcare domain for secondary use of data from clinical narratives, with a variety of applications including predicting the likelihood of emerging mental illnesses or clinical outcomes. The current state of research has not yet been summarized. This study presents results from a scoping review aiming at providing an overview of sentiment analysis of clinical narratives in order to summarize existing research and identify open research gaps. The scoping review was carried out in line with the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guideline. Studies were identified by searching 4 electronic databases (e.g., PubMed, IEEE Xplore) in addition to conducting backward and forward reference list checking of the included studies. We extracted information on use cases, methods and tools applied, used datasets and performance of the sentiment analysis approach. Of 1,200 citations retrieved, 29 unique studies were included in the review covering a period of 8 years. Most studies apply general domain tools (e.g. TextBlob) and sentiment lexicons (e.g. SentiWordNet) for realizing use cases such as prediction of clinical outcomes; others proposed new domain-specific sentiment analysis approaches based on machine learning. Accuracy values between 71.5-88.2% are reported. Data used for evaluation and test are often retrieved from MIMIC databases or i2b2 challenges. Latest developments related to artificial neural networks are not yet fully considered in this domain. We conclude that future research should focus on developing a gold standard sentiment lexicon, adapted to the specific characteristics of clinical narratives. Efforts have to be made to either augment existing or create new high-quality labeled data sets of clinical narratives. Last, the suitability of state-of-the-art machine learning methods for natural language processing and in particular transformer-based models should be investigated for their application for sentiment analysis of clinical narratives.

Digitizing the ECG Workflow - A State-of-the-Art Analysis.

ECG is one of the most common examinations in hospitals, e.g. for diagnosing cardiovascular diseases - the most frequent cause of death worldwide. Goal of this paper was to identify and describe the typical digitized workflow, IT systems and data formats for ECG in hospitals: A survey on current ECG-data management practices was conducted with four German speaking hospitals. A generic model of ECG data management was drafted. Today, these hospitals do not use DICOM as exchange format nor do they implement IHE profiles such as REWF for the ECG. Reasons include missing IT infrastructure such as Master Patient Index or electronic archive. ECG data management could be improved at different levels, with the chance to reduce error sources and to improves in patient safety. Storage of ECG raw data promises better diagnoses based on big data and machine learning technologies.

Assessing and Improving the Usability of the Medical Data Models Portal.

Case report forms (CRF) specify data definitions and encodings for data to be collected in clinical trials. To enable exchange of data definitions and in this way to avoid creation of variants of CRF for similar study designs, the Medical Data Model portal (MDM) has been developed since 2011. This work aims at studying the usability of the MDM portal. We identify issues that hamper its adoption by researchers in order to derive measurements for improving it. We selected relevant tools (e.g. Nibbler, Hotjar, SUPR-Q) for usability testing and generated a structured test protocol. More specifically, the portal was assessed by means of a static analysis, user analysis (n=10), a usability test (n=10) and statistical evaluations. Regarding accessibility and technology, the static code analysis resulted in high scores. Presentation of information and functions as well as interaction with the portal still has to be improved: The results show that only limited functions of the webpage are used regularly and some user navigation errors occur due to the portal's design. In total, six major problems were identified which will be addressed in future. A continuous evaluation using the same structured test protocol allows to continuously measure the website quality, to compare it after changes have been implemented and in this way, to realise a continuous improvement. The effort for a repeated evaluation of the same evaluation with 10 persons is estimated with 10 hours.