Person-based design and evaluation of MIA, a digital medical interview assistant for radiology.

Introduction: Radiologists frequently lack direct patient contact due to time constraints. Digital medical interview assistants aim to facilitate the collection of health information. In this paper, we propose leveraging conversational agents to realize a medical interview assistant to facilitate medical history taking, while at the same time offering patients the opportunity to ask questions on the examination. Methods: MIA, the digital medical interview assistant, was developed using a person-based design approach, involving patient opinions and expert knowledge during the design and development with a specific use case in collecting information before a mammography examination. MIA consists of two modules: the interview module and the question answering module (Q&A). To ensure interoperability with clinical information systems, we use HL7 FHIR to store and exchange the results collected by MIA during the patient interaction. The system was evaluated according to an existing evaluation framework that covers a broad range of aspects related to the technical quality of a conversational agent including usability, but also accessibility and security. Results: Thirty-six patients recruited from two Swiss hospitals (Lindenhof group and Inselspital, Bern) and two patient organizations conducted the usability test. MIA was favorably received by the participants, who particularly noted the clarity of communication. However, there is room for improvement in the perceived quality of the conversation, the information provided, and the protection of privacy. The Q&A module achieved a precision of 0.51, a recall of 0.87 and an F-Score of 0.64 based on 114 questions asked by the participants. Security and accessibility also require improvements. Conclusion: The applied person-based process described in this paper can provide best practices for future development of medical interview assistants. The application of a standardized evaluation framework helped in saving time and ensures comparability of results.

User perceptions and experiences of an AI-driven conversational agent for mental health support.

Background: The increasing prevalence of artificial intelligence (AI)-driven mental health conversational agents necessitates a comprehensive understanding of user engagement and user perceptions of this technology. This study aims to fill the existing knowledge gap by focusing on Wysa, a commercially available mobile conversational agent designed to provide personalized mental health support. Methods: A total of 159 user reviews posted between January, 2020 and March, 2024, on the Wysa app's Google Play page were collected. Thematic analysis was then used to perform open and inductive coding of the collected data. Results: Seven major themes emerged from the user reviews: "a trusting environment promotes wellbeing", "ubiquitous access offers real-time support", "AI limitations detract from the user experience", "perceived effectiveness of Wysa", "desire for cohesive and predictable interactions", "humanness in AI is welcomed", and "the need for improvements in the user interface". These themes highlight both the benefits and limitations of the AI-driven mental health conversational agents. Conclusions: Users find that Wysa is effective in fostering a strong connection with its users, encouraging them to engage with the app and take positive steps towards emotional resilience and self-improvement. However, its AI needs several improvements to enhance user experience with the application. The findings contribute to the design and implementation of more effective, ethical, and user-aligned AI-driven mental health support systems.

A Greek Conversational Agent for Hematologic Malignancies: Usability and User Experience Assessment.

Enabling patients to actively document their health information significantly improves understanding of how therapies work, disease progression, and overall life quality affects for those living with chronic disorders such as hematologic malignancies. Advancements in artificial intelligence, particularly in areas such as natural language processing and speech recognition, have resulted in the development of interactive tools tailored for healthcare. This paper introduces an innovative conversational agent tailored to the Greek language. The design and deployment of this tool, which incorporates sentiment analysis, aims at gathering detailed family histories and symptom data from individuals diagnosed with hematologic malignancies. Furthermore, we discuss the preliminary findings from a feasibility study assessing the tool's effectiveness. Initial feedback on the user experience suggests a positive reception towards the agent's usability, highlighting its potential to enhance patient engagement in a clinical setting.

Influence of Model Evolution and System Roles on ChatGPT's Performance in Chinese Medical Licensing Exams: Comparative Study.

Background: With the increasing application of large language models like ChatGPT in various industries, its potential in the medical domain, especially in standardized examinations, has become a focal point of research. Objective: The aim of this study is to assess the clinical performance of ChatGPT, focusing on its accuracy and reliability in the Chinese National Medical Licensing Examination (CNMLE). Methods: The CNMLE 2022 question set, consisting of 500 single-answer multiple choices questions, were reclassified into 15 medical subspecialties. Each question was tested 8 to 12 times in Chinese on the OpenAI platform from April 24 to May 15, 2023. Three key factors were considered: the version of GPT-3.5 and 4.0, the prompt's designation of system roles tailored to medical subspecialties, and repetition for coherence. A passing accuracy threshold was established as 60%. The χ2 tests and κ values were employed to evaluate the model's accuracy and consistency. Results: GPT-4.0 achieved a passing accuracy of 72.7%, which was significantly higher than that of GPT-3.5 (54%; P<.001). The variability rate of repeated responses from GPT-4.0 was lower than that of GPT-3.5 (9% vs 19.5%; P<.001). However, both models showed relatively good response coherence, with κ values of 0.778 and 0.610, respectively. System roles numerically increased accuracy for both GPT-4.0 (0.3%-3.7%) and GPT-3.5 (1.3%-4.5%), and reduced variability by 1.7% and 1.8%, respectively (P>.05). In subgroup analysis, ChatGPT achieved comparable accuracy among different question types (P>.05). GPT-4.0 surpassed the accuracy threshold in 14 of 15 subspecialties, while GPT-3.5 did so in 7 of 15 on the first response. Conclusions: GPT-4.0 passed the CNMLE and outperformed GPT-3.5 in key areas such as accuracy, consistency, and medical subspecialty expertise. Adding a system role insignificantly enhanced the model's reliability and answer coherence. GPT-4.0 showed promising potential in medical education and clinical practice, meriting further study.

Engagement With Conversational Agent-Enabled Interventions in Cardiometabolic Disease Management: Protocol for a Systematic Review.

BACKGROUND: Cardiometabolic diseases (CMDs) are a group of interrelated conditions, including heart failure and diabetes, that increase the risk of cardiovascular and metabolic complications. The rising number of Australians with CMDs has necessitated new strategies for those managing these conditions, such as digital health interventions. The effectiveness of digital health interventions in supporting people with CMDs is dependent on the extent to which users engage with the tools. Augmenting digital health interventions with conversational agents, technologies that interact with people using natural language, may enhance engagement because of their human-like attributes. To date, no systematic review has compiled evidence on how design features influence the engagement of conversational agent-enabled interventions supporting people with CMDs. This review seeks to address this gap, thereby guiding developers in creating more engaging and effective tools for CMD management. OBJECTIVE: The aim of this systematic review is to synthesize evidence pertaining to conversational agent-enabled intervention design features and their impacts on the engagement of people managing CMD. METHODS: The review is conducted in accordance with the Cochrane Handbook for Systematic Reviews of Interventions and reported in accordance with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Searches will be conducted in the Ovid (Medline), Web of Science, and Scopus databases, which will be run again prior to manuscript submission. Inclusion criteria will consist of primary research studies reporting on conversational agent-enabled interventions, including measures of engagement, in adults with CMD. Data extraction will seek to capture the perspectives of people with CMD on the use of conversational agent-enabled interventions. Joanna Briggs Institute critical appraisal tools will be used to evaluate the overall quality of evidence collected. RESULTS: This review was initiated in May 2023 and was registered with the International Prospective Register of Systematic Reviews (PROSPERO) in June 2023, prior to title and abstract screening. Full-text screening of articles was completed in July 2023 and data extraction began August 2023. Final searches were conducted in April 2024 prior to finalizing the review and the manuscript was submitted for peer review in July 2024. CONCLUSIONS: This review will synthesize diverse observations pertaining to conversational agent-enabled intervention design features and their impacts on engagement among people with CMDs. These observations can be used to guide the development of more engaging conversational agent-enabled interventions, thereby increasing the likelihood of regular intervention use and improved CMD health outcomes. Additionally, this review will identify gaps in the literature in terms of how engagement is reported, thereby highlighting areas for future exploration and supporting researchers in advancing the understanding of conversational agent-enabled interventions. TRIAL REGISTRATION: PROSPERO CRD42023431579; https://tinyurl.com/55cxkm26. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/52973.

Assessing the Feasibility and Acceptability of Smart Speakers in Behavioral Intervention Research With Older Adults: Mixed Methods Study.

BACKGROUND: Smart speakers, such as Amazon's Echo and Google's Nest Home, combine natural language processing with a conversational interface to carry out everyday tasks, like playing music and finding information. Easy to use, they are embraced by older adults, including those with limited physical function, vision, or computer literacy. While smart speakers are increasingly used for research purposes (eg, implementing interventions and automatically recording selected research data), information on the advantages and disadvantages of using these devices for studies related to health promotion programs is limited. OBJECTIVE: This study evaluates the feasibility and acceptability of using smart speakers to deliver a physical activity (PA) program designed to help older adults enhance their physical well-being. METHODS: Community-dwelling older adults (n=18) were asked to use a custom smart speaker app to participate in an evidence-based, low-impact PA program for 10 weeks. Collected data, including measures of technology acceptance, interviews, field notes, and device logs, were analyzed using a concurrent mixed analysis approach. Technology acceptance measures were evaluated using time series ANOVAs to examine acceptability, appropriateness, feasibility, and intention to adopt smart speaker technology. Device logs provided evidence of interaction with and adoption of the device and the intervention. Interviews and field notes were thematically coded to triangulate the quantitative measures and further expand on factors relating to intervention fidelity. RESULTS: Smart speakers were found to be acceptable for administering a PA program, as participants reported that the devices were highly usable (mean 5.02, SE 0.38) and had strong intentions to continue their use (mean 5.90, SE 0.39). Factors such as the voice-user interface and engagement with the device on everyday tasks were identified as meaningful to acceptability. The feasibility of the devices for research activity, however, was mixed. Despite the participants rating the smart speakers as easy to use (mean 5.55, SE 1.16), functional and technical factors, such as Wi-Fi connectivity and appropriate command phrasing, required the provision of additional support resources to participants and potentially impaired intervention fidelity. CONCLUSIONS: Smart speakers present an acceptable and appropriate behavioral intervention technology for PA programs directed at older adults but entail additional requirements for resource planning, technical support, and troubleshooting to ensure their feasibility for the research context and for fidelity of the intervention.

A Language Model-Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study.

BACKGROUND: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback. OBJECTIVE: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students' performance in history taking with a simulated patient. METHODS: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients' responses and provide immediate feedback on the comprehensiveness of the students' history taking. Students' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback. RESULTS: Most of the study's participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4's role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed "almost perfect" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model's assessments were overly specific or diverged from human judgement. CONCLUSIONS: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.

Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review.

BACKGROUND: Chatbots, or conversational agents, have emerged as significant tools in health care, driven by advancements in artificial intelligence and digital technology. These programs are designed to simulate human conversations, addressing various health care needs. However, no comprehensive synthesis of health care chatbots' roles, users, benefits, and limitations is available to inform future research and application in the field. OBJECTIVE: This review aims to describe health care chatbots' characteristics, focusing on their diverse roles in the health care pathway, user groups, benefits, and limitations. METHODS: A rapid review of published literature from 2017 to 2023 was performed with a search strategy developed in collaboration with a health sciences librarian and implemented in the MEDLINE and Embase databases. Primary research studies reporting on chatbot roles or benefits in health care were included. Two reviewers dual-screened the search results. Extracted data on chatbot roles, users, benefits, and limitations were subjected to content analysis. RESULTS: The review categorized chatbot roles into 2 themes: delivery of remote health services, including patient support, care management, education, skills building, and health behavior promotion, and provision of administrative assistance to health care providers. User groups spanned across patients with chronic conditions as well as patients with cancer; individuals focused on lifestyle improvements; and various demographic groups such as women, families, and older adults. Professionals and students in health care also emerged as significant users, alongside groups seeking mental health support, behavioral change, and educational enhancement. The benefits of health care chatbots were also classified into 2 themes: improvement of health care quality and efficiency and cost-effectiveness in health care delivery. The identified limitations encompassed ethical challenges, medicolegal and safety concerns, technical difficulties, user experience issues, and societal and economic impacts. CONCLUSIONS: Health care chatbots offer a wide spectrum of applications, potentially impacting various aspects of health care. While they are promising tools for improving health care efficiency and quality, their integration into the health care system must be approached with consideration of their limitations to ensure optimal, safe, and equitable use.

Assessing GPT-4's Performance in Delivering Medical Advice: Comparative Analysis With Human Experts.

Background: Accurate medical advice is paramount in ensuring optimal patient care, and misinformation can lead to misguided decisions with potentially detrimental health outcomes. The emergence of large language models (LLMs) such as OpenAI's GPT-4 has spurred interest in their potential health care applications, particularly in automated medical consultation. Yet, rigorous investigations comparing their performance to human experts remain sparse. Objective: This study aims to compare the medical accuracy of GPT-4 with human experts in providing medical advice using real-world user-generated queries, with a specific focus on cardiology. It also sought to analyze the performance of GPT-4 and human experts in specific question categories, including drug or medication information and preliminary diagnoses. Methods: We collected 251 pairs of cardiology-specific questions from general users and answers from human experts via an internet portal. GPT-4 was tasked with generating responses to the same questions. Three independent cardiologists (SL, JHK, and JJC) evaluated the answers provided by both human experts and GPT-4. Using a computer interface, each evaluator compared the pairs and determined which answer was superior, and they quantitatively measured the clarity and complexity of the questions as well as the accuracy and appropriateness of the responses, applying a 3-tiered grading scale (low, medium, and high). Furthermore, a linguistic analysis was conducted to compare the length and vocabulary diversity of the responses using word count and type-token ratio. Results: GPT-4 and human experts displayed comparable efficacy in medical accuracy ("GPT-4 is better" at 132/251, 52.6% vs "Human expert is better" at 119/251, 47.4%). In accuracy level categorization, humans had more high-accuracy responses than GPT-4 (50/237, 21.1% vs 30/238, 12.6%) but also a greater proportion of low-accuracy responses (11/237, 4.6% vs 1/238, 0.4%; P=.001). GPT-4 responses were generally longer and used a less diverse vocabulary than those of human experts, potentially enhancing their comprehensibility for general users (sentence count: mean 10.9, SD 4.2 vs mean 5.9, SD 3.7; P<.001; type-token ratio: mean 0.69, SD 0.07 vs mean 0.79, SD 0.09; P<.001). Nevertheless, human experts outperformed GPT-4 in specific question categories, notably those related to drug or medication information and preliminary diagnoses. These findings highlight the limitations of GPT-4 in providing advice based on clinical experience. Conclusions: GPT-4 has shown promising potential in automated medical consultation, with comparable medical accuracy to human experts. However, challenges remain particularly in the realm of nuanced clinical judgment. Future improvements in LLMs may require the integration of specific clinical reasoning pathways and regulatory oversight for safe use. Further research is needed to understand the full potential of LLMs across various medical specialties and conditions.

Can Large Language Models Replace Therapists? Evaluating Performance at Simple Cognitive Behavioral Therapy Tasks.

The advent of large language models (LLMs) such as ChatGPT has potential implications for psychological therapies such as cognitive behavioral therapy (CBT). We systematically investigated whether LLMs could recognize an unhelpful thought, examine its validity, and reframe it to a more helpful one. LLMs currently have the potential to offer reasonable suggestions for the identification and reframing of unhelpful thoughts but should not be relied on to lead CBT delivery.

Effects of Chatbot Components to Facilitate Mental Health Services Use in Individuals With Eating Disorders Following Online Screening: An Optimization Randomized Controlled Trial.

OBJECTIVE: Few individuals with eating disorders (EDs) receive treatment. Innovations are needed to identify individuals with EDs and address care barriers. We developed a chatbot for promoting services uptake that could be paired with online screening. However, it is not yet known which components drive effects. This study estimated individual and combined contributions of four chatbot components on mental health services use (primary), chatbot helpfulness, and attitudes toward changing eating/shape/weight concerns ("change attitudes," with higher scores indicating greater importance/readiness). METHODS: Two hundred five individuals screening with an ED but not in treatment were randomized in an optimization randomized controlled trial to receive up to four chatbot components: psychoeducation, motivational interviewing, personalized service recommendations, and repeated administration (follow-up check-ins/reminders). Assessments were at baseline and 2, 6, and 14 weeks. RESULTS: Participants who received repeated administration were more likely to report mental health services use, with no significant effects of other components on services use. Repeated administration slowed the decline in change attitudes participants experienced over time. Participants who received motivational interviewing found the chatbot more helpful, but this component was also associated with larger declines in change attitudes. Participants who received personalized recommendations found the chatbot more helpful, and receiving this component on its own was associated with the most favorable change attitude time trend. Psychoeducation showed no effects. DISCUSSION: Results indicated important effects of components on outcomes; findings will be used to finalize decision making about the optimized intervention package. The chatbot shows high potential for addressing the treatment gap for EDs.

Chatbot for Social Need Screening and Resource Sharing With Vulnerable Families: Iterative Design and Evaluation Study.

BACKGROUND: Health outcomes are significantly influenced by unmet social needs. Although screening for social needs has become common in health care settings, there is often poor linkage to resources after needs are identified. The structural barriers (eg, staffing, time, and space) to helping address social needs could be overcome by a technology-based solution. OBJECTIVE: This study aims to present the design and evaluation of a chatbot, DAPHNE (Dialog-Based Assistant Platform for Healthcare and Needs Ecosystem), which screens for social needs and links patients and families to resources. METHODS: This research used a three-stage study approach: (1) an end-user survey to understand unmet needs and perception toward chatbots, (2) iterative design with interdisciplinary stakeholder groups, and (3) a feasibility and usability assessment. In study 1, a web-based survey was conducted with low-income US resident households (n=201). Following that, in study 2, web-based sessions were held with an interdisciplinary group of stakeholders (n=10) using thematic and content analysis to inform the chatbot's design and development. Finally, in study 3, the assessment on feasibility and usability was completed via a mix of a web-based survey and focus group interviews following scenario-based usability testing with community health workers (family advocates; n=4) and social workers (n=9). We reported descriptive statistics and chi-square test results for the household survey. Content analysis and thematic analysis were used to analyze qualitative data. Usability score was descriptively reported. RESULTS: Among the survey participants, employed and younger individuals reported a higher likelihood of using a chatbot to address social needs, in contrast to the oldest age group. Regarding designing the chatbot, the stakeholders emphasized the importance of provider-technology collaboration, inclusive conversational design, and user education. The participants found that the chatbot's capabilities met expectations and that the chatbot was easy to use (System Usability Scale score=72/100). However, there were common concerns about the accuracy of suggested resources, electronic health record integration, and trust with a chatbot. CONCLUSIONS: Chatbots can provide personalized feedback for families to identify and meet social needs. Our study highlights the importance of user-centered iterative design and development of chatbots for social needs. Future research should examine the efficacy, cost-effectiveness, and scalability of chatbot interventions to address social needs.

Implementation of a Web-Based Chatbot to Guide Hospital Employees in Returning to Work During the COVID-19 Pandemic: Development and Before-and-After Evaluation.

BACKGROUND: Throughout the COVID-19 pandemic, multiple policies and guidelines were issued and updated for health care personnel (HCP) for COVID-19 testing and returning to work after reporting symptoms, exposures, or infection. The high frequency of changes and complexity of the policies made it difficult for HCP to understand when they needed testing and were eligible to return to work (RTW), which increased calls to Occupational Health Services (OHS), creating a need for other tools to guide HCP. Chatbots have been used as novel tools to facilitate immediate responses to patients' and employees' queries about COVID-19, assess symptoms, and guide individuals to appropriate care resources. OBJECTIVE: This study aims to describe the development of an RTW chatbot and report its impact on demand for OHS support services during the first Omicron variant surge. METHODS: This study was conducted at Mass General Brigham, an integrated health care system with over 80,000 employees. The RTW chatbot was developed using an agile design methodology. We mapped the RTW policy into a unified flow diagram that included all required questions and recommendations, then built and tested the chatbot using the Microsoft Azure Healthbot Framework. Using chatbot data and OHS call data from December 10, 2021, to February 17, 2022, we compared OHS resource use before and after the deployment of the RTW chatbot, including the number of calls to the OHS hotline, wait times, call length, and time OHS hotline staff spent on the phone. We also assessed Centers for Disease Control and Prevention data for COVID-19 case trends during the study period. RESULTS: In the 5 weeks post deployment, 5575 users used the RTW chatbot with a mean interaction time of 1 minute and 17 seconds. The highest engagement was on January 25, 2022, with 368 users, which was 2 weeks after the peak of the first Omicron surge in Massachusetts. Among users who completed all the chatbot questions, 461 (71.6%) met the RTW criteria. During the 10 weeks, the median (IQR) number of daily calls that OHS received before and after deployment of the chatbot were 633 (251-934) and 115 (62-167), respectively (U=163; P<.001). The median time from dialing the OHS phone number to hanging up decreased from 28 minutes and 22 seconds (IQR 25:14-31:05) to 6 minutes and 25 seconds (IQR 5:32-7:08) after chatbot deployment (U=169; P<.001). Over the 10 weeks, the median time OHS hotline staff spent on the phone declined from 3 hours and 11 minutes (IQR 2:32-4:15) per day to 47 (IQR 42-54) minutes (U=193; P<.001), saving approximately 16.8 hours per OHS staff member per week. CONCLUSIONS: Using the agile methodology, a chatbot can be rapidly designed and deployed for employees to efficiently receive guidance regarding RTW that complies with the complex and shifting RTW policies, which may reduce use of OHS resources.

Bringing together multimodal and multilevel approaches to study the emergence of social bonds between children and improve social AI.

This protocol paper outlines an innovative multimodal and multilevel approach to studying the emergence and evolution of how children build social bonds with their peers, and its potential application to improving social artificial intelligence (AI). We detail a unique hyperscanning experimental framework utilizing functional near-infrared spectroscopy (fNIRS) to observe inter-brain synchrony in child dyads during collaborative tasks and social interactions. Our proposed longitudinal study spans middle childhood, aiming to capture the dynamic development of social connections and cognitive engagement in naturalistic settings. To do so we bring together four kinds of data: the multimodal conversational behaviors that dyads of children engage in, evidence of their state of interpersonal rapport, collaborative performance on educational tasks, and inter-brain synchrony. Preliminary pilot data provide foundational support for our approach, indicating promising directions for identifying neural patterns associated with productive social interactions. The planned research will explore the neural correlates of social bond formation, informing the creation of a virtual peer learning partner in the field of Social Neuroergonomics. This protocol promises significant contributions to understanding the neural basis of social connectivity in children, while also offering a blueprint for designing empathetic and effective social AI tools, particularly for educational contexts.

The Value of a Virtual Assistant to Improve Engagement in Computerized Cognitive Training at Home: Exploratory Study.

BACKGROUND: Impaired cognitive function is observed in many pathologies, including neurodegenerative diseases such as Alzheimer disease. At present, the pharmaceutical treatments available to counter cognitive decline have only modest effects, with significant side effects. A nonpharmacological treatment that has received considerable attention is computerized cognitive training (CCT), which aims to maintain or improve cognitive functioning through repeated practice in standardized exercises. CCT allows for more regular and thorough training of cognitive functions directly at home, which represents a significant opportunity to prevent and fight cognitive decline. However, the presence of assistance during training seems to be an important parameter to improve patients' motivation and adherence to treatment. To compensate for the absence of a therapist during at-home CCT, a relevant option could be to include a virtual assistant to accompany patients throughout their training. OBJECTIVE: The objective of this exploratory study was to evaluate the interest of including a virtual assistant to accompany patients during CCT. We investigated the relationship between various individual factors (eg, age, psycho-affective functioning, personality, personal motivations, and cognitive skills) and the appreciation and usefulness of a virtual assistant during CCT. This study is part of the THERADIA (Thérapies Digitales Augmentées par l'Intelligence Artificielle) project, which aims to develop an empathetic virtual assistant. METHODS: A total of 104 participants were recruited, including 52 (50%) young adults (mean age 21.2, range 18 to 27, SD 2.9 years) and 52 (50%) older adults (mean age 67.9, range 60 to 79, SD 5.1 years). All participants were invited to the laboratory to answer several questionnaires and perform 1 CCT session, which consisted of 4 cognitive exercises supervised by a virtual assistant animated by a human pilot via the Wizard of Oz method. The participants evaluated the virtual assistant and CCT at the end of the session. RESULTS: Analyses were performed using the Bayesian framework. The results suggest that the virtual assistant was appreciated and perceived as useful during CCT in both age groups. However, older adults rated the assistant and CCT more positively overall than young adults. Certain characteristics of users, especially their current affective state (ie, arousal, intrinsic relevance, goal conduciveness, and anxiety state), appeared to be related to their evaluation of the session. CONCLUSIONS: This study provides, for the first time, insight into how young and older adults perceive a virtual assistant during CCT. The results suggest that such an assistant could have a beneficial influence on users' motivation, provided that it can handle different situations, particularly their emotional state. The next step of our project will be to evaluate our device with patients experiencing mild cognitive impairment and to test its effectiveness in long-term cognitive training.

A Chatbot-Delivered Stress Management Coaching for Students (MISHA App): Pilot Randomized Controlled Trial.

BACKGROUND: Globally, students face increasing mental health challenges, including elevated stress levels and declining well-being, leading to academic performance issues and mental health disorders. However, due to stigma and symptom underestimation, students rarely seek effective stress management solutions. Conversational agents in the health sector have shown promise in reducing stress, depression, and anxiety. Nevertheless, research on their effectiveness for students with stress remains limited. OBJECTIVE: This study aims to develop a conversational agent-delivered stress management coaching intervention for students called MISHA and to evaluate its effectiveness, engagement, and acceptance. METHODS: In an unblinded randomized controlled trial, Swiss students experiencing stress were recruited on the web. Using a 1:1 randomization ratio, participants (N=140) were allocated to either the intervention or waitlist control group. Treatment effectiveness on changes in the primary outcome, that is, perceived stress, and secondary outcomes, including depression, anxiety, psychosomatic symptoms, and active coping, were self-assessed and evaluated using ANOVA for repeated measure and general estimating equations. RESULTS: The per-protocol analysis revealed evidence for improvement of stress, depression, and somatic symptoms with medium effect sizes (Cohen d=-0.36 to Cohen d=-0.60), while anxiety and active coping did not change (Cohen d=-0.29 and Cohen d=0.13). In the intention-to-treat analysis, similar results were found, indicating reduced stress (ß estimate=-0.13, 95% CI -0.20 to -0.05; P<.001), depressive symptoms (ß estimate=-0.23, 95% CI -0.38 to -0.08; P=.003), and psychosomatic symptoms (ß estimate=-0.16, 95% CI -0.27 to -0.06; P=.003), while anxiety and active coping did not change. Overall, 60% (42/70) of the participants in the intervention group completed the coaching by completing the postintervention survey. They particularly appreciated the quality, quantity, credibility, and visual representation of information. While individual customization was rated the lowest, the target group fitting was perceived as high. CONCLUSIONS: Findings indicate that MISHA is feasible, acceptable, and effective in reducing perceived stress among students in Switzerland. Future research is needed with different populations, for example, in students with high stress levels or compared to active controls. TRIAL REGISTRATION: German Clinical Trials Register DRKS 00030004; https://drks.de/search/en/trial/DRKS00030004.

Conversational Chatbot for Cigarette Smoking Cessation: Results From the 11-Step User-Centered Design Development Process and Randomized Controlled Trial.

BACKGROUND: Conversational chatbots are an emerging digital intervention for smoking cessation. No studies have reported on the entire development process of a cessation chatbot. OBJECTIVE: We aim to report results of the user-centered design development process and randomized controlled trial for a novel and comprehensive quit smoking conversational chatbot called QuitBot. METHODS: The 4 years of formative research for developing QuitBot followed an 11-step process: (1) specifying a conceptual model; (2) conducting content analysis of existing interventions (63 hours of intervention transcripts); (3) assessing user needs; (4) developing the chat's persona ("personality"); (5) prototyping content and persona; (6) developing full functionality; (7) programming the QuitBot; (8) conducting a diary study; (9) conducting a pilot randomized controlled trial (RCT); (10) reviewing results of the RCT; and (11) adding a free-form question and answer (QnA) function, based on user feedback from pilot RCT results. The process of adding a QnA function itself involved a three-step process: (1) generating QnA pairs, (2) fine-tuning large language models (LLMs) on QnA pairs, and (3) evaluating the LLM outputs. RESULTS: We developed a quit smoking program spanning 42 days of 2- to 3-minute conversations covering topics ranging from motivations to quit, setting a quit date, choosing Food and Drug Administration-approved cessation medications, coping with triggers, and recovering from lapses and relapses. In a pilot RCT with 96% three-month outcome data retention, QuitBot demonstrated high user engagement and promising cessation rates compared to the National Cancer Institute's SmokefreeTXT text messaging program, particularly among those who viewed all 42 days of program content: 30-day, complete-case, point prevalence abstinence rates at 3-month follow-up were 63% (39/62) for QuitBot versus 38.5% (45/117) for SmokefreeTXT (odds ratio 2.58, 95% CI 1.34-4.99; P=.005). However, Facebook Messenger intermittently blocked participants' access to QuitBot, so we transitioned from Facebook Messenger to a stand-alone smartphone app as the communication channel. Participants' frustration with QuitBot's inability to answer their open-ended questions led to us develop a core conversational feature, enabling users to ask open-ended questions about quitting cigarette smoking and for the QuitBot to respond with accurate and professional answers. To support this functionality, we developed a library of 11,000 QnA pairs on topics associated with quitting cigarette smoking. Model testing results showed that Microsoft's Azure-based QnA maker effectively handled questions that matched our library of 11,000 QnA pairs. A fine-tuned, contextualized GPT-3.5 (OpenAI) responds to questions that are not within our library of QnA pairs. CONCLUSIONS: The development process yielded the first LLM-based quit smoking program delivered as a conversational chatbot. Iterative testing led to significant enhancements, including improvements to the delivery channel. A pivotal addition was the inclusion of a core LLM-supported conversational feature allowing users to ask open-ended questions. TRIAL REGISTRATION: ClinicalTrials.gov NCT03585231; https://clinicaltrials.gov/study/NCT03585231.

Language adaptations of mental health interventions: User interaction comparisons with an AI-enabled conversational agent (Wysa) in English and Spanish.

Background: In recent times, digital mental health interventions (DMHIs) have been proven to be efficacious; however, most are available only for English speakers, leaving limited options for non-English languages like Spanish. Research shows that mental health services in one's dominant language show better outcomes. Conversational agents (CAs) offer promise in supporting mental health in non-English populations. This study compared a culturally adapted version of an artificial intelligence (AI)-led mental health app, called Wysa, in Spanish and English. Objectives: To compare user engagement patterns on Wysa-Spanish and Wysa-English and to understand expressions of distress and preferred language in both versions of Wysa. Methods: We adopted a cross-sectional retrospective exploratory design with mixed methods, analyzing users from 10 Spanish-speaking countries between 1 February and 1 August 2022. A quantitative sample A (n = 2767) was used for descriptive statistics, including user engagement metrics with a Wilcoxon test. A subset qualitative sample B (n = 338) was examined for word count differences based on valence, and a content analysis was conducted to examine idioms of distress. Results: Compared to Wysa-English, Wysa-Spanish had more sessions (P < .001, d = 0.18) and a greater volume of disclosure of distress. In Wysa-Spanish, the average length of a conversation was significantly longer than in Wysa-English (P < .001, d = 0.44). Users preferred interventions with free text responses ("Thought recording") in Spanish (P < .01, d = 0.41), and Spanish messages were significantly longer (P < .01, d = 0.24). Wysa-Spanish saw more frequent expressions of negative emotions and feelings of self-harm and suicide. Conclusion: Given the high engagement within the Spanish version of Wysa, the findings demonstrate the need for culturally adapted DMHIs among non-English populations, emphasizing the importance of considering linguistic and cultural differences in the development of DMHIs to improve accessibility for diverse populations.

An artificial intelligence-based dental semantic search engine as a reliable tool for dental students and educators.

PURPOSE/OBJECTIVES: This study proposes the utilization of a Natural Language Processing tool to create a semantic search engine for dental education while addressing the increasing concerns of accuracy, bias, and hallucination in outputs generated by AI tools. The paper focuses on developing and evaluating DentQA, a specialized question-answering tool that makes it easy for students to seek information to access information located in handouts or study material distributed by an institution. METHODS: DentQA is structured upon the GPT3.5 language model, utilizing prompt engineering to extract information from external dental documents that experts have verified. Evaluation involves non-human metrics (BLEU scores) and human metrics for the tool's performance, relevance, accuracy, and functionality. RESULTS: Non-human metrics confirm DentQA's linguistic proficiency, achieving a Unigram BLEU score of 0.85. Human metrics reveal DentQA's superiority over GPT3.5 in terms of accuracy (p = 0.00004) and absence of hallucination (p = 0.026). Additional metrics confirmed consistent performance across different question types (X2 (4, N = 200) = 13.0378, p = 0.012). User satisfaction and performance metrics support DentQA's usability and effectiveness, with a response time of 3.5 s and over 70% satisfaction across all evaluated parameters. CONCLUSIONS: The study advocates using a semantic search engine in dental education, mitigating concerns of misinformation and hallucination. By outlining the workflow and the utilization of open-source tools and methods, the study encourages the utilization of similar tools for dental education while underscoring the importance of customizing AI models for dentistry. Further optimizations, testing, and utilization of recent advances can contribute to dental education significantly.

Multi-Method Formative Evaluation of a Digital Online Grocery Shopping Assistant Among Special Supplemental Nutrition Program for Women, Infants, and Children Participants.

OBJECTIVE: Assess the acceptability of a digital grocery shopping assistant among rural women with low income. DESIGN: Simulated shopping experience, semistructured interviews, and a choice experiment. SETTING: Rural central North Carolina Special Supplemental Nutrition Program for Women, Infants, and Children clinic. PARTICIPANTS: Thirty adults (aged ≥18 years) recruited from a Special Supplemental Nutrition Program for Women, Infants, and Children clinic. PHENOMENON OF INTEREST: A simulated grocery shopping experience with the Retail Online Shopping Assistant (ROSA) and mixed-methods feedback on the experience. ANALYSIS: Deductive and inductive qualitative content analysis to independently code and identify themes and patterns among interview responses and quantitative analysis of simulated shopping experience and choice experiment. RESULTS: Most participants liked ROSA (28/30, 93%) and found it helpful and likely to change their purchase across various food categories and at checkout. Retail Online Shopping Assistant's reminders and suggestions could reduce less healthy shopping habits and diversify food options. Participants desired dynamic suggestions and help with various health conditions. Participants preferred a racially inclusive, approachable, cartoon-like, and clinically dressed character. CONCLUSIONS AND IMPLICATIONS: This formative study suggests ROSA could be a beneficial tool for facilitating healthy online grocery shopping among rural shoppers. Future research should investigate the impact of ROSA on dietary behaviors further.