Attrition in Conversational Agent-Delivered Mental Health Interventions: Systematic Review and Meta-Analysis.

BACKGROUND: Conversational agents (CAs) or chatbots are computer programs that mimic human conversation. They have the potential to improve access to mental health interventions through automated, scalable, and personalized delivery of psychotherapeutic content. However, digital health interventions, including those delivered by CAs, often have high attrition rates. Identifying the factors associated with attrition is critical to improving future clinical trials. OBJECTIVE: This review aims to estimate the overall and differential rates of attrition in CA-delivered mental health interventions (CA interventions), evaluate the impact of study design and intervention-related aspects on attrition, and describe study design features aimed at reducing or mitigating study attrition. METHODS: We searched PubMed, Embase (Ovid), PsycINFO (Ovid), Cochrane Central Register of Controlled Trials, and Web of Science, and conducted a gray literature search on Google Scholar in June 2022. We included randomized controlled trials that compared CA interventions against control groups and excluded studies that lasted for 1 session only and used Wizard of Oz interventions. We also assessed the risk of bias in the included studies using the Cochrane Risk of Bias Tool 2.0. Random-effects proportional meta-analysis was applied to calculate the pooled dropout rates in the intervention groups. Random-effects meta-analysis was used to compare the attrition rate in the intervention groups with that in the control groups. We used a narrative review to summarize the findings. RESULTS: The systematic search retrieved 4566 records from peer-reviewed databases and citation searches, of which 41 (0.90%) randomized controlled trials met the inclusion criteria. The meta-analytic overall attrition rate in the intervention group was 21.84% (95% CI 16.74%-27.36%; I2=94%). Short-term studies that lasted ≤8 weeks showed a lower attrition rate (18.05%, 95% CI 9.91%- 27.76%; I2=94.6%) than long-term studies that lasted >8 weeks (26.59%, 95% CI 20.09%-33.63%; I2=93.89%). Intervention group participants were more likely to attrit than control group participants for short-term (log odds ratio 1.22, 95% CI 0.99-1.50; I2=21.89%) and long-term studies (log odds ratio 1.33, 95% CI 1.08-1.65; I2=49.43%). Intervention-related characteristics associated with higher attrition include stand-alone CA interventions without human support, not having a symptom tracker feature, no visual representation of the CA, and comparing CA interventions with waitlist controls. No participant-level factor reliably predicted attrition. CONCLUSIONS: Our results indicated that approximately one-fifth of the participants will drop out from CA interventions in short-term studies. High heterogeneities made it difficult to generalize the findings. Our results suggested that future CA interventions should adopt a blended design with human support, use symptom tracking, compare CA intervention groups against active controls rather than waitlist controls, and include a visual representation of the CA to reduce the attrition rate. TRIAL REGISTRATION: PROSPERO International Prospective Register of Systematic Reviews CRD42022341415; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022341415.

Spaced Digital Education for Health Professionals: Systematic Review and Meta-Analysis.

BACKGROUND: Spaced digital education applies digital tools to deliver educational content via multiple, repeated learning sessions separated by prespecified time intervals. Spaced digital education appears to promote acquisition and long-term retention of knowledge, skills, and change in clinical behavior. OBJECTIVE: The aim of this review was to assess the effectiveness of spaced digital education in improving pre- and postregistration health care professionals' knowledge, skills, attitudes, satisfaction, and change in clinical behavior. METHODS: This review followed Cochrane's methodology and PRISMA (Preferred Reporting Items of Systematic Reviews and Meta-Analyses) reporting guidelines. We searched MEDLINE, Embase, Web of Science, ERIC, PsycINFO, CINAHL, CENTRAL, and ProQuest Dissertation and Theses databases from January 1990 to February 2023. We included randomized controlled trials (RCTs), cluster RCTs, and quasi-RCTs comparing spaced digital education with nonspaced education, spaced nondigital education, traditional learning, or no intervention for pre- or postregistration health care professionals. Study selection, data extraction, study quality, and certainty of evidence were assessed by 2 independent reviewers. Meta-analyses were conducted using random effect models. RESULTS: We included 23 studies evaluating spaced online education (n=17, 74%) or spaced digital simulation (n=6, 26%) interventions. Most studies assessed 1 or 2 outcomes, including knowledge (n=15, 65%), skills (n=9, 39%), attitudes (n=8, 35%), clinical behavior change (n=8, 35%), and satisfaction (n=7, 30%). Most studies had an unclear or a high risk of bias (n=19, 83%). Spaced online education was superior to massed online education for postintervention knowledge (n=9, 39%; standardized mean difference [SMD] 0.32, 95% CI 0.13-0.51, I2=66%, moderate certainty of evidence). Spaced online education (n=3, 13%) was superior to massed online education (n=2, 9%) and no intervention (n=1, 4%; SMD 0.67, 95% CI 0.43-0.91, I2=5%, moderate certainty of evidence) for postintervention clinical behavior change. Spaced digital simulation was superior to massed simulation for postintervention surgical skills (n=2, 9%; SMD 1.15, 95% CI 0.34-1.96, I2=74%, low certainty of evidence). Spaced digital education positively impacted confidence and satisfaction with the intervention. CONCLUSIONS: Spaced digital education is effective in improving knowledge, particularly in substantially improving surgical skills and promoting clinical behavior change in pre- and postregistration health care professionals. Our findings support the use of spaced digital education interventions in undergraduate and postgraduate health profession education. Trial Registration: PROSPERO CRD42021241969. TRIAL REGISTRATION: PROSPERO CRD42021241969; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=241969.

Information-seeking behaviour of primary care clinicians in Singapore at the point-of-care: A qualitative study.

BACKGROUND: Clinicians' information-seeking behaviours impact patient care quality. Earlier studies indicated that barriers to accessing information deter clinicians from seeking answers to clinical questions. OBJECTIVES: To explore primary care clinicians' information-seeking behaviour at point-of-care, focusing on when and how they seek answers to clinical questions. METHODS: Semi-structured interviews were conducted with 45 clinicians after clinical sessions to investigate their information-seeking habits. Follow-up interviews were conducted after a week for those intending to address unanswered queries. RESULTS: Two thirds of clinicians encountered questions during care, with nearly three quarters resolving them during the session. Colleagues, guidelines and online platforms were common information sources, with smartphones being used to access Google, WhatsApp or UpToDate®. Facilitators included reliable sources and the drive to confirm knowledge, while barriers included ineffective search methods and high workload. Despite challenges, most clinicians expressed satisfaction with their information-seeking process. DISCUSSION: The findings underscore the increasing use of smartphones for accessing clinical information among Singaporean primary care clinicians and suggest the need for tailored training programmes and guidelines to optimise information-seeking practices. CONCLUSION: Insights from this study can inform the development of training programmes and guidelines aimed at improving information-seeking practices among primary care clinicians, potentially enhancing patient care quality.

Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments.

BACKGROUND: Rapid proliferation of mental health interventions delivered through conversational agents (CAs) calls for high-quality evidence to support their implementation and adoption. Selecting appropriate outcomes, instruments for measuring outcomes, and assessment methods are crucial for ensuring that interventions are evaluated effectively and with a high level of quality. OBJECTIVE: We aimed to identify the types of outcomes, outcome measurement instruments, and assessment methods used to assess the clinical, user experience, and technical outcomes in studies that evaluated the effectiveness of CA interventions for mental health. METHODS: We undertook a scoping review of the relevant literature to review the types of outcomes, outcome measurement instruments, and assessment methods in studies that evaluated the effectiveness of CA interventions for mental health. We performed a comprehensive search of electronic databases, including PubMed, Cochrane Central Register of Controlled Trials, Embase (Ovid), PsychINFO, and Web of Science, as well as Google Scholar and Google. We included experimental studies evaluating CA mental health interventions. The screening and data extraction were performed independently by 2 review authors in parallel. Descriptive and thematic analyses of the findings were performed. RESULTS: We included 32 studies that targeted the promotion of mental well-being (17/32, 53%) and the treatment and monitoring of mental health symptoms (21/32, 66%). The studies reported 203 outcome measurement instruments used to measure clinical outcomes (123/203, 60.6%), user experience outcomes (75/203, 36.9%), technical outcomes (2/203, 1.0%), and other outcomes (3/203, 1.5%). Most of the outcome measurement instruments were used in only 1 study (150/203, 73.9%) and were self-reported questionnaires (170/203, 83.7%), and most were delivered electronically via survey platforms (61/203, 30.0%). No validity evidence was cited for more than half of the outcome measurement instruments (107/203, 52.7%), which were largely created or adapted for the study in which they were used (95/107, 88.8%). CONCLUSIONS: The diversity of outcomes and the choice of outcome measurement instruments employed in studies on CAs for mental health point to the need for an established minimum core outcome set and greater use of validated instruments. Future studies should also capitalize on the affordances made available by CAs and smartphones to streamline the evaluation and reduce participants' input burden inherent to self-reporting.

Conversational Agents in Health Care: Expert Interviews to Inform the Definition, Classification, and Conceptual Framework.

BACKGROUND: Conversational agents (CAs), or chatbots, are computer programs that simulate conversations with humans. The use of CAs in health care settings is recent and rapidly increasing, which often translates to poor reporting of the CA development and evaluation processes and unreliable research findings. We developed and published a conceptual framework, designing, developing, evaluating, and implementing a smartphone-delivered, rule-based conversational agent (DISCOVER), consisting of 3 iterative stages of CA design, development, and evaluation and implementation, complemented by 2 cross-cutting themes (user-centered design and data privacy and security). OBJECTIVE: This study aims to perform in-depth, semistructured interviews with multidisciplinary experts in health care CAs to share their views on the definition and classification of health care CAs and evaluate and validate the DISCOVER conceptual framework. METHODS: We conducted one-on-one semistructured interviews via Zoom (Zoom Video Communications) with 12 multidisciplinary CA experts using an interview guide based on our framework. The interviews were audio recorded, transcribed by the research team, and analyzed using thematic analysis. RESULTS: Following participants' input, we defined CAs as digital interfaces that use natural language to engage in a synchronous dialogue using ≥1 communication modality, such as text, voice, images, or video. CAs were classified by 13 categories: response generation method, input and output modalities, CA purpose, deployment platform, CA development modality, appearance, length of interaction, type of CA-user interaction, dialogue initiation, communication style, CA personality, human support, and type of health care intervention. Experts considered that the conceptual framework could be adapted for artificial intelligence-based CAs. However, despite recent advances in artificial intelligence, including large language models, the technology is not able to ensure safety and reliability in health care settings. Finally, aligned with participants' feedback, we present an updated iteration of the conceptual framework for health care conversational agents (CHAT) with key considerations for CA design, development, and evaluation and implementation, complemented by 3 cross-cutting themes: ethics, user involvement, and data privacy and security. CONCLUSIONS: We present an expanded, validated CHAT and aim at guiding researchers from a variety of backgrounds and with different levels of expertise in the design, development, and evaluation and implementation of rule-based CAs in health care settings.

Exploring the Perceptions of mHealth Interventions for the Prevention of Common Mental Disorders in University Students in Singapore: Qualitative Study.

BACKGROUND: Mental health interventions delivered through mobile health (mHealth) technologies can increase the access to mental health services, especially among university students. The development of mHealth intervention is complex and needs to be context sensitive. There is currently limited evidence on the perceptions, needs, and barriers related to these interventions in the Southeast Asian context. OBJECTIVE: This qualitative study aimed to explore the perception of university students and mental health supporters in Singapore about mental health services, campaigns, and mHealth interventions with a focus on conversational agent interventions for the prevention of common mental disorders such as anxiety and depression. METHODS: We conducted 6 web-based focus group discussions with 30 university students and one-to-one web-based interviews with 11 mental health supporters consisting of faculty members tasked with student pastoral care, a mental health first aider, counselors, psychologists, a clinical psychologist, and a psychiatrist. The qualitative analysis followed a reflexive thematic analysis framework. RESULTS: The following 6 main themes were identified: a healthy lifestyle as students, access to mental health services, the role of mental health promotion campaigns, preferred mHealth engagement features, factors that influence the adoption of mHealth interventions, and cultural relevance of mHealth interventions. The interpretation of our findings shows that students were reluctant to use mental health services because of the fear of stigma and a possible lack of confidentiality. CONCLUSIONS: Study participants viewed mHealth interventions for mental health as part of a blended intervention. They also felt that future mental health mHealth interventions should be more personalized and capable of managing adverse events such as suicidal ideation.

Scope, Characteristics, Behavior Change Techniques, and Quality of Conversational Agents for Mental Health and Well-Being: Systematic Assessment of Apps.

BACKGROUND: Mental disorders cause substantial health-related burden worldwide. Mobile health interventions are increasingly being used to promote mental health and well-being, as they could improve access to treatment and reduce associated costs. Behavior change is an important feature of interventions aimed at improving mental health and well-being. There is a need to discern the active components that can promote behavior change in such interventions and ultimately improve users' mental health. OBJECTIVE: This study systematically identified mental health conversational agents (CAs) currently available in app stores and assessed the behavior change techniques (BCTs) used. We further described their main features, technical aspects, and quality in terms of engagement, functionality, esthetics, and information using the Mobile Application Rating Scale. METHODS: The search, selection, and assessment of apps were adapted from a systematic review methodology and included a search, 2 rounds of selection, and an evaluation following predefined criteria. We conducted a systematic app search of Apple's App Store and Google Play using 42matters. Apps with CAs in English that uploaded or updated from January 2020 and provided interventions aimed at improving mental health and well-being and the assessment or management of mental disorders were tested by at least 2 reviewers. The BCT taxonomy v1, a comprehensive list of 93 BCTs, was used to identify the specific behavior change components in CAs. RESULTS: We found 18 app-based mental health CAs. Most CAs had <1000 user ratings on both app stores (12/18, 67%) and targeted several conditions such as stress, anxiety, and depression (13/18, 72%). All CAs addressed >1 mental disorder. Most CAs (14/18, 78%) used cognitive behavioral therapy (CBT). Half (9/18, 50%) of the CAs identified were rule based (ie, only offered predetermined answers) and the other half (9/18, 50%) were artificial intelligence enhanced (ie, included open-ended questions). CAs used 48 different BCTs and included on average 15 (SD 8.77; range 4-30) BCTs. The most common BCTs were 3.3 "Social support (emotional)," 4.1 "Instructions for how to perform a behavior," 11.2 "Reduce negative emotions," and 6.1 "Demonstration of the behavior." One-third (5/14, 36%) of the CAs claiming to be CBT based did not include core CBT concepts. CONCLUSIONS: Mental health CAs mostly targeted various mental health issues such as stress, anxiety, and depression, reflecting a broad intervention focus. The most common BCTs identified serve to promote the self-management of mental disorders with few therapeutic elements. CA developers should consider the quality of information, user confidentiality, access, and emergency management when designing mental health CAs. Future research should assess the role of artificial intelligence in promoting behavior change within CAs and determine the choice of BCTs in evidence-based psychotherapies to enable systematic, consistent, and transparent development and evaluation of effective digital mental health interventions.

Information needs and sources of information among people with depression and anxiety: a scoping review.

BACKGROUND: Previous studies have identified substantial unmet information needs in people with depression and anxiety. Sufficient information about the disorder, treatment, available services, and strategies for self-management is essential as it may influence quality of care and patients' quality of life. This scoping review aimed to provide a broad overview of information needs of people with depression and anxiety as well as the sources that they use to seek this information. METHODS: We included all primary research published in English that investigated information needs or information sources in people with depression or anxiety, with no restrictions imposed on the study design, location, setting, or participant characteristics. Six electronic databases (MEDLINE, Embase, PsycINFO, CINAHL, LISTA, Web of Science) and the grey literature (Google and Google Scholar) were searched for relevant studies published up to November 2021. Two reviewers independently screened articles and extracted data. Narrative synthesis was performed to identify key themes of information needs and information sources. Factors associated with information needs/sources such as demographic variables and symptom severity were also identified. RESULTS: Fifty-six studies (comprising 8320 participants) were included. Information needs were categorised into seven themes, including general facts, treatment, lived experience, healthcare services, coping, financial/legal, and other information. The most frequently reported needs in both people with depression and anxiety were general facts and treatment information. Subclinical samples who self-reported depressive/anxious symptoms appeared less interested in treatment information than patients with clinical diagnoses. Information sources were summarised into five categories: health professionals, written materials, media, interpersonal interactions, and organisational resources. Health professionals and media (including the internet) were the most frequently adopted and preferred sources. Although few studies have examined factors associated with information needs and information sources, there is preliminary evidence that symptom severity and disease subtypes are related to information needs/sources, whereas findings on demographic factors were mixed. CONCLUSIONS: Information needs appear to be high in people with depression and anxiety. Future research should examine differences between subgroups and associated factors such as the treatment course. Personalised information provision strategies are also needed to customise information according to individual needs and patient profiles. TRIAL REGISTRATION: The protocol of this scoping review was registered on Open Science Framework (OSF; link: https://doi.org/10.17605/OSF.IO/DF2M6 ).

Education on Depression in Mental Health Apps: Systematic Assessment of Characteristics and Adherence to Evidence-Based Guidelines.

BACKGROUND: Suboptimal understanding of depression and mental health disorders by the general population is an important contributor to the wide treatment gap in depression. Mental health literacy encompasses knowledge and beliefs about mental disorders and supports their recognition, management, and prevention. Besides knowledge improvement, psychoeducational interventions reduce symptoms of depression, enhance help-seeking behavior, and decrease stigma. Mental health apps often offer educational content, but the trustworthiness of the included information is unclear. OBJECTIVE: The aim of this study is to systematically evaluate adherence to clinical guidelines on depression of the information offered by mental health apps available in major commercial app stores. METHODS: A systematic assessment of the educational content regarding depression in the apps available in the Apple App Store and Google Play was conducted in July 2020. A systematic search for apps published or updated since January 2019 was performed using 42matters. Apps meeting the inclusion criteria were downloaded and assessed using two smartphones: an iPhone 7 (iOS version 14.0.1) and a Sony XPERIA XZs (Android version 8.0.0). The 156-question assessment checklist comprised general characteristics of apps, appraisal of 38 educational topics and their adherence to evidence-based clinical guidelines, as well as technical aspects and quality assurance. The results were tabulated and reported as a narrative review, using descriptive statistics. RESULTS: The app search retrieved 2218 apps, of which 58 were included in the analysis (Android apps: n=29, 50%; iOS apps: n=29, 50%). Of the 58 included apps, 37 (64%) apps offered educational content within a more comprehensive depression or mental health management app. Moreover, 21% (12/58) of apps provided non-evidence-based information. Furthermore, 88% (51/58) of apps included up to 20 of the educational topics, the common ones being listing the symptoms of depression (52/58, 90%) and available treatments (48/58, 83%), particularly psychotherapy. Depression-associated stigma was mentioned by 38% (22/58) of the apps, whereas suicide risk was mentioned by 71% (41/58), generally as an item in a list of symptoms. Of the 58 included apps, 44 (76%) highlighted the importance of help seeking, 29 (50%) emphasized the importance of involving the user's support network. In addition, 52% (30/58) of apps referenced their content, and 17% (10/58) included advertisements. CONCLUSIONS: Information in mental health and depression apps is often brief and incomplete, with 1 in 5 apps providing non-evidence-based information. Given the unmet needs and stigma associated with the disease, it is imperative that apps seize the opportunity to offer quality, evidence-based education or point the users to relevant resources. A multistakeholder consensus on a more stringent development and publication process for mental health apps is essential.

Conversational Agents in Health Care: Scoping Review of Their Behavior Change Techniques and Underpinning Theory.

BACKGROUND: Conversational agents (CAs) are increasingly used in health care to deliver behavior change interventions. Their evaluation often includes categorizing the behavior change techniques (BCTs) using a classification system of which the BCT Taxonomy v1 (BCTTv1) is one of the most common. Previous studies have presented descriptive summaries of behavior change interventions delivered by CAs, but no in-depth study reporting the use of BCTs in these interventions has been published to date. OBJECTIVE: This review aims to describe behavior change interventions delivered by CAs and to identify the BCTs and theories guiding their design. METHODS: We searched PubMed, Embase, Cochrane's Central Register of Controlled Trials, and the first 10 pages of Google and Google Scholar in April 2021. We included primary, experimental studies evaluating a behavior change intervention delivered by a CA. BCTs coding followed the BCTTv1. Two independent reviewers selected the studies and extracted the data. Descriptive analysis and frequent itemset mining to identify BCT clusters were performed. RESULTS: We included 47 studies reporting on mental health (n=19, 40%), chronic disorders (n=14, 30%), and lifestyle change (n=14, 30%) interventions. There were 20/47 embodied CAs (43%) and 27/47 CAs (57%) represented a female character. Most CAs were rule based (34/47, 72%). Experimental interventions included 63 BCTs, (mean 9 BCTs; range 2-21 BCTs), while comparisons included 32 BCTs (mean 2 BCTs; range 2-17 BCTs). Most interventions included BCTs 4.1 "Instruction on how to perform a behavior" (34/47, 72%), 3.3 "Social support" (emotional; 27/47, 57%), and 1.2 "Problem solving" (24/47, 51%). A total of 12/47 studies (26%) were informed by a behavior change theory, mainly the Transtheoretical Model and the Social Cognitive Theory. Studies using the same behavior change theory included different BCTs. CONCLUSIONS: There is a need for the more explicit use of behavior change theories and improved reporting of BCTs in CA interventions to enhance the analysis of intervention effectiveness and improve the reproducibility of research.

Digital Behavior Change Interventions for the Prevention and Management of Type 2 Diabetes: Systematic Market Analysis.

BACKGROUND: Advancements in technology offer new opportunities for the prevention and management of type 2 diabetes. Venture capital companies have been investing in digital diabetes companies that offer digital behavior change interventions (DBCIs). However, little is known about the scientific evidence underpinning such interventions or the degree to which these interventions leverage novel technology-driven automated developments such as conversational agents (CAs) or just-in-time adaptive intervention (JITAI) approaches. OBJECTIVE: Our objectives were to identify the top-funded companies offering DBCIs for type 2 diabetes management and prevention, review the level of scientific evidence underpinning the DBCIs, identify which DBCIs are recognized as evidence-based programs by quality assurance authorities, and examine the degree to which these DBCIs include novel automated approaches such as CAs and JITAI mechanisms. METHODS: A systematic search was conducted using 2 venture capital databases (Crunchbase Pro and Pitchbook) to identify the top-funded companies offering interventions for type 2 diabetes prevention and management. Scientific publications relating to the identified DBCIs were identified via PubMed, Google Scholar, and the DBCIs' websites, and data regarding intervention effectiveness were extracted. The Diabetes Prevention Recognition Program (DPRP) of the Center for Disease Control and Prevention in the United States was used to identify the recognition status. The DBCIs' publications, websites, and mobile apps were reviewed with regard to the intervention characteristics. RESULTS: The 16 top-funded companies offering DBCIs for type 2 diabetes received a total funding of US $2.4 billion as of June 15, 2021. Only 4 out of the 50 identified publications associated with these DBCIs were fully powered randomized controlled trials (RCTs). Further, 1 of those 4 RCTs showed a significant difference in glycated hemoglobin A1c (HbA1c) outcomes between the intervention and control groups. However, all the studies reported HbA1c improvements ranging from 0.2% to 1.9% over the course of 12 months. In addition, 6 interventions were fully recognized by the DPRP to deliver evidence-based programs, and 2 interventions had a pending recognition status. Health professionals were included in the majority of DBCIs (13/16, 81%,), whereas only 10% (1/10) of accessible apps involved a CA as part of the intervention delivery. Self-reports represented most of the data sources (74/119, 62%) that could be used to tailor JITAIs. CONCLUSIONS: Our findings suggest that the level of funding received by companies offering DBCIs for type 2 diabetes prevention and management does not coincide with the level of evidence on the intervention effectiveness. There is considerable variation in the level of evidence underpinning the different DBCIs and an overall need for more rigorous effectiveness trials and transparent reporting by quality assurance authorities. Currently, very few DBCIs use automated approaches such as CAs and JITAIs, limiting the scalability and reach of these solutions.

Digital Education for Health Professionals: An Evidence Map, Conceptual Framework, and Research Agenda.

BACKGROUND: Health professions education has undergone major changes with the advent and adoption of digital technologies worldwide. OBJECTIVE: This study aims to map the existing evidence and identify gaps and research priorities to enable robust and relevant research in digital health professions education. METHODS: We searched for systematic reviews on the digital education of practicing and student health care professionals. We searched MEDLINE, Embase, Cochrane Library, Educational Research Information Center, CINAHL, and gray literature sources from January 2014 to July 2020. A total of 2 authors independently screened the studies, extracted the data, and synthesized the findings. We outlined the key characteristics of the included reviews, the quality of the evidence they synthesized, and recommendations for future research. We mapped the empirical findings and research recommendations against the newly developed conceptual framework. RESULTS: We identified 77 eligible systematic reviews. All of them included experimental studies and evaluated the effectiveness of digital education interventions in different health care disciplines or different digital education modalities. Most reviews included studies on various digital education modalities (22/77, 29%), virtual reality (19/77, 25%), and online education (10/77, 13%). Most reviews focused on health professions education in general (36/77, 47%), surgery (13/77, 17%), and nursing (11/77, 14%). The reviews mainly assessed participants' skills (51/77, 66%) and knowledge (49/77, 64%) and included data from high-income countries (53/77, 69%). Our novel conceptual framework of digital health professions education comprises 6 key domains (context, infrastructure, education, learners, research, and quality improvement) and 16 subdomains. Finally, we identified 61 unique questions for future research in these reviews; these mapped to framework domains of education (29/61, 47% recommendations), context (17/61, 28% recommendations), infrastructure (9/61, 15% recommendations), learners (3/61, 5% recommendations), and research (3/61, 5% recommendations). CONCLUSIONS: We identified a large number of research questions regarding digital education, which collectively reflect a diverse and comprehensive research agenda. Our conceptual framework will help educators and researchers plan, develop, and study digital education. More evidence from low- and middle-income countries is needed.

Priorities for family physician and general practitioner recruitment and retention in Singapore: a PRIORITIZE study.

BACKGROUND: A shortage of primary care physicians has been reported in many countries. Primary care systems are diverse and the challenges leading to a decline in workforce are at times context-specific and require tailored solutions. Inviting frontline clinicians to share their insights can help identify optimal strategies for a particular setting. To determine priorities for family physicians' and general practitioners' recruitment and retention in Singapore, we invited primary care physicians to rank pertinent strategies using PRIORITIZE, a transparent, systematic priority-setting approach. METHODS: The study advisory board, consisting of representatives of Singapore's key primary care stakeholders, determined the criteria for prioritising of general practitioners (GPs) and family physicians (FPs) recruitment and retention strategies in Singapore. A comprehensive list of GPs and FPs recruitment and retention strategies was extracted from a recent systematic review of the relevant literature. A questionnaire listing the strategies and the scoring criteria was administered online to doctors practicing in public and private sector in Singapore. Respondents' scores were combined to create a ranked list of locally most relevant strategies for improving GPs and FPs recruitment and retention. RESULTS: We recruited a diverse sample of 50 GPs and FPs practicing in a variety of primary care settings, many with a range of additional professional responsibilities. Around 60 and 66% of respondents thought that there was a problem with recruitment and retention of GPs and FPs in Singapore, respectively. Strategies focusing on promoting primary care by emphasizing the advantages and enhancing the status of the profession as well as training-related strategies, such as sub-specialisation and high-quality rotations were considered priorities for improving recruitment. For retention of GPs and FPs, improving working conditions by increasing GPs' and FPs' salary and recognition, as well as varying or reducing time commitment, were seen as the most important strategies. The ranking between physicians working in public and private sector was mostly similar, with nine out of the top ten recruitment and retention strategies being the same. CONCLUSION: Primary care physicians' ranking of recruitment and retention strategies for GPs and FPs in Singapore provide important insight into the challenges and the solutions as seen by the members of the profession themselves. This information can guide future policy and decision making in this area.

Long-term Effectiveness of mHealth Physical Activity Interventions: Systematic Review and Meta-analysis of Randomized Controlled Trials.

BACKGROUND: Mobile health (mHealth) interventions can increase physical activity (PA); however, their long-term impact is not well understood. OBJECTIVE: The primary aim of this study is to understand the immediate and long-term effects of mHealth interventions on PA. The secondary aim is to explore potential effect moderators. METHODS: We performed this study according to the Cochrane and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. We searched PubMed, the Cochrane Library, SCOPUS, and PsycINFO in July 2020. Eligible studies included randomized controlled trials of mHealth interventions targeting PA as a primary outcome in adults. Eligible outcome measures were walking, moderate-to-vigorous physical activity (MVPA), total physical activity (TPA), and energy expenditure. Where reported, we extracted data for 3 time points (ie, end of intervention, follow-up ≤6 months, and follow-up >6 months). To explore effect moderators, we performed subgroup analyses by population, intervention design, and control group type. Results were summarized using random effects meta-analysis. Risk of bias was assessed using the Cochrane Collaboration tool. RESULTS: Of the 2828 identified studies, 117 were included. These studies reported on 21,118 participants with a mean age of 52.03 (SD 14.14) years, of whom 58.99% (n=12,459) were female. mHealth interventions significantly increased PA across all the 4 outcome measures at the end of intervention (walking standardized mean difference [SMD] 0.46, 95% CI 0.36-0.55; P<.001; MVPA SMD 0.28, 95% CI 0.21-0.35; P<.001; TPA SMD 0.34, 95% CI 0.20-0.47; P<.001; energy expenditure SMD 0.44, 95% CI 0.13-0.75; P=.01). Only 33 studies reported short-term follow-up measurements, and 8 studies reported long-term follow-up measurements in addition to end-of-intervention results. In the short term, effects were sustained for walking (SMD 0.26, 95% CI 0.09-0.42; P=.002), MVPA (SMD 0.20, 95% CI 0.05-0.35; P=.008), and TPA (SMD 0.53, 95% CI 0.13-0.93; P=.009). In the long term, effects were also sustained for walking (SMD 0.25, 95% CI 0.10-0.39; P=.001) and MVPA (SMD 0.19, 95% CI 0.11-0.27; P<.001). We found the study population to be an effect moderator, with higher effect scores in sick and at-risk populations. PA was increased both in scalable and nonscalable mHealth intervention designs and regardless of the control group type. The risk of bias was rated high in 80.3% (94/117) of the studies. Heterogeneity was significant, resulting in low to very low quality of evidence. CONCLUSIONS: mHealth interventions can foster small to moderate increases in PA. The effects are maintained long term; however, the effect size decreases over time. The results encourage using mHealth interventions in at-risk and sick populations and support the use of scalable mHealth intervention designs to affordably reach large populations. However, given the low evidence quality, further methodologically rigorous studies are warranted to evaluate the long-term effects.

Just-in-Time Adaptive Mechanisms of Popular Mobile Apps for Individuals With Depression: Systematic App Search and Literature Review.

BACKGROUND: The number of smartphone apps that focus on the prevention, diagnosis, and treatment of depression is increasing. A promising approach to increase the effectiveness of the apps while reducing the individual's burden is the use of just-in-time adaptive intervention (JITAI) mechanisms. JITAIs are designed to improve the effectiveness of the intervention and reduce the burden on the person using the intervention by providing the right type of support at the right time. The right type of support and the right time are determined by measuring the state of vulnerability and the state of receptivity, respectively. OBJECTIVE: The aim of this study is to systematically assess the use of JITAI mechanisms in popular apps for individuals with depression. METHODS: We systematically searched for apps addressing depression in the Apple App Store and Google Play Store, as well as in curated lists from the Anxiety and Depression Association of America, the United Kingdom National Health Service, and the American Psychological Association in August 2020. The relevant apps were ranked according to the number of reviews (Apple App Store) or downloads (Google Play Store). For each app, 2 authors separately reviewed all publications concerning the app found within scientific databases (PubMed, Cochrane Register of Controlled Trials, PsycINFO, Google Scholar, IEEE Xplore, Web of Science, ACM Portal, and Science Direct), publications cited on the app's website, information on the app's website, and the app itself. All types of measurements (eg, open questions, closed questions, and device analytics) found in the apps were recorded and reviewed. RESULTS: None of the 28 reviewed apps used JITAI mechanisms to tailor content to situations, states, or individuals. Of the 28 apps, 3 (11%) did not use any measurements, 20 (71%) exclusively used self-reports that were insufficient to leverage the full potential of the JITAIs, and the 5 (18%) apps using self-reports and passive measurements used them as progress or task indicators only. Although 34% (23/68) of the reviewed publications investigated the effectiveness of the apps and 21% (14/68) investigated their efficacy, no publication mentioned or evaluated JITAI mechanisms. CONCLUSIONS: Promising JITAI mechanisms have not yet been translated into mainstream depression apps. Although the wide range of passive measurements available from smartphones were rarely used, self-reported outcomes were used by 71% (20/28) of the apps. However, in both cases, the measured outcomes were not used to tailor content and timing along a state of vulnerability or receptivity. Owing to this lack of tailoring to individual, state, or situation, we argue that the apps cannot be considered JITAIs. The lack of publications investigating whether JITAI mechanisms lead to an increase in the effectiveness or efficacy of the apps highlights the need for further research, especially in real-world apps.

Information needs and information seeking behaviour of people with dementia and their non-professional caregivers: a scoping review.

BACKGROUND: People with dementia often require full-time caregivers especially in the later stages of their condition. People with dementia and caregivers' access to reliable information on dementia is essential as it may have an important impact on patient care and quality of life. This study aims to provide an overview of the information needs and information seeking behaviour of people with dementia and their non-professional caregivers. METHODS: We conducted a scoping review of the literature and searched four electronic databases for eligible studies published up to August 2018. Two reviewers independently screened studies and extracted data. Information needs were classified according to emerging themes in the literature, and information seeking behaviour was categorized using Wilson's model of information behaviour. RESULTS: Twenty studies with a total of 4140 participants, were included in this review. Reported information needs focused on: (i) disease; (ii) patient care provision; (iii) healthcare services; and (iv) caregiver self-care. The most commonly reported information need was on healthcare service-related information. Characteristics found to influence information needs were the severity of dementia as well as patient and caregiver status. People with dementia and non-professional caregivers mainly displayed active searching, information seeking behaviour and preferred using electronic sources to obtain health information. CONCLUSION: Current dementia information sources available in English are extensive in the information they offer, but more emphasis needs to be placed on healthcare service-related information. All studies originated from high income countries and focused on information needs of non-professional caregivers only. The only variables found to be associated to information needs were severity of dementia condition as well as patient/caregiver status. The information needs identified in this review can be used to inform development and design of future dementia resources for people with dementia and their non-professional caregivers.

A Digitally Competent Health Workforce: Scoping Review of Educational Frameworks.

BACKGROUND: Digital health technologies can be key to improving health outcomes, provided health care workers are adequately trained to use these technologies. There have been efforts to identify digital competencies for different health care worker groups; however, an overview of these efforts has yet to be consolidated and analyzed. OBJECTIVE: The review aims to identify and study existing digital health competency frameworks for health care workers and provide recommendations for future digital health training initiatives and framework development. METHODS: A literature search was performed to collate digital health competency frameworks published from 2000. A total of 6 databases including gray literature sources such as OpenGrey, ResearchGate, Google Scholar, Google, and websites of relevant associations were searched in November 2019. Screening and data extraction were performed in parallel by the reviewers. The included evidence is narratively described in terms of characteristics, evolution, and structural composition of frameworks. A thematic analysis was also performed to identify common themes across the included frameworks. RESULTS: In total, 30 frameworks were included in this review, a majority of which aimed at nurses, originated from high-income countries, were published since 2016, and were developed via literature reviews, followed by expert consultations. The thematic analysis uncovered 28 digital health competency domains across the included frameworks. The most prevalent domains pertained to basic information technology literacy, health information management, digital communication, ethical, legal, or regulatory requirements, and data privacy and security. The Health Information Technology Competencies framework was found to be the most comprehensive framework, as it presented 21 out of the 28 identified domains, had the highest number of competencies, and targeted a wide variety of health care workers. CONCLUSIONS: Digital health training initiatives should focus on competencies relevant to a particular health care worker group, role, level of seniority, and setting. The findings from this review can inform and guide digital health training initiatives. The most prevalent competency domains identified represent essential interprofessional competencies to be incorporated into health care workers' training. Digital health frameworks should be regularly updated with novel digital health technologies, be applicable to low- and middle-income countries, and include overlooked health care worker groups such as allied health professionals.

Use of Smartphones to Detect Diabetic Retinopathy: Scoping Review and Meta-Analysis of Diagnostic Test Accuracy Studies.

BACKGROUND: Diabetic retinopathy (DR), a common complication of diabetes mellitus, is the leading cause of impaired vision in adults worldwide. Smartphone ophthalmoscopy involves using a smartphone camera for digital retinal imaging. Utilizing smartphones to detect DR is potentially more affordable, accessible, and easier to use than conventional methods. OBJECTIVE: This study aimed to determine the diagnostic accuracy of various smartphone ophthalmoscopy approaches for detecting DR in diabetic patients. METHODS: We performed an electronic search on the Medical Literature Analysis and Retrieval System Online (MEDLINE), EMBASE, and Cochrane Library for literature published from January 2000 to November 2018. We included studies involving diabetic patients, which compared the diagnostic accuracy of smartphone ophthalmoscopy for detecting DR to an accurate or commonly employed reference standard, such as indirect ophthalmoscopy, slit-lamp biomicroscopy, and tabletop fundus photography. Two reviewers independently screened studies against the inclusion criteria, extracted data, and assessed the quality of included studies using the Quality Assessment of Diagnostic Accuracy Studies-2 tool, with disagreements resolved via consensus. Sensitivity and specificity were pooled using the random effects model. A summary receiver operating characteristic (SROC) curve was constructed. This review is reported in line with the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies guidelines. RESULTS: In all, nine studies involving 1430 participants were included. Most studies were of high quality, except one study with limited applicability because of its reference standard. The pooled sensitivity and specificity for detecting any DR was 87% (95% CI 74%-94%) and 94% (95% CI 81%-98%); mild nonproliferative DR (NPDR) was 39% (95% CI 10%-79%) and 95% (95% CI 91%-98%); moderate NPDR was 71% (95% CI 57%-81%) and 95% (95% CI 88%-98%); severe NPDR was 80% (95% CI 49%-94%) and 97% (95% CI 88%-99%); proliferative DR (PDR) was 92% (95% CI 79%-97%) and 99% (95% CI 96%-99%); diabetic macular edema was 79% (95% CI 63%-89%) and 93% (95% CI 82%-97%); and referral-warranted DR was 91% (95% CI 86%-94%) and 89% (95% CI 56%-98%). The area under SROC curve ranged from 0.879 to 0.979. The diagnostic odds ratio ranged from 11.3 to 1225. CONCLUSIONS: We found heterogeneous evidence showing that smartphone ophthalmoscopy performs well in detecting DR. The diagnostic accuracy for PDR was highest. Future studies should standardize reference criteria and classification criteria and evaluate other available forms of smartphone ophthalmoscopy in primary care settings.

Conversational Agents in Health Care: Scoping Review and Conceptual Analysis.

BACKGROUND: Conversational agents, also known as chatbots, are computer programs designed to simulate human text or verbal conversations. They are increasingly used in a range of fields, including health care. By enabling better accessibility, personalization, and efficiency, conversational agents have the potential to improve patient care. OBJECTIVE: This study aimed to review the current applications, gaps, and challenges in the literature on conversational agents in health care and provide recommendations for their future research, design, and application. METHODS: We performed a scoping review. A broad literature search was performed in MEDLINE (Medical Literature Analysis and Retrieval System Online; Ovid), EMBASE (Excerpta Medica database; Ovid), PubMed, Scopus, and Cochrane Central with the search terms "conversational agents," "conversational AI," "chatbots," and associated synonyms. We also searched the gray literature using sources such as the OCLC (Online Computer Library Center) WorldCat database and ResearchGate in April 2019. Reference lists of relevant articles were checked for further articles. Screening and data extraction were performed in parallel by 2 reviewers. The included evidence was analyzed narratively by employing the principles of thematic analysis. RESULTS: The literature search yielded 47 study reports (45 articles and 2 ongoing clinical trials) that matched the inclusion criteria. The identified conversational agents were largely delivered via smartphone apps (n=23) and used free text only as the main input (n=19) and output (n=30) modality. Case studies describing chatbot development (n=18) were the most prevalent, and only 11 randomized controlled trials were identified. The 3 most commonly reported conversational agent applications in the literature were treatment and monitoring, health care service support, and patient education. CONCLUSIONS: The literature on conversational agents in health care is largely descriptive and aimed at treatment and monitoring and health service support. It mostly reports on text-based, artificial intelligence-driven, and smartphone app-delivered conversational agents. There is an urgent need for a robust evaluation of diverse health care conversational agents' formats, focusing on their acceptability, safety, and effectiveness.

Facilitators and barriers of using digital technology for the management of diabetic foot ulcers: A qualitative systematic review.

The use of digital technology has been shown to be effective in managing chronic conditions. Telemedicine and mobile application are two common applications of digital technology in managing diabetic foot ulcers (DFU). The facilitators and barriers of using it for DFU management are yet to be explored. This is a qualitative systematic review. Five bibliography databases and grey literature sources were searched (2000-2019). Two reviewers independently screened the citations, extracted the data, assessed the quality of the included studies, and performed thematic synthesis. Three studies on patients and five studies on healthcare practitioners (HCPs) were included. Two studies focused on the use of mobile applications and six on telemedicine. In studies on patients, four analytical themes were generated: the relationships with HCPs; the attitude towards the usage of digital technology; the role of wound image taking; and impact of digital technology on DFU care, encompassing 15 facilitators (eg, enabling community support, improving wound care knowledge) and 12 barriers (eg, lack of technological savviness, difficulty reading on smartphones). Three analytical themes were generated from studies on HCPs: the impact of digital technology on HCPs; the role of digital technology in DFU care; and organisation of DFU care delivery, encompassing 17 facilitators (eg, adequate wound care training, digital technology enables holistic care) and 16 barriers (eg, lack of multidisciplinary approach in caring for DFU, lack of direct contact in care provision). Patients and HCPs reported various barriers and facilitators relating to different aspects of using digital technology in DFU management. Our findings can help inform future research as well as the adoption of digital technology in DFU management.