Using Adaptive Bandit Experiments to Increase and Investigate Engagement in Mental Health.

Digital mental health (DMH) interventions, such as text-message-based lessons and activities, offer immense potential for accessible mental health support. While these interventions can be effective, real-world experimental testing can further enhance their design and impact. Adaptive experimentation, utilizing algorithms like Thompson Sampling for (contextual) multi-armed bandit (MAB) problems, can lead to continuous improvement and personalization. However, it remains unclear when these algorithms can simultaneously increase user experience rewards and facilitate appropriate data collection for social-behavioral scientists to analyze with sufficient statistical confidence. Although a growing body of research addresses the practical and statistical aspects of MAB and other adaptive algorithms, further exploration is needed to assess their impact across diverse real-world contexts. This paper presents a software system developed over two years that allows text-messaging intervention components to be adapted using bandit and other algorithms while collecting data for side-by-side comparison with traditional uniform random non-adaptive experiments. We evaluate the system by deploying a text-message-based DMH intervention to 1100 users, recruited through a large mental health non-profit organization, and share the path forward for deploying this system at scale. This system not only enables applications in mental health but could also serve as a model testbed for adaptive experimentation algorithms in other domains.

Exploring User Perspectives on Brief Reflective Questioning Activities for Stress Management: Mixed Methods Study.

BACKGROUND: Current online interventions dedicated to assisting individuals in managing stress and negative emotions often necessitate substantial time commitments. This can be burdensome for users, leading to high dropout rates and reducing the effectiveness of these interventions. This highlights an urgent need for concise digital activities that individuals can swiftly access during instances of negative emotions or stress in their daily lives. OBJECTIVE: The primary aim of this study was to investigate the viability of using a brief digital exercise, specifically a reflective questioning activity (RQA), to help people reflect on their thoughts and emotions about a troubling situation. The RQA is designed to be quick, applicable to the general public, and scalable without requiring a significant support structure. METHODS: We conducted 3 simultaneous studies. In the first study, we recruited 48 participants who completed the RQA and provided qualitative feedback on its design through surveys and semistructured interviews. In the second study, which involved 215 participants from Amazon Mechanical Turk, we used a between-participants design to compare the RQA with a single-question activity. Our hypotheses posited that the RQA would yield greater immediate stress relief and higher perceived utility, while not significantly altering the perception of time commitment. To assess these, we measured survey completion times and gathered multiple self-reported scores. In the third study, we assessed the RQA's real-world impact as a periodic intervention, exploring engagement via platforms such as email and SMS text messaging, complemented by follow-up interviews with participants. RESULTS: In our first study, participants appreciated the RQA for facilitating structured reflection, enabling expression through writing, and promoting problem-solving. However, some of the participants experienced confusion and frustration, particularly when they were unable to find solutions or alternative perspectives on their thoughts. In the second study, the RQA condition resulted in significantly higher ratings (P=.003) for the utility of the activity and a statistically significant decrease (P<.001) in perceived stress rating compared with the single-question activity. Although the RQA required significantly more time to be completed (P<.001), there was no statistically significant difference in participants' subjective perceived time commitment (P=.37). Deploying the RQA over 2 weeks in the third study identified some potential challenges to consider for such activities, such as the monotony of doing the same activity several times, the limited affordances of mobile phones, and the importance of having the prompts align with the occurrence of new troubling situations. CONCLUSIONS: This paper describes the design and evaluation of a brief online self-reflection activity based on cognitive behavioral therapy principles. Our findings can inform practitioners and researchers in the design and exploration of formats for brief interventions to help people with everyday struggles.

A flexible micro-randomized trial design and sample size considerations.

Technological advancements have made it possible to deliver mobile health interventions to individuals. A novel framework that has emerged from such advancements is the just-in-time adaptive intervention, which aims to suggest the right support to the individuals when their needs arise. The micro-randomized trial design has been proposed recently to test the proximal effects of the components of these just-in-time adaptive interventions. However, the extant micro-randomized trial framework only considers components with a fixed number of categories added at the beginning of the study. We propose a more flexible micro-randomized trial design which allows addition of more categories to the components during the study. Note that the number and timing of the categories added during the study need to be fixed initially. The proposed design is motivated by collaboration on the Diabetes and Mental Health Adaptive Notification Tracking and Evaluation study, which learns to deliver effective text messages to encourage physical activity among patients with diabetes and depression. We developed a new test statistic and the corresponding sample size calculator for the flexible micro-randomized trial using an approach similar to the generalized estimating equation for longitudinal data. Simulation studies were conducted to evaluate the sample size calculators and an R shiny application for the calculators was developed.

Investigating the Role of Context in the Delivery of Text Messages for Supporting Psychological Wellbeing.

Without a nuanced understanding of users' perspectives and contexts, text messaging tools for supporting psychological wellbeing risk delivering interventions that are mismatched to users' dynamic needs. We investigated the contextual factors that influence young adults' day-to-day experiences when interacting with such tools. Through interviews and focus group discussions with 36 participants, we identified that people's daily schedules and affective states were dominant factors that shape their messaging preferences. We developed two messaging dialogues centered around these factors, which we deployed to 42 participants to test and extend our initial understanding of users' needs. Across both studies, participants provided diverse opinions of how they could be best supported by messages, particularly around when to engage users in more passive versus active ways. They also proposed ways of adjusting message length and content during periods of low mood. Our findings provide design implications and opportunities for context-aware mental health management systems.

Ratings and experiences in using a mobile application to increase physical activity among university students: implications for future design.

University students have low levels of physical activity and are at risk of mental health disorders. Mobile apps to encourage physical activity can help students, who are frequent smartphone-users, to improve their physical and mental health. Here we report students' qualitative feedback on a physical activity smartphone app with motivational text messaging. We provide recommendations for the design of future apps. 103 students used the app for 6 weeks in the context of a clinical trial (NCT04440553) and answered open-ended questions before the start of the study and at follow-up. A subsample (n = 39) provided additional feedback via text message, and a phone interview (n = 8). Questions focused on the perceived encouragement and support by the app, text messaging content, and recommendations for future applications. We analyzed all transcripts for emerging themes using qualitative coding in Dedoose. The majority of participants were female (69.9%), Asian or Pacific Islander (53.4%), with a mean age of 20.2 years, and 63% had elevated depressive symptoms. 26% felt encouraged or neutral toward the app motivating them to be more physically active. Participants liked messages on physical activity benefits on (mental) health, encouraging them to complete their goal, and feedback on their activity. Participants disliked messages that did not match their motivations for physical activity and their daily context (e.g., time, weekday, stress). Physical activity apps for students should be adapted to their motivations, changing daily context, and mental health issues. Feedback from this sample suggests a key to effectiveness is finding effective ways to personalize digital interventions.

"I Kind of Bounce off It": Translating Mental Health Principles into Real Life Through Story-Based Text Messages.

Adopting new psychological strategies to improve mental wellness can be challenging since people are often unable to anticipate how new habits are applicable to their circumstances. Narrative-based interventions have the potential to alleviate this burden by illustrating psychological principles in an applied context. In this work, we explore how stories can be delivered via the ubiquitous and scalable medium of text messaging. Through formative work consisting of interviews and focus group discussions with 15 participants, we identified desirable elements of stories about mental health, including authenticity and relatability. We then deployed story-based text messages to 42 participants to explore challenges regarding both the stories' content (e.g., specific versus generalized) and format (e.g., story length). We observed that our stories helped participants reflect on and identify flaws in their thinking patterns. Our findings highlight design implications and opportunities for mental wellness interventions that utilize stories in text messaging services.

Developing Messaging Content for a Physical Activity Smartphone App Tailored to Low-Income Patients: User-Centered Design and Crowdsourcing Approach.

BACKGROUND: Text messaging interventions can be an effective and efficient way to improve health behavioral changes. However, most texting interventions are neither tested nor designed with diverse end users, which could reduce their impact, and there is limited evidence regarding the optimal design methodology of health text messages tailored to low-income, low-health literacy populations and non-English speakers. OBJECTIVE: This study aims to combine participant feedback, crowdsourced data, and researcher expertise to develop motivational text messages in English and Spanish that will be used in a smartphone app-based texting intervention that seeks to encourage physical activity in low-income minority patients with diabetes diagnoses and depression symptoms. METHODS: The design process consisted of 5 phases and was iterative in nature, given that the findings from each step informed the subsequent steps. First, we designed messages to increase physical activity based on the behavior change theory and knowledge from the available evidence. Second, using user-centered design methods, we refined these messages after a card sorting task and semistructured interviews (N=10) and evaluated their likeability during a usability testing phase of the app prototype (N=8). Third, the messages were tested by English- and Spanish-speaking participants on the Amazon Mechanical Turk (MTurk) crowdsourcing platform (N=134). Participants on MTurk were asked to categorize the messages into overarching theoretical categories based on the capability, opportunity, motivation, and behavior framework. Finally, each coauthor rated the messages for their overall quality from 1 to 5. All messages were written at a sixth-grade or lower reading level and culturally adapted and translated into neutral Spanish by bilingual research staff. RESULTS: A total of 200 messages were iteratively refined according to the feedback from target users gathered through user-centered design methods, crowdsourced results of a categorization test, and an expert review. User feedback was leveraged to discard unappealing messages and edit the thematic aspects of messages that did not resonate well with the target users. Overall, 54 messages were sorted into the correct theoretical categories at least 50% of the time in the MTurk categorization tasks and were rated 3.5 or higher by the research team members. These were included in the final text message bank, resulting in 18 messages per motivational category. CONCLUSIONS: By using an iterative process of expert opinion, feedback from participants that were reflective of our target study population, crowdsourcing, and feedback from the research team, we were able to acquire valuable inputs for the design of motivational text messages developed in English and Spanish with a low literacy level to increase physical activity. We describe the design considerations and lessons learned for the text messaging development process and provide a novel, integrative framework for future developers of health text messaging interventions.

Nonprofessional Peer Support to Improve Mental Health: Randomized Trial of a Scalable Web-Based Peer Counseling Course.

BACKGROUND: Millions of people worldwide are underserved by the mental health care system. Indeed, most mental health problems go untreated, often because of resource constraints (eg, limited provider availability and cost) or lack of interest or faith in professional help. Furthermore, subclinical symptoms and chronic stress in the absence of a mental illness diagnosis often go unaddressed, despite their substantial health impact. Innovative and scalable treatment delivery methods are needed to supplement traditional therapies to fill these gaps in the mental health care system. OBJECTIVE: This study aims to investigate whether a self-guided web-based course can teach pairs of nonprofessional peers to deliver psychological support to each other. METHODS: In this experimental study, a community sample of 30 dyads (60 participants, mostly friends), many of whom presented with mild to moderate psychological distress, were recruited to complete a web-based counseling skills course. Dyads were randomized to either immediate or delayed access to training. Before and after training, dyads were recorded taking turns discussing stressors. Participants' skills in the helper role were assessed before and after taking the course: the first author and a team of trained research assistants coded recordings for the presence of specific counseling behaviors. When in the client role, participants rated the session on helpfulness in resolving their stressors and supportiveness of their peers. We hypothesized that participants would increase the use of skills taught by the course and decrease the use of skills discouraged by the course, would increase their overall adherence to the guidelines taught in the course, and would perceive posttraining counseling sessions as more helpful and their peers as more supportive. RESULTS: The course had large effects on most helper-role speech behaviors: helpers decreased total speaking time, used more restatements, made fewer efforts to influence the speaker, and decreased self-focused and off-topic utterances (ds=0.8-1.6). When rating the portion of the session in which they served as clients, participants indicated that they made more progress in addressing their stressors during posttraining counseling sessions compared with pretraining sessions (d=1.1), but they did not report substantive changes in feelings of closeness and supportiveness of their peers (d=0.3). CONCLUSIONS: The results provide proof of concept that nonprofessionals can learn basic counseling skills from a scalable web-based course. The course serves as a promising model for the development of web-based counseling skills training, which could provide accessible mental health support to some of those underserved by traditional psychotherapy.

Scaling up behavioral science interventions in online education.

Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, from nearly every country, across 247 online courses offered by Harvard, the Massachusetts Institute of Technology, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically preregistering new hypotheses in between waves of data collection, we identified individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We found minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude. However, iterative scientific investigations can uncover what works where for whom.

The hazards of explanation: overgeneralization in the face of exceptions.

Seeking explanations is central to science, education, and everyday thinking, and prompting learners to explain is often beneficial. Nonetheless, in 2 category learning experiments across artifact and social domains, we demonstrate that the very properties of explanation that support learning can impair learning by fostering overgeneralizations. We find that explaining encourages learners to seek broad patterns, hindering learning when patterns involve exceptions. By revealing how effects of explanation depend on the structure of what is being learned, these experiments simultaneously demonstrate the hazards of explaining and provide evidence for why explaining is so often beneficial. For better or for worse, explaining recruits the remarkable human capacity to seek underlying patterns that go beyond individual observations.