Automated Behavioral Coding to Enhance the Effectiveness of Motivational Interviewing in a Chat-Based Suicide Prevention Helpline: Secondary Analysis of a Clinical Trial.

BACKGROUND: With the rise of computer science and artificial intelligence, analyzing large data sets promises enormous potential in gaining insights for developing and improving evidence-based health interventions. One such intervention is the counseling strategy motivational interviewing (MI), which has been found effective in improving a wide range of health-related behaviors. Despite the simplicity of its principles, MI can be a challenging skill to learn and requires expertise to apply effectively. OBJECTIVE: This study aims to investigate the performance of artificial intelligence models in classifying MI behavior and explore the feasibility of using these models in online helplines for mental health as an automated support tool for counselors in clinical practice. METHODS: We used a coded data set of 253 MI counseling chat sessions from the 113 Suicide Prevention helpline. With 23,982 messages coded with the MI Sequential Code for Observing Process Exchanges codebook, we trained and evaluated 4 machine learning models and 1 deep learning model to classify client- and counselor MI behavior based on language use. RESULTS: The deep learning model BERTje outperformed all machine learning models, accurately predicting counselor behavior (accuracy=0.72, area under the curve [AUC]=0.95, Cohen κ=0.69). It differentiated MI congruent and incongruent counselor behavior (AUC=0.92, κ=0.65) and evocative and nonevocative language (AUC=0.92, κ=0.66). For client behavior, the model achieved an accuracy of 0.70 (AUC=0.89, κ=0.55). The model's interpretable predictions discerned client change talk and sustain talk, counselor affirmations, and reflection types, facilitating valuable counselor feedback. CONCLUSIONS: The results of this study demonstrate that artificial intelligence techniques can accurately classify MI behavior, indicating their potential as a valuable tool for enhancing MI proficiency in online helplines for mental health. Provided that the data set size is sufficiently large with enough training samples for each behavioral code, these methods can be trained and applied to other domains and languages, offering a scalable and cost-effective way to evaluate MI adherence, accelerate behavioral coding, and provide therapists with personalized, quick, and objective feedback.

Detecting changes in help seeker conversations on a suicide prevention helpline during the COVID- 19 pandemic: in-depth analysis using encoder representations from transformers.

BACKGROUND: Preventatives measures to combat the spread of COVID- 19 have introduced social isolation, loneliness and financial stress. This study aims to identify whether the COVID-19 pandemic is related to changes in suicide-related problems for help seekers on a suicide prevention helpline. METHODS: A retrospective cohort study was conducted using chat data from a suicide prevention helpline in the Netherlands. The natural language processing method BERTopic was used to detect common topics in messages from December 1, 2019 until June 1, 2020 (N = 8589). Relative topic occurrence was compared before and during the lock down starting on March 23, 2020. The observed changes in topic usage were likewise analyzed for male and female, younger and older help seekers and help seekers living alone. RESULTS: The topic of the COVID-19 pandemic saw an 808% increase in relative occurrence after the lockdown. Furthermore, the results show that help seeker increased mention of thanking the counsellor (+ 15%), and male and young help seekers were grateful for the conversation (+ 45% and + 32% respectively). Coping methods such as watching TV (- 21%) or listening to music (- 15%) saw a decreased mention. Plans for suicide (- 9%) and plans for suicide at a specific location (- 15%) also saw a decreased mention. However, plans for suicide were mentioned more frequently by help seekers over 30 years old (+ 11%) or who live alone and (+ 52%). Furthermore, male help seekers talked about contact with emergency care (+ 43%) and panic and anxiety (+ 24%) more often. Negative emotions (+ 22%) and lack of self-confidence (+ 15%) were mentioned more often by help seekers under 30, and help seekers over 30 saw an increased mention of substance abuse (+ 9%). CONCLUSION: While mentions of distraction, social interaction and plans for suicide decreased, expressions of gratefulness for the helpline increased, highlighting the importance of contact to help seekers during the lockdown. Help seekers under 30, male or who live alone, showed changes that negatively related to suicidality and should be monitored closely.

Content-Based Recommender Support System for Counselors in a Suicide Prevention Chat Helpline: Design and Evaluation Study.

BACKGROUND: The working environment of a suicide prevention helpline requires high emotional and cognitive awareness from chat counselors. A shared opinion among counselors is that as a chat conversation becomes more difficult, it takes more effort and a longer amount of time to compose a response, which, in turn, can lead to writer's block. OBJECTIVE: This study evaluates and then designs supportive technology to determine if a support system that provides inspiration can help counselors resolve writer's block when they encounter difficult situations in chats with help-seekers. METHODS: A content-based recommender system with sentence embedding was used to search a chat corpus for similar chat situations. The system showed a counselor the most similar parts of former chat conversations so that the counselor would be able to use approaches previously taken by their colleagues as inspiration. In a within-subject experiment, counselors' chat replies when confronted with a difficult situation were analyzed to determine if experts could see a noticeable difference in chat replies that were obtained in 3 conditions: (1) with the help of the support system, (2) with written advice from a senior counselor, or (3) when receiving no help. In addition, the system's utility and usability were measured, and the validity of the algorithm was examined. RESULTS: A total of 24 counselors used a prototype of the support system; the results showed that, by reading chat replies, experts were able to significantly predict if counselors had received help from the support system or from a senior counselor (P=.004). Counselors scored the information they received from a senior counselor (M=1.46, SD 1.91) as significantly more helpful than the information received from the support system or when no help was given at all (M=-0.21, SD 2.26). Finally, compared with randomly selected former chat conversations, counselors rated the ones identified by the content-based recommendation system as significantly more similar to their current chats (ß=.30, P<.001). CONCLUSIONS: Support given to counselors influenced how they responded in difficult conversations. However, the higher utility scores given for the advice from senior counselors seem to indicate that specific actionable instructions are preferred. We expect that these findings will be beneficial for developing a system that can use similar chat situations to generate advice in a descriptive style, hence helping counselors through writer's block.

The Most Effective Interventions for Classification Model Development to Predict Chat Outcomes Based on the Conversation Content in Online Suicide Prevention Chats: Machine Learning Approach.

BACKGROUND: For the provision of optimal care in a suicide prevention helpline, it is important to know what contributes to positive or negative effects on help seekers. Helplines can often be contacted through text-based chat services, which produce large amounts of text data for use in large-scale analysis. OBJECTIVE: We trained a machine learning classification model to predict chat outcomes based on the content of the chat conversations in suicide helplines and identified the counsellor utterances that had the most impact on its outputs. METHODS: From August 2021 until January 2023, help seekers (N=6903) scored themselves on factors known to be associated with suicidality (eg, hopelessness, feeling entrapped, will to live) before and after a chat conversation with the suicide prevention helpline in the Netherlands (113 Suicide Prevention). Machine learning text analysis was used to predict help seeker scores on these factors. Using 2 approaches for interpreting machine learning models, we identified text messages from helpers in a chat that contributed the most to the prediction of the model. RESULTS: According to the machine learning model, helpers' positive affirmations and expressing involvement contributed to improved scores of the help seekers. Use of macros and ending the chat prematurely due to the help seeker being in an unsafe situation had negative effects on help seekers. CONCLUSIONS: This study reveals insights for improving helpline chats, emphasizing the value of an evocative style with questions, positive affirmations, and practical advice. It also underscores the potential of machine learning in helpline chat analysis.