Automating the assessment of multicultural orientation through machine learning and natural language processing.

Recent scholarship has highlighted the value of therapists adopting a multicultural orientation (MCO) within psychotherapy. A newly developed performance-based measure of MCO capacities exists (MCO-performance task [MCO-PT]) in which therapists respond to video-based vignettes of clients sharing culturally relevant information in therapy. The MCO-PT provides scores related to the three aspects of MCO: cultural humility (i.e., adoption of a nonsuperior and other-oriented stance toward clients), cultural opportunities (i.e., seizing or making moments in session to ask about clients' cultural identities), and cultural comfort (i.e., therapists' comfort in cultural conversations). Although a promising measure, the MCO-PT relies on labor-intensive human coding. The present study evaluated the ability to automate the scoring of the MCO-PT transcripts using modern machine learning and natural language processing methods. We included a sample of 100 participants (n = 613 MCO-PT responses). Results indicated that machine learning models were able to achieve near-human reliability on the average across all domains (Spearman's ρ = .75, p < .0001) and opportunity (ρ = .81, p < .0001). Performance was less robust for cultural humility (ρ = .46, p < .001) and was poorest for cultural comfort (ρ = .41, p < .001). This suggests that we may be on the cusp of being able to develop machine learning-based training paradigms that could allow therapists opportunities for feedback and deliberate practice of some key therapist behaviors, including aspects of MCO. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Mental Health Counseling From Conversational Content With Transformer-Based Machine Learning.

Importance: Use of asynchronous text-based counseling is rapidly growing as an easy-to-access approach to behavioral health care. Similar to in-person treatment, it is challenging to reliably assess as measures of process and content do not scale. Objective: To use machine learning to evaluate clinical content and client-reported outcomes in a large sample of text-based counseling episodes of care. Design, Setting, and Participants: In this quality improvement study, participants received text-based counseling between 2014 and 2019; data analysis was conducted from September 22, 2022, to November 28, 2023. The deidentified content of messages was retained as a part of ongoing quality assurance. Treatment was asynchronous text-based counseling via an online and mobile therapy app (Talkspace). Therapists were licensed to provide mental health treatment and were either independent contractors or employees of the product company. Participants were self-referred via online sign-up and received services via their insurance or self-pay and were assigned a diagnosis from their health care professional. Exposure: All clients received counseling services from a licensed mental health clinician. Main Outcomes and Measures: The primary outcomes were client engagement in counseling (number of weeks), treatment satisfaction, and changes in client symptoms, measured via the 8-item version of Patient Health Questionnaire (PHQ-8). A previously trained, transformer-based, deep learning model automatically categorized messages into types of therapist interventions and summaries of clinical content. Results: The total sample included 166 644 clients treated by 4973 therapists (20 600 274 messages). Participating clients were predominantly female (75.23%), aged 26 to 35 years (55.4%), single (37.88%), earned a bachelor's degree (59.13%), and were White (61.8%). There was substantial variability in intervention use and treatment content across therapists. A series of mixed-effects regressions indicated that collectively, interventions and clinical content were associated with key outcomes: engagement (multiple R = 0.43), satisfaction (multiple R = 0.46), and change in PHQ-8 score (multiple R = 0.13). Conclusions and Relevance: This quality improvement study found associations between therapist interventions, clinical content, and client-reported outcomes. Consistent with traditional forms of counseling, higher amounts of supportive counseling were associated with improved outcomes. These findings suggest that machine learning-based evaluations of content may increase the scale and specificity of psychotherapy research.

Improving the quality of counseling and clinical supervision in opioid treatment programs: how can technology help?

BACKGROUND: The opioid epidemic has resulted in expanded substance use treatment services and strained the clinical workforce serving people with opioid use disorder. Focusing on evidence-based counseling practices like motivational interviewing may be of interest to counselors and their supervisors, but time-intensive adherence tasks like recording and feedback are aspirational in busy community-based opioid treatment programs. The need to improve and systematize clinical training and supervision might be addressed by the growing field of machine learning and natural language-based technology, which can promote counseling skill via self- and supervisor-monitoring of counseling session recordings. METHODS: Counselors in an opioid treatment program were provided with an opportunity to use an artificial intelligence based, HIPAA compliant recording and supervision platform (Lyssn.io) to record counseling sessions. We then conducted four focus groups-two with counselors and two with supervisors-to understand the integration of technology with practice and supervision. Questions centered on the acceptability of the clinical supervision software and its potential in an OTP setting; we conducted a thematic coding of the responses. RESULTS: The clinical supervision software was experienced by counselors and clinical supervisors as beneficial to counselor training, professional development, and clinical supervision. Focus group participants reported that the clinical supervision software could help counselors learn and improve motivational interviewing skills. Counselors said that using the technology highlights the value of counseling encounters (versus paperwork). Clinical supervisors noted that the clinical supervision software could help meet national clinical supervision guidelines and local requirements. Counselors and clinical supervisors alike talked about some of the potential challenges of requiring session recording. CONCLUSIONS: Implementing evidence-based counseling practices can help the population served in OTPs; another benefit of focusing on clinical skills is to emphasize and hold up counselors' roles as worthy. Machine learning technology can have a positive impact on clinical practices among counselors and clinical supervisors in opioid treatment programs, settings whose clinical workforce continues to be challenged by the opioid epidemic. Using technology to focus on clinical skill building may enhance counselors' and clinical supervisors' overall experiences in their places of work.

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care.

Background: Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner. Methods: In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes. Results: Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%. Conclusions: These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented. Trial registration: NCT03013309 ClinicalTrials.gov.

Research has shown that evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. However, if they are not implemented with fidelity, there is a potential that they will not produce the same effects. Gold-standard methods of implementation monitoring include observations of program sessions. This is expensive and difficult to implement in delivery settings with limited resources. Using data from a trial of the Family Check-Up 4 Health program in primary care settings that served Latino families, we investigated the potential to make use of a form of machine learning called natural language processing (NLP) to monitor program delivery. NLP-based ratings were significantly associated with independent observer ratings of fidelity and participant engagement outcomes. These results demonstrate the potential for NLP methods to monitor implementation in evidence-based parenting programs delivered at scale.

The Silent Treatment?: Changes in patient emotional expression after silence.

Psychotherapy can be an emotionally laden conversation, where both verbal and non-verbal interventions may impact the therapeutic process. Prior research has postulated mixed results in how clients emotionally react following a silence after the therapist is finished talking, potentially due to studying a limited range of silences with primarily qualitative and self-report methodologies. A quantitative exploration may illuminate new findings. Utilizing research and automatic data processing from the field of linguistics, we analysed the full range of silence lengths (0.2 to 24.01 seconds), and measures of emotional expression - vocally encoded arousal and emotional valence from the works spoken - of 84 audio recordings Motivational Interviewing sessions. We hypothesized that both the level and the variance of client emotional expression would change as a function of silence length, however, due to the mixed results in the literature the direction of emotional change was unclear. We conducted a multilevel linear regression to examine how the level of client emotional expression changed across silence length, and an ANOVA to examine the variability of client emotional expression across silence lengths. Results indicated in both analyses that as silence length increased, emotional expression largely remained the same. Broadly, we demonstrated a weak connection between silence length and emotional expression, indicating no persuasive evidence that silence leads to client emotional processing and expression.

Injunctive and descriptive normative feedback for college drinking prevention: Is the whole greater than the sum of its parts?

OBJECTIVE: Single-component personalized normative feedback (PNF) interventions and multicomponent personalized feedback interventions (PFI) have been shown to reduce alcohol consumption among college students. The present study compared the efficacy of PNF interventions targeting descriptive norms alone (descriptive PNF), injunctive norms alone (injunctive PNF), or their combination (combined PNF), against a multicomponent PFI and an attention control condition. METHOD: Undergraduates (N = 1,137) across two universities who reported a minimum of one past-month episode of heavy episodic drinking (i.e., 4 +/5 + drinks on a single occasion for females/males) completed assessments at baseline and 3, 6, and 12 months postintervention. RESULTS: Relative to the attention control, participants in each of the four intervention conditions showed greater reductions in perceived descriptive/injunctive norms, total drinks per week, and alcohol-related consequences. Peak estimated blood alcohol concentration was also reduced in the injunctive PNF, combined PNF, and multicomponent PFI conditions, with the latter two conditions showing an advantage for duration of effects. The multicomponent PFI condition also evidenced greater reductions than the injunctive PNF in descriptive norms at 3-month and injunctive norms at 6- and 12-month follow-ups. No other group comparisons on any outcome were significant. CONCLUSIONS: Each intervention has merit for use in college student harm reduction efforts. Single-component or combined PNF could be considered a potential starting point, as PNF is less burdensome than a multicomponent PFI when considering ease and length of delivery. Results can inform optimization of norms-based interventions and guide recommendations on efficacious components for reducing alcohol use and harms on college campuses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): a study protocol.

BACKGROUND: Each year, millions of Americans receive evidence-based psychotherapies (EBPs) like cognitive behavioral therapy (CBT) for the treatment of mental and behavioral health problems. Yet, at present, there is no scalable method for evaluating the quality of psychotherapy services, leaving EBP quality and effectiveness largely unmeasured and unknown. Project AFFECT will develop and evaluate an AI-based software system to automatically estimate CBT fidelity from a recording of a CBT session. Project AFFECT is an NIMH-funded research partnership between the Penn Collaborative for CBT and Implementation Science and Lyssn.io, Inc. ("Lyssn") a start-up developing AI-based technologies that are objective, scalable, and cost efficient, to support training, supervision, and quality assurance of EBPs. Lyssn provides HIPAA-compliant, cloud-based software for secure recording, sharing, and reviewing of therapy sessions, which includes AI-generated metrics for CBT. The proposed tool will build from and be integrated into this core platform. METHODS: Phase I will work from an existing software prototype to develop a LyssnCBT user interface geared to the needs of community mental health (CMH) agencies. Core activities include a user-centered design focus group and interviews with community mental health therapists, supervisors, and administrators to inform the design and development of LyssnCBT. LyssnCBT will be evaluated for usability and implementation readiness in a final stage of Phase I. Phase II will conduct a stepped-wedge, hybrid implementation-effectiveness randomized trial (N = 1,875 clients) to evaluate the effectiveness of LyssnCBT to improve therapist CBT skills and client outcomes and reduce client drop-out. Analyses will also examine the hypothesized mechanism of action underlying LyssnCBT. DISCUSSION: Successful execution will provide automated, scalable CBT fidelity feedback for the first time ever, supporting high-quality training, supervision, and quality assurance, and providing a core technology foundation that could support the quality delivery of a range of EBPs in the future. TRIAL REGISTRATION: ClinicalTrials.gov; NCT05340738 ; approved 4/21/2022.

Delivering Remote Measurement-Based Care in Community Addiction Treatment: Engagement and Usability Over a 6-Month Clinical Pilot.

Objective: Measurement-based care (MBC) is an evidence-based practice in which patients routinely complete standardized measures throughout treatment to help monitor clinical progress and inform clinical decision-making. Despite its potential benefits, MBC is rarely used in community-based substance use disorder (SUD) treatment. In this pilot study, we evaluated the feasibility of incorporating a digital and remotely delivered MBC system into SUD treatment within a community setting by characterizing patients' and clinicians' engagement with and usability ratings toward the MBC system that was piloted. Methods: A pilot study was conducted with 30 patients receiving SUD treatment and eight clinicians providing SUD treatment in a large, publicly funded addiction and mental health treatment clinic. Services as usual within the clinic included individual psychotherapy, case management, group therapy, peer support, and medication management for mental health and SUD, including buprenorphine. Patients who enrolled in the pilot continued to receive services as usual and were automatically sent links to complete a 22-item questionnaire, called the weekly check-in, via text message or email weekly for 24 weeks. Results of the weekly check-in were summarized on a clinician-facing web-based dashboard. Engagement was characterized by calculating the mean number of weekly check-ins completed by patients and the mean number times clinicians logged into the MBC system. Ratings of the MBC system's usability and clinical utility were provided by patients and clinicians. Results: Patient participants (53.3% male, 56.7% white, 90% Medicaid enrolled) completed a mean of 20.60 weekly check-ins (i.e., 85.8% of the 24 expected per patient). All but one participating clinician with a patient enrolled in the study logged into the clinician-facing dashboard at least once, with an average of 12.20 logins per clinician. Patient and clinician ratings of usability and clinical utility were favorable: most patients agreed with statements that the weekly check-in was easy to navigate and aided self-reflection. All clinicians who completed usability questionnaires agreed with statements indicating that the dashboard was easy to navigate and that it provided meaningful information for SUD treatment. Conclusions: A digital and remotely delivered MBC system can yield high rates of patient and clinician engagement and high ratings of usability and clinical utility when added into SUD treatment as usual. The success of this clinical pilot may be attributable, in part, to the user-centered design processes that were used to develop and refine the MBC system that was piloted. Future efforts may focus on strategies to test whether MBC can be sustainably implemented and offers clinical benefits to patients in community SUD treatment settings.

An Automated Quality Evaluation Framework of Psychotherapy Conversations with Local Quality Estimates.

Text-based computational approaches for assessing the quality of psychotherapy are being developed to support quality assurance and clinical training. However, due to the long durations of typical conversation based therapy sessions, and due to limited annotated modeling resources, computational methods largely rely on frequency-based lexical features or dialogue acts to assess the overall session level characteristics. In this work, we propose a hierarchical framework to automatically evaluate the quality of transcribed Cognitive Behavioral Therapy (CBT) interactions. Given the richly dynamic nature of the spoken dialog within a talk therapy session, to evaluate the overall session level quality, we propose to consider modeling it as a function of local variations across the interaction. To implement that empirically, we divide each psychotherapy session into conversation segments and initialize the segment-level qualities with the session-level scores. First, we produce segment embeddings by fine-tuning a BERT-based model, and predict segment-level (local) quality scores. These embeddings are used as the lower-level input to a Bidirectional LSTM-based neural network to predict the session-level (global) quality estimates. In particular, we model the global quality as a linear function of the local quality scores, which allows us to update the segment-level quality estimates based on the session-level quality prediction. These newly estimated segment-level scores benefit the BERT fine-tuning process, which in turn results in better segment embeddings. We evaluate the proposed framework on automatically derived transcriptions from real-world CBT clinical recordings to predict session-level behavior codes. The results indicate that our approach leads to improved evaluation accuracy for most codes when used for both regression and classification tasks.

Knowledge and Attitudes Toward an Artificial Intelligence-Based Fidelity Measurement in Community Cognitive Behavioral Therapy Supervision.

To capitalize on investments in evidence-based practices, technology is needed to scale up fidelity assessment and supervision. Stakeholder feedback may facilitate adoption of such tools. This evaluation gathered stakeholder feedback and preferences to explore whether it would be fundamentally feasible or possible to implement an automated fidelity-scoring supervision tool in community mental health settings. A partially mixed, sequential research method design was used including focus group discussions with community mental health therapists (n = 18) and clinical leadership (n = 12) to explore typical supervision practices, followed by discussion of an automated fidelity feedback tool embedded in a cloud-based supervision platform. Interpretation of qualitative findings was enhanced through quantitative measures of participants' use of technology and perceptions of acceptability, appropriateness, and feasibility of the tool. Initial perceptions of acceptability, appropriateness, and feasibility of automated fidelity tools were positive and increased after introduction of an automated tool. Standard supervision was described as collaboratively guided and focused on clinical content, self-care, and documentation. Participants highlighted the tool's utility for supervision, training, and professional growth, but questioned its ability to evaluate rapport, cultural responsiveness, and non-verbal communication. Concerns were raised about privacy and the impact of low scores on therapist confidence. Desired features included intervention labeling and transparency about how scores related to session content. Opportunities for asynchronous, remote, and targeted supervision were particularly valued. Stakeholder feedback suggests that automated fidelity measurement could augment supervision practices. Future research should examine the relations among use of such supervision tools, clinician skill, and client outcomes.

Automated evaluation of psychotherapy skills using speech and language technologies.

With the growing prevalence of psychological interventions, it is vital to have measures which rate the effectiveness of psychological care to assist in training, supervision, and quality assurance of services. Traditionally, quality assessment is addressed by human raters who evaluate recorded sessions along specific dimensions, often codified through constructs relevant to the approach and domain. This is, however, a cost-prohibitive and time-consuming method that leads to poor feasibility and limited use in real-world settings. To facilitate this process, we have developed an automated competency rating tool able to process the raw recorded audio of a session, analyzing who spoke when, what they said, and how the health professional used language to provide therapy. Focusing on a use case of a specific type of psychotherapy called "motivational interviewing", our system gives comprehensive feedback to the therapist, including information about the dynamics of the session (e.g., therapist's vs. client's talking time), low-level psychological language descriptors (e.g., type of questions asked), as well as other high-level behavioral constructs (e.g., the extent to which the therapist understands the clients' perspective). We describe our platform and its performance using a dataset of more than 5000 recordings drawn from its deployment in a real-world clinical setting used to assist training of new therapists. Widespread use of automated psychotherapy rating tools may augment experts' capabilities by providing an avenue for more effective training and skill improvement, eventually leading to more positive clinical outcomes.

Multi-label Multi-task Deep Learning for Behavioral Coding.

We propose a methodology for estimating human behaviors in psychotherapy sessions using mutli-label and multi-task learning paradigms. We discuss the problem of behavioral coding in which data of human interactions is the annotated with labels to describe relevant human behaviors of interest. We describe two related, yet distinct, corpora consisting of therapist client interactions in psychotherapy sessions. We experimentally compare the proposed learning approaches for estimating behaviors of interest in these datasets. Specifically, we compare single and multiple label learning approaches, single and multiple task learning approaches, and evaluate the performance of these approaches when incorporating turn context. We demonstrate the prediction performance gains which can be achieved by using the proposed paradigms and discuss the insights these models provide into these complex interactions.

Feature Fusion Strategies for End-to-End Evaluation of Cognitive Behavior Therapy Sessions.

Cognitive Behavioral Therapy (CBT) is a goal-oriented psychotherapy for mental health concerns implemented in a conversational setting. The quality of a CBT session is typically assessed by trained human raters who manually assign pre-defined session-level behavioral codes. In this paper, we develop an end-to-end pipeline that converts speech audio to diarized and transcribed text and extracts linguistic features to code the CBT sessions automatically. We investigate both word-level and utterance-level features and propose feature fusion strategies to combine them. The utterance level features include dialog act tags as well as behavioral codes drawn from another well-known talk psychotherapy called Motivational Interviewing (MI). We propose a novel method to augment the word-based features with the utterance level tags for subsequent CBT code estimation. Experiments show that our new fusion strategy outperforms all the studied features, both when used individually and when fused by direct concatenation. We also find that incorporating a sentence segmentation module can further improve the overall system given the preponderance of multi-utterance conversational turns in CBT sessions.

Harnessing Innovative Technologies to Train Nurses in Suicide Safety Planning With Hospitalized Patients: Protocol for Formative and Pilot Feasibility Research.

BACKGROUND: Suicide is the 10th leading cause of death in the United States, with >47,000 deaths in 2019. Most people who died by suicide had contact with the health care system in the year before their death. Health care provider training is a top research priority identified by the National Action Alliance for Suicide Prevention; however, evidence-based approaches that target skill-building are resource intensive and difficult to implement. Advances in artificial intelligence technology hold promise for improving the scalability and sustainability of training methods, as it is now possible for computers to assess the intervention delivery skills of trainees and provide feedback to guide skill improvements. Much remains to be known about how best to integrate these novel technologies into continuing education for health care providers. OBJECTIVE: In Project WISE (Workplace Integrated Support and Education), we aim to develop e-learning training in suicide safety planning, enhanced with novel skill-building technologies that can be integrated into the routine workflow of nurses serving patients hospitalized for medical or surgical reasons or traumatic injury. The research aims include identifying strategies for the implementation and workflow integration of both the training and safety planning with patients, adapting 2 existing technologies to enhance general counseling skills for use in suicide safety planning (a conversational agent and an artificial intelligence-based feedback system), observing training acceptability and nurse engagement with the training components, and assessing the feasibility of recruitment, retention, and collection of longitudinal self-report and electronic health record data for patients identified as at risk of suicide. METHODS: Our developmental research includes qualitative and observational methods to explore the implementation context and technology usability, formative evaluation of the training paradigm, and pilot research to assess the feasibility of conducting a future cluster randomized pragmatic trial. The trial will examine whether patients hospitalized for medical or surgical reasons or traumatic injury who are at risk of suicide have better suicide-related postdischarge outcomes when admitted to a unit with nurses trained using the skill-building technology than those admitted to a unit with untrained nurses. The research takes place at a level 1 trauma center, which is also a safety-net hospital and academic medical center. RESULTS: Project WISE was funded in July 2019. As of September 2021, we have completed focus groups and usability testing with 27 acute care and 3 acute and intensive care nurses. We began data collection for research aims 3 and 4 in November 2021. All research has been approved by the University of Washington institutional review board. CONCLUSIONS: Project WISE aims to further the national agenda to improve suicide prevention in health care settings by training nurses in suicide prevention with medically hospitalized patients using novel e-learning technologies. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/33695.

Automated quality assessment of cognitive behavioral therapy sessions through highly contextualized language representations.

During a psychotherapy session, the counselor typically adopts techniques which are codified along specific dimensions (e.g., 'displays warmth and confidence', or 'attempts to set up collaboration') to facilitate the evaluation of the session. Those constructs, traditionally scored by trained human raters, reflect the complex nature of psychotherapy and highly depend on the context of the interaction. Recent advances in deep contextualized language models offer an avenue for accurate in-domain linguistic representations which can lead to robust recognition and scoring of such psychotherapy-relevant behavioral constructs, and support quality assurance and supervision. In this work, we propose a BERT-based model for automatic behavioral scoring of a specific type of psychotherapy, called Cognitive Behavioral Therapy (CBT), where prior work is limited to frequency-based language features and/or short text excerpts which do not capture the unique elements involved in a spontaneous long conversational interaction. The model focuses on the classification of therapy sessions with respect to the overall score achieved on the widely-used Cognitive Therapy Rating Scale (CTRS), but is trained in a multi-task manner in order to achieve higher interpretability. BERT-based representations are further augmented with available therapy metadata, providing relevant non-linguistic context and leading to consistent performance improvements. We train and evaluate our models on a set of 1,118 real-world therapy sessions, recorded and automatically transcribed. Our best model achieves an F1 score equal to 72.61% on the binary classification task of low vs. high total CTRS.

What do clinicians want? Understanding frontline addiction treatment clinicians' preferences and priorities to improve the design of measurement-based care technology.

BACKGROUND: Measurement-based care (MBC) is the practice of routinely administering standardized measures to support clinical decision-making and monitor treatment progress. Despite evidence of its effectiveness, MBC is rarely adopted in routine substance use disorder (SUD) treatment settings and little is known about the factors that may improve its adoptability in these settings. The current study gathered qualitative data from SUD treatment clinicians about their perceptions of MBC, the clinical outcomes they would most like to monitor in MBC, and suggestions for the design and implementation of MBC systems in their settings. METHODS: Fifteen clinicians from one publicly-funded and two privately-funded outpatient SUD treatment clinics participated in one-on-one research interviews. Interviews focused on clinicians' perceived benefits, drawbacks, and ideas related to implementing MBC technology into their clinical workflows. Interviews were audio recorded, transcribed, and coded to allow for thematic analysis using a mixed deductive and inductive approach. Clinicians also completed a card sorting task to rate the perceived helpfulness of routinely measuring and monitoring different treatment outcomes. RESULTS: Clinicians reported several potential benefits of MBC, including improved patient-provider communication, client empowerment, and improved communication between clinicians. Clinicians also expressed potential drawbacks, including concerns about subjectivity in patient self-reports, limits to personalization, increased time burdens, and needing to learn to use new technologies. Clinicians generated several ideas and preferences aimed at minimizing burden of MBC, illustrating clinical changes over time, improving ease of use, and improving personalization. Numerous patient outcomes were identified as "very helpful" to track, including coping skills, social support, and motivation for change. CONCLUSIONS: MBC may be a beneficial tool for improving clinical care in SUD treatment settings. MBC tools may be particularly adoptable if they are compatible with existing workflows, help illustrate gradual and nonlinear progress in SUD treatment, measure outcomes perceived as clinically useful, accommodate multiple use cases and stakeholder groups, and are framed as an additional source of information meant to augment, rather than replace, existing practices and information sources.

Health behaviour outcomes of a family based intervention for paediatric obesity in primary care: A randomized type II hybrid effectiveness-implementation trial.

BACKGROUND: Paediatric obesity is a multifaceted public health problem. Family based behavioural interventions are the recommended approach for the prevention of excess weight gain in children and adolescents, yet few have been tested under "real-world" conditions. OBJECTIVES: To evaluate the effectiveness of a family based intervention, delivered in coordination with paediatric primary care, on child and family health outcomes. METHODS: A sample of 240 families with racially and ethnically diverse (86% non-White) and predominantly low-income children (49% female) ages 6 to 12 years (M = 9.5 years) with body mass index (BMI) ≥85th percentile for age and gender were identified in paediatric primary care. Participants were randomized to either the Family Check-Up 4 Health (FCU4Health) program (N = 141) or usual care plus information (N = 99). FCU4Health, an assessment-driven individually tailored intervention designed to preempt excess weight gain by improving parenting skills was delivered for 6 months in clinic, at home and in the community. Child BMI and body fat were assessed using a bioelectrical impedance scale and caregiver-reported health behaviours (eg, diet, physical activity and family health routines) were obtained at baseline, 3, 6 and 12 months. RESULTS: Change in child BMI and percent body fat did not differ by group assignment. Path analysis indicated significant group differences in child health behaviours at 12 months, mediated by improved family health routines at 6 months. CONCLUSION: The FCU4Health, delivered in coordination with paediatric primary care, significantly impacted child and family health behaviours that are associated with the development and maintenance of paediatric obesity. BMI did not significantly differ.

How do you feel? Using natural language processing to automatically rate emotion in psychotherapy.

Emotional distress is a common reason for seeking psychotherapy, and sharing emotional material is central to the process of psychotherapy. However, systematic research examining patterns of emotional exchange that occur during psychotherapy sessions is often limited in scale. Traditional methods for identifying emotion in psychotherapy rely on labor-intensive observer ratings, client or therapist ratings obtained before or after sessions, or involve manually extracting ratings of emotion from session transcripts using dictionaries of positive and negative words that do not take the context of a sentence into account. However, recent advances in technology in the area of machine learning algorithms, in particular natural language processing, have made it possible for mental health researchers to identify sentiment, or emotion, in therapist-client interactions on a large scale that would be unattainable with more traditional methods. As an attempt to extend prior findings from Tanana et al. (2016), we compared their previous sentiment model with a common dictionary-based psychotherapy model, LIWC, and a new NLP model, BERT. We used the human ratings from a database of 97,497 utterances from psychotherapy to train the BERT model. Our findings revealed that the unigram sentiment model (kappa = 0.31) outperformed LIWC (kappa = 0.25), and ultimately BERT outperformed both models (kappa = 0.48).

Automated rating of patient and physician emotion in primary care visits.

OBJECTIVE: Train machine learning models that automatically predict emotional valence of patient and physician in primary care visits. METHODS: Using transcripts from 353 primary care office visits with 350 patients and 84 physicians (Cook, 2002 [1], Tai-Seale et al., 2015 [2]), we developed two machine learning models (a recurrent neural network with a hierarchical structure and a logistic regression classifier) to recognize the emotional valence (positive, negative, neutral) (Posner et al., 2005 [3]) of each utterance. We examined the agreement of human-generated ratings of emotional valence with machine learning model ratings of emotion. RESULTS: The agreement of emotion ratings from the recurrent neural network model with human ratings was comparable to that of human-human inter-rater agreement. The weighted-average of the correlation coefficients for the recurrent neural network model with human raters was 0.60, and the human rater agreement was also 0.60. CONCLUSIONS: The recurrent neural network model predicted the emotional valence of patients and physicians in primary care visits with similar reliability as human raters. PRACTICE IMPLICATIONS: As the first machine learning-based evaluation of emotion recognition in primary care visit conversations, our work provides valuable baselines for future applications that might help monitor patient emotional signals, supporting physicians in empathic communication, or examining the role of emotion in patient-centered care.

Can a computer detect interpersonal skills? Using machine learning to scale up the Facilitative Interpersonal Skills task.

Objective: Therapist interpersonal skills are foundational to psychotherapy. However, assessment is labor intensive and infrequent. This study evaluated if machine learning (ML) tools can automatically assess therapist interpersonal skills. Method: Data were drawn from a previous study in which 164 undergraduate students (i.e., not clinical trainees) completed the Facilitative Interpersonal Skills (FIS) task. This task involves responding to video vignettes depicting interpersonally challenging moments in psychotherapy. Trained raters scored the responses. We used an elastic net model on top of a term frequency-inverse document frequency representation to predict FIS scores. Results: Models predicted FIS total and item-level scores above chance (rhos = .27-.53, ps < .001), achieving 31-60% of human reliability. Models explained 13-24% of the variance in FIS total and item-level scores on a held out set of data (R2), with the exception of the two items most reliant on vocal cues (verbal fluency, emotional expression), for which models explained ≤1% of variance. Conclusion: ML may be a promising approach for automating assessment of constructs like interpersonal skill previously coded by humans. ML may perform best when the standardized stimuli limit the "space" of potential responses (vs. naturalistic psychotherapy) and when models have access to the same data available to raters (i.e., transcripts).