Automating the assessment of multicultural orientation through machine learning and natural language processing.

Recent scholarship has highlighted the value of therapists adopting a multicultural orientation (MCO) within psychotherapy. A newly developed performance-based measure of MCO capacities exists (MCO-performance task [MCO-PT]) in which therapists respond to video-based vignettes of clients sharing culturally relevant information in therapy. The MCO-PT provides scores related to the three aspects of MCO: cultural humility (i.e., adoption of a nonsuperior and other-oriented stance toward clients), cultural opportunities (i.e., seizing or making moments in session to ask about clients' cultural identities), and cultural comfort (i.e., therapists' comfort in cultural conversations). Although a promising measure, the MCO-PT relies on labor-intensive human coding. The present study evaluated the ability to automate the scoring of the MCO-PT transcripts using modern machine learning and natural language processing methods. We included a sample of 100 participants (n = 613 MCO-PT responses). Results indicated that machine learning models were able to achieve near-human reliability on the average across all domains (Spearman's ρ = .75, p < .0001) and opportunity (ρ = .81, p < .0001). Performance was less robust for cultural humility (ρ = .46, p < .001) and was poorest for cultural comfort (ρ = .41, p < .001). This suggests that we may be on the cusp of being able to develop machine learning-based training paradigms that could allow therapists opportunities for feedback and deliberate practice of some key therapist behaviors, including aspects of MCO. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Automated quality assessment of cognitive behavioral therapy sessions through highly contextualized language representations.

During a psychotherapy session, the counselor typically adopts techniques which are codified along specific dimensions (e.g., 'displays warmth and confidence', or 'attempts to set up collaboration') to facilitate the evaluation of the session. Those constructs, traditionally scored by trained human raters, reflect the complex nature of psychotherapy and highly depend on the context of the interaction. Recent advances in deep contextualized language models offer an avenue for accurate in-domain linguistic representations which can lead to robust recognition and scoring of such psychotherapy-relevant behavioral constructs, and support quality assurance and supervision. In this work, we propose a BERT-based model for automatic behavioral scoring of a specific type of psychotherapy, called Cognitive Behavioral Therapy (CBT), where prior work is limited to frequency-based language features and/or short text excerpts which do not capture the unique elements involved in a spontaneous long conversational interaction. The model focuses on the classification of therapy sessions with respect to the overall score achieved on the widely-used Cognitive Therapy Rating Scale (CTRS), but is trained in a multi-task manner in order to achieve higher interpretability. BERT-based representations are further augmented with available therapy metadata, providing relevant non-linguistic context and leading to consistent performance improvements. We train and evaluate our models on a set of 1,118 real-world therapy sessions, recorded and automatically transcribed. Our best model achieves an F1 score equal to 72.61% on the binary classification task of low vs. high total CTRS.

Smartphone-Based Passive Assessment of Mobility in Depression: Challenges and Opportunities.

Advances in technology have ushered in exciting potential for smartphone sensors to inform mental health care. This commentary addresses the practical challenges of collecting smartphone-based physical activity data. Using data (N = 353) from a large scale, fully remote randomized clinical trial for depression, we discuss findings and limitations associated with using passively collected mobility data to make inferences about depressive symptom severity. We highlight a range of issues in associating mobility data with mental health symptoms, including a high degree of variability, data featurization, granularity, and sparsity. Given the considerable efforts toward leveraging technology in mental health care, it is important to consider these challenges to optimize assessment and guide best practices. Clinical Trials.gov identifier: NCT01808976.

Comparing Approaches to Mobile Depression Assessment for Measurement-Based Care: Prospective Study.

BACKGROUND: To inform measurement-based care, practice guidelines suggest routine symptom monitoring, often on a weekly or monthly basis. Increasingly, patient-provider contacts occur remotely (eg, by telephone and Web-based portals), and mobile health tools can now monitor depressed mood daily or more frequently. However, the reliability and utility of daily ratings are unclear. OBJECTIVE: This study aimed to examine the association between a daily depressive symptom measure and the Patient Health Questionnaire-9 (PHQ-9), the most widely adopted depression self-report measure, and compare how well these 2 assessment methods predict patient outcomes. METHODS: A total of 547 individuals completed smartphone-based measures, including the Patient Health Questionnaire-2 (PHQ-2) modified for daily administration, the PHQ-9, and the Sheehan Disability Scale. Multilevel factor analyses evaluated the reliability of latent depression based on the PHQ-2 (for repeated measures) between weeks 2 and 4 and its correlation with the PHQ-9 at week 4. Regression models predicted week 8 depressive symptoms and disability ratings with daily PHQ-2 and PHQ-9. RESULTS: The daily PHQ-2 and PHQ-9 are highly reliable (range: 0.80-0.88) and highly correlated (r=.80). Findings were robust across demographic groups (age, gender, and ethnic minority status). Daily PHQ-2 and PHQ-9 were comparable in predicting week 8 disability and were independent predictors of week 8 depressive symptoms and disability, though the unique contribution of the PHQ-2 was small in magnitude. CONCLUSIONS: Daily completion of the PHQ-2 is a reasonable proxy for the PHQ-9 and is comparable to the PHQ-9 in predicting future outcomes. Mobile assessment methods offer researchers and clinicians reliable and valid new methods for depression assessment that may be leveraged for measurement-based depression care.

A Randomized Controlled Trial of the Collaborative Assessment and Management of Suicidality versus Enhanced Care as Usual With Suicidal Soldiers.

OBJECTIVE: This study describes a randomized controlled trial called "Operation Worth Living" (OWL) which compared the use of the Collaborative Assessment and Management of Suicidality (CAMS) to enhanced care as usual (E-CAU). We hypothesized that CAMS would be more effective than E-CAU for reducing suicidal ideation (SI) and suicide attempts (SA), along with secondary behavioral health and health care utilization markers for U.S. Army Soldier outpatients with significant SI (i.e., > 13 on Beck's Scale for Suicide Ideation). METHOD: Study participants were 148 Soldiers who presented to a military outpatient behavioral health clinic. There were 73 Soldiers in the experimental arm of the trial who received adherent CAMS; 75 Soldiers received E-CAU. Nine a-priori treatment outcomes (SI, past year SA, suicide-related emergency department (ED) admits, behavioral health-related ED admits, suicide-related inpatient psychiatric unit (IPU) days, behavioral health-related IPU days, mental health, psychiatric distress, resiliency) were measured through assessments at Baseline and at 1, 3, 6, and 12 months post-Baseline (with a 78% retention of intent-to-treat participants at 12 months). RESULTS: Soldiers in both arms of the trial responded to study treatments in terms of all primary and secondary outcomes (effect sizes ranged from 0.63 to 12.04). CAMS participants were significantly less likely to have any suicidal thoughts by 3 months in comparison to those in E-CAU (Cohen's d = 0.93, p=.028). CONCLUSIONS: Soldiers receiving CAMS and E-CAU significantly improved post-treatment. Those who received CAMS were less likely to report SI at 3 months; further group differences were not otherwise seen.

Barriers and Facilitators to Coordinating Care With High-Risk, High-Cost Disabled Medicaid Beneficiaries: Perspectives of Frontline Staff and Participating Clients.

This evaluation was designed to examine the perspectives of 15 frontline staff who implemented a managed care program and 154 high-risk, high-cost disabled Medicaid clients who were participants in the program. Results indicated that positive relationships between staff and clients played a key role in facilitating program implementation. Challenges included finding ways to provide a wide breadth of services including food, shelter, and transportation; handling difficulties following from staff turnover; and creating transitions of care for clients to community health clinics. Staff identified training in motivational interviewing and having both nurse care managers and social workers collaboratively deliver the intervention as among the most powerful components of the program. Staff and clients expressed high levels of satisfaction with the program, and clients believed they were experiencing positive impacts of the program. Lessons learned from this study may inform the design of services as the Affordable Care Act continues to unfold.

"It sounds like...": A natural language processing approach to detecting counselor reflections in motivational interviewing.

The dissemination and evaluation of evidence-based behavioral treatments for substance abuse problems rely on the evaluation of counselor interventions. In Motivational Interviewing (MI), a treatment that directs the therapist to utilize a particular linguistic style, proficiency is assessed via behavioral coding-a time consuming, nontechnological approach. Natural language processing techniques have the potential to scale up the evaluation of behavioral treatments such as MI. We present a novel computational approach to assessing components of MI, focusing on 1 specific counselor behavior-reflections, which are believed to be a critical MI ingredient. Using 57 sessions from 3 MI clinical trials, we automatically detected counselor reflections in a maximum entropy Markov modeling framework using the raw linguistic data derived from session transcripts. We achieved 93% recall, 90% specificity, and 73% precision. Results provide insight into the linguistic information used by coders to make ratings and demonstrate the feasibility of new computational approaches to scaling up the evaluation of behavioral treatments.

Predicting medication adherence in multiple sclerosis using telephone-based home monitoring.

BACKGROUND: Poor medication adherence exerts a substantial negative impact on the health and well-being of individuals with multiple sclerosis (MS). Improving adherence rates requires a proactive approach of frequent and ongoing monitoring; however, this can be difficult to achieve within traditional, reactive health care systems that generally emphasize acute care services. Telephone-based home monitoring may circumvent these barriers and facilitate optimal care coordination and management for individuals with MS and other chronic illnesses. OBJECTIVE: The current study evaluated the utility of a one-item, telephone-administered measure of adherence expectations as a prospective predictor of medication adherence across a six month period among individuals with MS. METHODS: As part of a longitudinal study, Veterans with MS (N = 89) who were receiving medical services through the Veterans Health Administration completed monthly telephone-based interviews for six months. RESULTS: Using mixed model regression analyses, adherence expectations predicted adherence after adjusting for demographic, illness-related, and psychosocial factors (B = -5.54, p < .01). CONCLUSIONS: Brief, telephone-based assessments of adherence expectations may represent an easy and efficient method for monitoring medication use among individuals with MS. The results offer an efficient method to detect and provide support for individuals who may benefit from interventions to promote medication adherence.

Clinical Needs of Patients with Problem Drug Use.

INTRODUCTION: Illicit drug use is a serious public health problem associated with significant co-occurring medical disorders, mental disorders, and social problems. Yet most individuals with drug use disorders have never been treated, though they often seek medical treatment in primary care. The purpose of this study was to examine the baseline characteristics of people presenting in primary care with a range of problem drug use severity to identify their clinical needs. METHODS: We examined sociodemographic characteristics, medical and psychiatric comorbidities, drug use severity, social and legal problems, and service utilization for 868 patients with drug problems. These patients were recruited from primary care clinics in a medical safety net setting. Based on Drug Abuse Screening Test results, individuals were categorized as having low, intermediate, or substantial/severe drug use severity. RESULTS: Patients with substantial/severe drug use severity had serious drug use (opiates, stimulants, sedatives, intravenous drugs); high levels of homelessness (50%), psychiatric comorbidity (69%), and arrests for serious crimes (24%); and frequent use of expensive emergency department and inpatient hospitals. Patients with low drug use severity were primarily users of marijuana, with little reported use of other drugs, less psychiatric comorbidity, and more stable lifestyles. Patients with intermediate drug use severity fell in between the substantial/severe and low drug use severity subgroups on most variables. CONCLUSIONS: Patients with the highest drug use severity are likely to require specialized psychiatric and substance abuse care, in addition to ongoing medical care that is equipped to address the consequences of severe/substantial drug use, including intravenous drug use. Because of their milder symptoms, patients with low drug use severity may benefit from a collaborative care model that integrates psychiatric and substance abuse care in the primary care setting. Patients with intermediate drug use severity may benefit from selective application of interventions suggested for patients with the highest and lowest drug use severity. Primary care safety net clinics are in a key position to serve patients with problem drug use by developing a range of responses that are locally effective and that may also inform national efforts to establish patient-centered medical homes and to implement the Affordable Care Act.

Removing very low-performing therapists: A simulation of performance-based retention in psychotherapy.

Therapists can impact the likelihood a given patient will benefit from psychotherapy. However, therapists are rarely held accountable for their patients' outcomes. As a result, low-performing providers likely continue to practice alongside providers with high response rates. In the current study, we conducted a Monte Carlo simulation to illustrate a thought experiment-what happens to patient outcomes if therapists with the worst outcomes were removed from practice? We drew initial samples of 50 therapists from 3 simulated populations of 1,000 therapists with a mean patient response rate of 50% and different effect sizes for therapist variability in outcomes. We simulated 30 patient outcomes for each therapist, with outcome defined as response to treatment versus no response. We removed therapists with response rates in the bottom 5% and replaced them with a random sample of therapists from the population. Over 10 years, the difference in responses between the lowest and highest performing therapists was substantial (between 697 and 997 additional responses to treatment). After repeatedly removing the lowest performing providers 40 times (simulating a 10-year time span), response rates increased substantially. The cumulative number of patient responses (i.e., summing the total number of responses across 10 years) increased by 4,266, 6,404, and 9,307 when therapists accounted for 5%, 10%, or 20% of the patient outcome variance, respectively. These findings indicate that performance-based retention of therapists could improve the quality of psychotherapy in health systems by improving the average response rate and decreasing the probability that a patient will be treated by a therapist who consistently has poor outcomes.

A randomized controlled trial of intensive care management for disabled Medicaid beneficiaries with high health care costs.

OBJECTIVE: To evaluate outcomes of a registered nurse-led care management intervention for disabled Medicaid beneficiaries with high health care costs. DATA SOURCES/STUDY SETTING: Washington State Department of Social and Health Services Client Outcomes Database, 2008-2011. STUDY DESIGN: In a randomized controlled trial with intent-to-treat analysis, outcomes were compared for the intervention (n = 557) and control groups (n = 563). A quasi-experimental subanalysis compared outcomes for program participants (n = 251) and propensity score-matched controls (n = 251). DATA COLLECTION/EXTRACTION METHODS: Administrative data were linked to describe costs and use of health services, criminal activity, homelessness, and death. PRINCIPAL FINDINGS: In the intent-to-treat analysis, the intervention group had higher odds of outpatient mental health service use and higher prescription drug costs than controls in the postperiod. In the subanalysis, participants had fewer unplanned hospital admissions and lower associated costs; higher prescription drug costs; higher odds of long-term care service use; higher drug/alcohol treatment costs; and lower odds of homelessness. CONCLUSIONS: We found no health care cost savings for disabled Medicaid beneficiaries randomized to intensive care management. Among participants, care management may have the potential to increase access to needed care, slow growth in the number and therefore cost of unplanned hospitalizations, and prevent homelessness. These findings apply to start-up care management programs targeted at high-cost, high-risk Medicaid populations.

Collaborative assessment and management of suicidality (CAMS): feasibility trial for next-day appointment services.

BACKGROUND: Despite the ubiquity of suicidality in behavioral health settings, empirically supported interventions for suicidality are surprisingly rare. Given the importance of resolving suicidality and therapists' anxieties about treating suicidal patients, there is a clear need for innovative services and clinical approaches. The purpose of the current study was an attempt to address some of these needs by examining the feasibility and use of a new intervention called the "Collaborative Assessment and Management of Suicidality" (CAMS) within a "Next-Day Appointment" (NDA) outpatient treatment setting. METHODS: As part of a larger feasibility study, n = 32 suicidal patients were randomly assigned to CAMS care versus Enhanced Care as Usual (E-CAU) in an outpatient crisis intervention setting attached to a safety net hospital. Intent to treat suicidal patients were seen and assessed before, during, and after treatment (with follow-up assessments conducted at 2, 4, 6, and 12 months). RESULTS: The feasibility of using CAMS in the NDA setting was clear; both groups appeared to initially benefit from their respective treatments in terms of decreased suicidal ideation and overall symptom distress. Although patients rated both treatments favorably, the CAMS group had significantly higher satisfaction and better treatment retention than E-CAU. At 12 months post-treatment, CAMS patients showed significantly better and sustained reductions in suicidal ideation, overall symptom distress, and increased hope in comparison to E-CAU patients. CONCLUSIONS: CAMS was both feasible in this NDA setting and effective in treating suicidal ideation, distress, and hopelessness (particularly at 12 months followup).

Racial/ethnic disparities in therapist effectiveness: a conceptualization and initial study of cultural competence.

As a result of mental health disparities between White and racial/ethnic minority clients, researchers have argued that some therapists may be generally competent to provide effective services but lack cultural competence. This distinction assumes that client racial/ethnic background is a source of variability in therapist effectiveness. However, there have been no direct tests of the therapist as a source of health disparities. We provided an initial test of the distinction between general and cultural competence by examining client racial/ethnic background as a source of variability in therapist effectiveness. We analyzed cannabis use outcomes from a psychotherapy trial (N = 582) for adolescent cannabis abuse and dependence using Bayesian multilevel models for count outcomes. We first tested whether therapists differed in their effectiveness and then tested whether disparities in treatment outcomes varied across therapist caseloads. Results suggested that therapists differed in their effectiveness in general and that effectiveness varied according to client racial/ethnic background. Therapist effectiveness may depend partially on client racial/ethnic minority background, providing evidence that it is valid to distinguish between general and cultural competence.

Health care and public service use and costs before and after provision of housing for chronically homeless persons with severe alcohol problems.

CONTEXT: Chronically homeless individuals with severe alcohol problems often have multiple medical and psychiatric problems and use costly health and criminal justice services at high rates. OBJECTIVE: To evaluate association of a "Housing First" intervention for chronically homeless individuals with severe alcohol problems with health care use and costs. DESIGN, SETTING, AND PARTICIPANTS: Quasi-experimental design comparing 95 housed participants (with drinking permitted) with 39 wait-list control participants enrolled between November 2005 and March 2007 in Seattle, Washington. MAIN OUTCOME MEASURES: Use and cost of services (jail bookings, days incarcerated, shelter and sobering center use, hospital-based medical services, publicly funded alcohol and drug detoxification and treatment, emergency medical services, and Medicaid-funded services) for Housing First participants relative to wait-list controls. RESULTS: Housing First participants had total costs of $8,175,922 in the year prior to the study, or median costs of $4066 per person per month (interquartile range [IQR], $2067-$8264). Median monthly costs decreased to $1492 (IQR, $337-$5709) and $958 (IQR, $98-$3200) after 6 and 12 months in housing, respectively. Poisson generalized estimating equation regressions using propensity score adjustments showed total cost rate reduction of 53% for housed participants relative to wait-list controls (rate ratio, 0.47; 95% confidence interval, 0.25-0.88) over the first 6 months. Total cost offsets for Housing First participants relative to controls averaged $2449 per person per month after accounting for housing program costs. CONCLUSIONS: In this population of chronically homeless individuals with high service use and costs, a Housing First program was associated with a relative decrease in costs after 6 months. These benefits increased to the extent that participants were retained in housing longer.

Demand-withdraw communication in severely distressed, moderately distressed, and nondistressed couples: rigidity and polarity during relationship and personal problem discussions.

This study investigated demand-withdraw communication among 68 severely distressed couples seeking therapy, 66 moderately distressed couples seeking therapy, and 48 nondistressed couples. Self-report and videotaped discussions replicated previous research, demonstrating that greater demand-withdraw during relationship problem discussions was associated with greater distress and that overall, wife-demand/husband-withdraw was greater than husband-demand/wife-withdraw. Results extended the conflict structure view of demand-withdraw by indicating that this gender polarity in demand-withdraw roles varied in strength and direction depending on who chose the topic for discussion, distress level, and marriage length. Further, in videotaped personal problem discussions, typical gender patterns of demand-withdraw were reversed. Across the relationship and personal problem discussions, a pattern of gender polarity emerged when husbands held the burden of changing.