Threat alert: The effect of outliers on the alliance-outcome correlation.

Meta-analyses have established the alliance as the most robust predictor of outcome in psychotherapy. A growing number of studies have evaluated potential threats to the conclusion that alliance is a causal factor in psychotherapy. One potential threat that has not been systematically examined is the possibility that the alliance-outcome association is driven by low alliance outliers. We examined the influence of removing low alliance outliers on the alliance-outcome association using data drawn from two large-scale, naturalistic psychotherapy data sets (Ns = 1,052; 11,029). These data sets differed in setting (university counseling center, community mental health center), country (United States and Canada), alliance measure (four-item Working Alliance Inventory Short Form Revised, 10-item Session Rating Scale), and outcome measure (Counseling Center Assessment of Psychological Symptoms-34, Outcome Questionnaire-45). We examined the impact of treating outliers in five different ways: retaining them, removing values three or two standard deviations from the mean, and winsorizing values three or two standard deviations from the mean. We also examined the effect of outliers after disaggregating alliance ratings into within-therapist and between-therapist components. The alliance-outcome correlation and the proportion of variance in posttest outcomes explained by alliance when controlling for pretest outcomes were similar regardless of how low alliance outliers were treated (change in r ≤ .04, change in R² ≤ 1%). Results from the disaggregation were similar. Thus, it appears that the alliance-outcome association is not an artifact of the influence of low alliance outliers. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Automated evaluation of psychotherapy skills using speech and language technologies.

With the growing prevalence of psychological interventions, it is vital to have measures which rate the effectiveness of psychological care to assist in training, supervision, and quality assurance of services. Traditionally, quality assessment is addressed by human raters who evaluate recorded sessions along specific dimensions, often codified through constructs relevant to the approach and domain. This is, however, a cost-prohibitive and time-consuming method that leads to poor feasibility and limited use in real-world settings. To facilitate this process, we have developed an automated competency rating tool able to process the raw recorded audio of a session, analyzing who spoke when, what they said, and how the health professional used language to provide therapy. Focusing on a use case of a specific type of psychotherapy called "motivational interviewing", our system gives comprehensive feedback to the therapist, including information about the dynamics of the session (e.g., therapist's vs. client's talking time), low-level psychological language descriptors (e.g., type of questions asked), as well as other high-level behavioral constructs (e.g., the extent to which the therapist understands the clients' perspective). We describe our platform and its performance using a dataset of more than 5000 recordings drawn from its deployment in a real-world clinical setting used to assist training of new therapists. Widespread use of automated psychotherapy rating tools may augment experts' capabilities by providing an avenue for more effective training and skill improvement, eventually leading to more positive clinical outcomes.

Technology-enhanced human interaction in psychotherapy.

Psychotherapy is on the verge of a technology-inspired revolution. The concurrent maturation of communication, signal processing, and machine learning technologies begs an earnest look at how these technologies may be used to improve the quality of psychotherapy. Here, we discuss 3 research domains where technology is likely to have a significant impact: (1) mechanism and process, (2) training and feedback, and (3) technology-mediated treatment modalities. For each domain, we describe current and forthcoming examples of how new technologies may change established applications. Moreover, for each domain we present research questions that touch on theoretical, systemic, and implementation issues. Ultimately, psychotherapy is a decidedly human endeavor, and thus the application of modern technology to therapy must capitalize on-and enhance-our human capacities as counselors, students, and supervisors. (PsycINFO Database Record

Effectiveness of telemental health during the COVID-19 pandemic: A propensity score noninferiority analysis of outcomes.

The COVID-19 pandemic forced governments to implement a range of public health measures that disrupted the personal and professional lives of many, including an abrupt adoption of telemental health services. Using data from a nonprofit counseling practice, we tested whether telemental health services delivered during the pandemic were inferior to face-to-face services delivered prior to the pandemic. We first characterized patients seeking therapy services before and during the pandemic to ascertain whether the demographics and presenting concerns of patients pre- and during COVID-19 differed and found that pandemic patients reported greater anxiety, greater overall distress, were more likely female and not partnered, and earned less than before the pandemic. We used a propensity score matching analysis to account for these differences and investigated whether or not telemental health therapy was inferior to face-to-face therapy. Based on the propensity-matched samples (2,180 patients in each condition), telemental health services were found not to be inferior to in-person services, allaying concerns about the effectiveness of telemental health services delivered during the COVID-19 pandemic. The present study also illustrates the usefulness of propensity matching for examining treatment effects in naturalistic settings. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The Structure of Competence: Evaluating the Factor Structure of the Cognitive Therapy Rating Scale.

The Cognitive Therapy Rating Scale (CTRS) is an observer-rated measure of cognitive behavioral therapy (CBT) treatment fidelity. Although widely used, the factor structure and psychometric properties of the CTRS are not well established. Evaluating the factorial validity of the CTRS may increase its utility for training and fidelity monitoring in clinical practice and research. The current study used multilevel exploratory factor analysis to examine the factor structure of the CTRS in a large sample of therapists (n = 413) and observations (n = 1,264) from community-based CBT training. Examination of model fit and factor loadings suggested that three within-therapist factors and one between-therapist factor provided adequate fit and the most parsimonious and interpretable factor structure. The three within-therapist factors included items related to (a) session structure, (b) CBT-specific skills and techniques, and (c) therapeutic relationship skills, although three items showed some evidence of cross-loading. All items showed moderate to high loadings on the single between-therapist factor. Results support continued use of the CTRS and suggest factors that may be a relevant focus for therapists, trainers, and researchers.

Rating motivational interviewing fidelity from thin slices.

Monitoring fidelity to psychosocial treatments is critical to dissemination, process and outcome research, and internal validity in efficacy trials. However, the costs required to behavior code fidelity to treatments like motivational interviewing (MI) over many therapists and sessions quickly become intractable. Coding less of a session accelerates the process, but it is not clear how much of a session must be evaluated to capture the fidelity of the entire session. The present study used a "thin slice" (Ambady & Rosenthal, 1992) paradigm to explore the degree to which variously sized thin slices of MI fidelity related to fidelity ratings for a full session. We randomly selected contiguous and noncontiguous segments of MI sessions at each whole percent of sessions (i.e., a slice consisting of 1% of session utterances, another at 2%, etc.). We then computed MI fidelity scores from these segments and calculated agreement with fidelity ratings obtained from the full session. We compared thin slice agreement with full sessions against interrater agreement and found that approximately a third of a session (9 min, 26 seconds in our sample) had sufficient agreement to approach interrater levels. These results provide a reference for researchers and clinicians to make efficient and informed use of their behavior coding resources. In addition, our results add to the behavior slicing literature, indicating that small therapist behavior samples adequately describe overall session behavior. (PsycINFO Database Record

"It's hard to argue with a computer:" Investigating Psychotherapists' Attitudes towards Automated Evaluation.

We present CORE-MI, an automated evaluation and assessment system that provides feedback to mental health counselors on the quality of their care. CORE-MI is the first system of its kind for psychotherapy, and an early example of applied machine-learning in a human service context. In this paper, we describe the CORE-MI system and report on a qualitative evaluation with 21 counselors and trainees. We discuss the applicability of CORE-MI to clinical practice and explore user perceptions of surveillance, workplace misuse, and notions of objectivity, and system reliability that may apply to automated evaluation systems generally.