A framework for language technologies in behavioral research and clinical applications: Ethical challenges, implications, and solutions.

Technological advances in the assessment and understanding of speech and language within the domains of automatic speech recognition, natural language processing, and machine learning present a remarkable opportunity for psychologists to learn more about human thought and communication, evaluate a variety of clinical conditions, and predict cognitive and psychological states. These innovations can be leveraged to automate traditionally time-intensive assessment tasks (e.g., educational assessment), provide psychological information and care (e.g., chatbots), and when delivered remotely (e.g., by mobile phone or wearable sensors) promise underserved communities greater access to health care. Indeed, the automatic analysis of speech provides a wealth of information that can be used for patient care in a wide range of settings (e.g., mHealth applications) and for diverse purposes (e.g., behavioral and clinical research, medical tools that are implemented into practice) and patient types (e.g., numerous psychological disorders and in psychiatry and neurology). However, automation of speech analysis is a complex task that requires the integration of several different technologies within a large distributed process with numerous stakeholders. Many organizations have raised awareness about the need for robust systems for ensuring transparency, oversight, and regulation of technologies utilizing artificial intelligence. Since there is limited knowledge about the ethical and legal implications of these applications in psychological science, we provide a balanced view of both the optimism that is widely published on and also the challenges and risks of use, including discrimination and exacerbation of structural inequalities. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Frequency and predictors of virtual hope box use in individuals experiencing suicidal ideation: An ecological momentary assessment investigation.

INTRODUCTION: The Virtual Hope Box (VHB) is a smartphone application designed to support emotion regulation when one is distressed, in a crisis, or experiencing suicidal ideation (SI). Initial proof of concept studies indicate that individuals are more likely to use the VHB than traditional hope boxes, and find it both easy to setup and helpful. To our knowledge, no studies have harnessed ambulatory assessment methodology to assess VHB use as it relates to incidence of suicidal thinking. METHODS: As such, we recruited N = 50 undergraduates who endorsed SI either the past year or past 2 weeks to complete a 10-day investigation. At baseline, participants were oriented to the VHB and instructed on how to use the application. Over the next 10 days, participants responded to prompts five times per day on their personal smartphones regarding their current experiences of SI and stress as well as VHB usage. RESULTS: Results found that most participants used the VHB at least once, rated its usefulness as high, and rated their perceived likelihood of future use as high. In addition, increases in state SI severity were related to subsequent VHB use. CONCLUSION: The VHB may be a useful tool for managing crises in undergraduates experiencing suicidal thoughts.

Computerized analysis of facial expression reveals objective indices of blunted facial affect.

Blunted affect is associated with severe mental illness, particularly schizophrenia. Mechanisms of blunted affect are poorly understood, potentially due to a lack of phenomenological clarity. Here, we examine clinician rated blunted affect and computerized facial metrics derived from ambulatory video assessment using machine learning. With high predictive accuracy (80-82%), we found that head orientation, eye movement, and facets of mouth movement were associated with clinical ratings of blunted affect. Features denoting larger muscle movements were associated with social cognition (R2 = 0.37) and cognition (R2 = 0.40). Findings provide potential insights on psychological and pathophysiological contributors to blunted affect.

Using Automated Speech Processing for Repeated Measurements in a Clinical Setting of the Behavioral Variability in the Stroop Task.

The Stroop interference task is indispensable to current neuropsychological practice. Despite this, it is limited in its potential for repeated administration, its sensitivity and its demands on professionals and their clients. We evaluated a digital Stroop deployed using a smart device. Spoken responses were timed using automated speech recognition. Participants included adult nonpatients (N = 113; k = 5 sessions over 5 days) and patients with psychiatric diagnoses (N = 85; k = 3-4 sessions per week over 4 weeks). Traditional interference (difference in response time between color incongruent words vs. color neutral words; M = 0.121 s) and facilitation (neutral vs. color congruent words; M = 0.085 s) effects were robust and temporally stable over testing sessions (ICCs 0.50-0.86). The performance showed little relation to clinical symptoms for a two-week window for either nonpatients or patients but was related to self-reported concentration at the time of testing for both groups. Performance was also related to treatment outcomes in patients. The duration of response word utterances was longer in patients than in nonpatients. Measures of intra-individual variability showed promise for understanding clinical state and treatment outcome but were less temporally stable than measures based solely on average response time latency. This framework of remote assessment using speech processing technology enables the fine-grained longitudinal charting of cognition and verbal behavior. However, at present, there is a problematic lower limit to the absolute size of the effects that can be examined when using voice in such a brief 'out-of-the-laboratory condition' given the temporal resolution of the speech-to-text detection system (in this case, 10 ms). This resolution will limit the parsing of meaningful effect sizes.

Attentional biases in facial emotion processing in individuals at clinical high risk for psychosis.

Individuals at clinical high risk (CHR) for psychosis exhibit altered facial emotion processing (FEP) and poor social functioning. It is unclear whether FEP deficits result from attentional biases, and further, how these abnormalities are linked to symptomatology (e.g., negative symptoms) and highly comorbid disorders that are also tied to abnormal FEP (e.g., depression). In the present study, we employed an eye-tracking paradigm to assess attentional biases and clinical interviews to examine differences between CHR (N = 34) individuals and healthy controls (HC; N = 46), as well as how such biases relate to symptoms and functioning in CHR individuals. Although no CHR-HC differences emerged in attentional biases, within the CHR group, symptoms and functioning were related to biases. Depressive symptoms were related to some free-view attention switching biases (e.g., to and from fearful faces, r = .50). Negative symptoms were related to more slowly disengaging from happy faces (r = .44), spending less time looking at neutral faces (r = - .42), and more time looking at no face (Avolition, r = .44). In addition, global social functioning was related to processes that overlapped with both depression and negative symptoms, including time looking at no face (r = - .68) and free-view attention switching with fearful faces (r = - .40). These findings are consistent with previous research, indicating that negative symptoms play a prominent role in the CHR syndrome, with distinct mechanisms relative to depression. Furthermore, the results suggest that attentional bias indices from eye-tracking paradigms may be predictive of social functioning.

Towards a temporospatial framework for measurements of disorganization in speech using semantic vectors.

Incoherent speech in schizophrenia has long been described as the mind making "leaps" of large distances between thoughts and ideas. Such a view seems intuitive, and for almost two decades, attempts to operationalize these conceptual "leaps" in spoken word meanings have used language-based embedding spaces. An embedding space represents meaning of words as numerical vectors where a greater proximity between word vectors represents more shared meaning. However, there are limitations with word vector-based operationalizations of coherence which can limit their appeal and utility in clinical practice. First, the use of esoteric word embeddings can be conceptually hard to grasp, and this is complicated by several different operationalizations of incoherent speech. This problem can be overcome by a better visualization of methods. Second, temporal information from the act of speaking has been largely neglected since models have been built using written text, yet speech is spoken in real time. This issue can be resolved by leveraging time stamped transcripts of speech. Third, contextual information - namely the situation of where something is spoken - has often only been inferred and never explicitly modeled. Addressing this situational issue opens up new possibilities for models with increased temporal resolution and contextual relevance. In this paper, direct visualizations of semantic distances are used to enable the inspection of examples of incoherent speech. Some common operationalizations of incoherence are illustrated, and suggestions are made for how temporal and spatial contextual information can be integrated in future implementations of measures of incoherence.

Alogia and pressured speech do not fall on a continuum of speech production using objective speech technologies.

Speech production is affected in a variety of serious mental illnesses (SMI; e.g., schizophrenia, unipolar depression, bipolar disorders) and at its extremes can be observed in the gross reduction of speech (e.g., alogia) or increase of speech (e.g., pressured speech). The present study evaluated whether clinically-rated alogia and pressured speech represent antithetical constructs when analyzed using objective metrics of speech production. We examined natural speech using acoustic and natural language processing features from two archival studies using several different speaking tasks and a combined 107 patients meeting criteria for SMI. Contrary to expectations, we did not find that alogia and pressured speech presented as opposing ends of a speech production continuum. Objective speech markers were associated with clinically rated alogia but not pressured speech, and these results were consistent across speaking tasks and studies. Implications for our understanding of speech production symptoms in SMI are discussed, as well as implications for Natural Language Processing and digital phenotyping efforts more generally.

Reflections on the nature of measurement in language-based automated assessments of patients' mental state and cognitive function.

Modern advances in computational language processing methods have enabled new approaches to the measurement of mental processes. However, the field has primarily focused on model accuracy in predicting performance on a task or a diagnostic category. Instead the field should be more focused on determining which computational analyses align best with the targeted neurocognitive/psychological functions that we want to assess. In this paper we reflect on two decades of experience with the application of language-based assessment to patients' mental state and cognitive function by addressing the questions of what we are measuring, how it should be measured and why we are measuring the phenomena. We address the questions by advocating for a principled framework for aligning computational models to the constructs being assessed and the tasks being used, as well as defining how those constructs relate to patient clinical states. We further examine the assumptions that go into the computational models and the effects that model design decisions may have on the accuracy, bias and generalizability of models for assessing clinical states. Finally, we describe how this principled approach can further the goal of transitioning language-based computational assessments to part of clinical practice while gaining the trust of critical stakeholders.

Talking about Health: A Topic Analysis of Narratives from Individuals with Schizophrenia and Other Serious Mental Illnesses.

Individuals with schizophrenia have higher mortality and shorter lifespans. There are a multitude of factors which create these conditions, but one aspect is worse physical health, particularly cardiovascular and metabolic health. Many interventions to improve the health of individuals with schizophrenia have been created, but on the whole, there has been limited effectiveness in improving quality of life or lifespan. One potential new avenue for inquiry involves a more patient-centric perspective; understanding aspects of physical health most important, and potentially most amenable to change, for individuals based on their life narratives. This study used topic modeling, a type of Natural Language Processing (NLP) on unstructured speech samples from individuals (n = 366) with serious mental illness, primarily schizophrenia, in order to extract topics. Speech samples were drawn from three studies collected over a decade in two geographically distinct regions of the United States. Several health-related topics emerged, primarily centered around food, living situation, and lifestyle (e.g., routine, hobbies). The implications of these findings for how individuals with serious mental illness and schizophrenia think about their health, and what may be most effective for future health promotion policies and interventions, are discussed.

Validation of accelerometry as a digital phenotyping measure of negative symptoms in schizophrenia.

Negative symptoms are commonly assessed via clinical rating scales; however, these measures have several inherent limitations that impact validity and utility for their use in clinical trials. Objective digital phenotyping measures that overcome some of these limitations are now available. The current study evaluated the validity of accelerometry (ACL), a passive digital phenotyping method that involves collecting data on the presence, vigor, and variability of movement. Outpatients with schizophrenia (SZ: n = 50) and demographically matched healthy controls (CN: n = 70) had ACL continuously recorded from a smartphone and smartband for 6 days. Active digital phenotyping assessments, including surveys related to activity context, were also collected via 8 daily surveys throughout the 6 day period. SZ participants had lower scores on phone ACL variables reflecting vigor and variability of movement compared to CN. ACL variables demonstrated convergent validity as indicated by significant correlations with active digital phenotyping self-reports of time spent in goal-directed activities and clinical ratings of negative symptoms. The discriminant validity of ACL was demonstrated by low correlations with clinical rating scale measures of positive, disorganized, and total symptoms. Collectively, findings suggest that ACL is a valid objective measure of negative symptoms that may complement traditional approaches to assessing the construct using clinical rating scales.

Using automated syllable counting to detect missing information in speech transcripts from clinical settings.

Speech rate and quantity reflect clinical state; thus automated transcription holds potential clinical applications. We describe two datasets where recording quality and speaker characteristics affected transcription accuracy. Transcripts of low-quality recordings omitted significant portions of speech. An automated syllable counter estimated actual speech output and quantified the amount of missing information. The efficacy of this method differed by audio quality: the correlation between missing syllables and word error rate was only significant when quality was low. Automatically counting syllables could be useful to measure and flag transcription omissions in clinical contexts where speaker characteristics and recording quality are problematic.

Natural Language Processing and Psychosis: On the Need for Comprehensive Psychometric Evaluation.

BACKGROUND AND HYPOTHESIS: Despite decades of "proof of concept" findings supporting the use of Natural Language Processing (NLP) in psychosis research, clinical implementation has been slow. One obstacle reflects the lack of comprehensive psychometric evaluation of these measures. There is overwhelming evidence that criterion and content validity can be achieved for many purposes, particularly using machine learning procedures. However, there has been very little evaluation of test-retest reliability, divergent validity (sufficient to address concerns of a "generalized deficit"), and potential biases from demographics and other individual differences. STUDY DESIGN: This article highlights these concerns in development of an NLP measure for tracking clinically rated paranoia from video "selfies" recorded from smartphone devices. Patients with schizophrenia or bipolar disorder were recruited and tracked over a week-long epoch. A small NLP-based feature set from 499 language samples were modeled on clinically rated paranoia using regularized regression. STUDY RESULTS: While test-retest reliability was high, criterion, and convergent/divergent validity were only achieved when considering moderating variables, notably whether a patient was away from home, around strangers, or alone at the time of the recording. Moreover, there were systematic racial and sex biases in the model, in part, reflecting whether patients submitted videos when they were away from home, around strangers, or alone. CONCLUSIONS: Advancing NLP measures for psychosis will require deliberate consideration of test-retest reliability, divergent validity, systematic biases and the potential role of moderators. In our example, a comprehensive psychometric evaluation revealed clear strengths and weaknesses that can be systematically addressed in future research.

How do social factors relate to blunted facial affect in schizophrenia? A digital phenotyping study using ambulatory video recordings.

Clinical interviews and laboratory-based emotional induction paradigms provide consistent evidence that facial affect is blunted in many individuals with schizophrenia. Although it is clear that blunted facial affect is not a by-product of diminished emotional experience in schizophrenia, factors contributing to blunted affect remain unclear. The current study used a combination of ambulatory video recordings that were evaluated via computerized facial affect analysis and concurrently completed ecological momentary assessment surveys to assess whether blunted affect reflects insufficient reactivity to affective or contextual factors. Specifically, whether individuals with schizophrenia require more intense affective experiences to produce expression, or whether they are less reactive to social factors (i.e. being in the presence of others, social motivation). Participants included outpatients with schizophrenia (n = 33) and healthy controls (n = 31) who completed six days of study procedures. Multilevel linear models were evaluated using both Null-Hypothesis Statistical Testing and Bayesian analyses. Individuals with schizophrenia displayed comparable expression of positive and negative emotion to controls during daily life, and no evidence was found for a different intensity of experience required for expression in either group. However, social factors differentially influenced facial expression in schizophrenia compared to controls, such that individuals with schizophrenia did not modulate their expressions based on social motivation to the same extent as controls. These findings suggest that social motivation may play an important role in determining when blunting occurs.

Ambulatory audio and video recording for digital phenotyping in schizophrenia: Adherence & data usability.

Ambulatory audio and video recording provides a wealth of information which can be used for a broad range of applications, including digital phenotyping, telepsychiatry, and telepsychology. However, these technologies are in their infancy, and guidelines for their use and analysis have yet to be established. The current project used ambulatory assessment data from individuals with schizophrenia (N = 52) and controls (N = 55) over a week to assess factors influencing sufficiency and useability of video and audio data. Logistic multilevel models examined the effect of relevant variables on video provision and video quality. There was no difference by group in video provision or quality. Videos were less likely to be provided later in the study and later in the day. Video quality was lower later in the day, particularly for controls. Participants were more likely to provide videos if alone or at home than in other settings. Black participants were less likely to have analyzable video frames than White participants. These results suggest potential racial disparities in camera technologies and/or facial analysis algorithms. Implications of these findings and recommendations for future study development, such as instructions to provide to participants to optimize video quality, are discussed.

Negative schizotypy attenuates the effect of momentary stress on social dysfunction related to COVID-19 social distancing.

Schizotypy is associated with a broad range of motivational and social dysfunctions. However, there is reason to suspect that negative schizotypy may limit social dysfunction in specific contexts that typically increase isolation and loneliness. We analyzed whether positive, negative and disorganized facets of schizotypy would mitigate detrimental reactions to stress in the context of a global stressor (the COVID-19 pandemic) that has widely necessitated social and physical isolation. Responses to two measures of schizotypy were compared to ecological momentary assessments of social dysfunction for 85 undergraduates using multilevel modelling techniques. Negative schizotypy moderated the relation between momentary stress and loss of social support, such that negative schizotypy limited the magnitude of social dysfunction stemming from increases in momentary stress. This pattern was not seen for other facets of schizotypy. Implications for the treatment and measurement of schizotypy symptoms are discussed.

Computerized analysis of facial expressions in serious mental illness.

Blunted facial affect is a transdiagnostic component of Serious Mental Illness (SMI) and is associated with a host of negative outcomes. However, blunted facial affect is a poorly understood phenomenon, with no known cures or treatments. A critical step in better understanding its phenotypic expression involves clarifying which facial expressions are altered in specific ways and under what contexts. The current literature suggests that individuals with SMI show decreased positive facial expressions, but typical, or even increased negative facial expressions during laboratory tasks. While this literature has coalesced around general trends, significantly more nuance is available regarding what components facial expressions are atypical and how those components are associated with increased severity of clinical ratings. The present project leveraged computerized facial analysis to test whether clinician-rated blunted affect is driven by decreases in duration, intensity, or frequency of positive versus other facial expressions during a structured clinical interview. Stable outpatients meeting criteria for SMI (N = 59) were examined. Facial expression did not generally vary as a function of clinical diagnosis. Overall, clinically-rated blunted affect was not associated with positive expressions, but was associated with decreased surprise and increased anger, sadness, and fear expressions. Blunted affect is not a monolithic lack of expressivity, and increased precision in operationally defining it is critical for uncovering its causes and maintaining factors. Our discussion focuses on this effort, and on advancing digital phenotyping of blunted facial affect more generally.

Sharing positive events: Ecological momentary assessment of emotion regulation via social capitalization in schizotypy.

Emotional deficits are prominent in schizophrenia-spectrum psychopathology and linked with poorer outcomes. Schizotypy, an underlying personality organization that putatively confers vulnerability to developing schizophrenia, has been associated with increased negative affect, decreased positive affect, and some difficulty with emotion regulation. This study explored the role of social capitalization, the upregulation of positive emotion when positive life events are shared with others, in schizotypy. Social capitalization is relevant for schizotypy given its association with social functioning and social motivation abnormalities. Using mobile assessment methods, a sample of college students (N=73) completed daily surveys via a mobile application two times per day for seven days and made daily ratings of mood and answered questions regarding any capitalization attempt for a positive event. Results indicated that higher schizotypy and not sharing an event were independently associated with lower happiness and increased sadness and anxiety. When an event was shared, lower schizotypy and supportive/enthusiastic response perception were independently associated with increased happiness. No significant interactions were observed between schizotypy and social capitalization variables. Future research would benefit from exploring the role that other common schizotypy concomitants, e.g., social anxiety or social disconnection, play in social capitalization and the extent this is helped or hindered via mass personal technological mediums.

High Predictive Accuracy of Negative Schizotypy With Acoustic Measures.

Negative schizotypal traits potentially can be digitally phenotyped using objective vocal analysis. Prior attempts have shown mixed success in this regard, potentially because acoustic analysis has relied on small, constrained feature sets. We employed machine learning to (a) optimize and cross-validate predictive models of self-reported negative schizotypy using a large acoustic feature set, (b) evaluate model performance as a function of sex and speaking task, (c) understand potential mechanisms underlying negative schizotypal traits by evaluating the key acoustic features within these models, and (d) examine model performance in its convergence with clinical symptoms and cognitive functioning. Accuracy was good (> 80%) and was improved by considering speaking task and sex. However, the features identified as most predictive of negative schizotypal traits were generally not considered critical to their conceptual definitions. Implications for validating and implementing digital phenotyping to understand and quantify negative schizotypy are discussed.

Alterations in facial expressions of emotion: Determining the promise of ultrathin slicing approaches and comparing human and automated coding methods in psychosis risk.

Alterations in facial expressions of emotion are a hallmark of psychopathology and may be present before the onset of mental illness. Technological advances have spurred interest in examining alterations based on "thin slices" of behavior using automated approaches. However, questions remain. First, can alterations be detected in ultrathin slices of behavior? Second, how do automated approaches converge with human coding techniques? The present study examined ultrathin (i.e., 1-min) slices of video-recorded clinical interviews of 42 individuals at clinical high risk (CHR) for psychosis and 42 matched controls. Facial expressions of emotion (e.g., joy, anger) were examined using two automated facial analysis programs and coded by trained human raters (using the Expressive Emotional Behavior Coding System). Results showed that ultrathin (i.e., 1-min) slices of behavior were sufficient to reveal alterations in facial expressions of emotion, specifically blunted joy expressions in individuals at CHR (with supplementary analyses probing links with attenuated positive symptoms and functioning). Furthermore, both automated analysis programs converged in the ability to detect blunted joy expressions and were consistent with human coding at the level of both second-by-second and aggregate data. Finally, there were areas of divergence across approaches for other emotional expressions beyond joy. These data suggest that ultrathin slices of behavior can yield clues about emotional dysfunction. Further, automated approaches (which do not require lengthy training and coder time but do lend well to mobile assessment and computational modeling) show promise, but careful evaluation of convergence with human coding is needed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).