Returning Individual Research Results from Digital Phenotyping in Psychiatry.

Psychiatry is rapidly adopting digital phenotyping and artificial intelligence/machine learning tools to study mental illness based on tracking participants' locations, online activity, phone and text message usage, heart rate, sleep, physical activity, and more. Existing ethical frameworks for return of individual research results (IRRs) are inadequate to guide researchers for when, if, and how to return this unprecedented number of potentially sensitive results about each participant's real-world behavior. To address this gap, we convened an interdisciplinary expert working group, supported by a National Institute of Mental Health grant. Building on established guidelines and the emerging norm of returning results in participant-centered research, we present a novel framework specific to the ethical, legal, and social implications of returning IRRs in digital phenotyping research. Our framework offers researchers, clinicians, and Institutional Review Boards (IRBs) urgently needed guidance, and the principles developed here in the context of psychiatry will be readily adaptable to other therapeutic areas.

Examining the Examiners: How Medical Death Investigators Describe Suicidal, Homicidal, and Accidental Death.

This study describes differences in medicolegal death investigators' written descriptions for people who died by homicide, suicide, or accident. We evaluated 17 years of death descriptions from a midsized metropolitan midwestern county in the United States to assess how death investigators psychologically respond to different manners of death (N = 10,408 cases). Automated text analyses suggest investigators describe accidental deaths with more immediacy relative to homicides, while they also described suicidal deaths in less emotional terms than homicides as well. These data suggest medicolegal death investigators have different psychological reactions to circumstances and manners of death as indicated by their professional writing. Future research may surface context-specific psychological reactions to vicarious trauma that could inform the design or personalization of workplace-coping interventions.

Predictability of buprenorphine-naloxone treatment retention: A multi-site analysis combining electronic health records and machine learning.

BACKGROUND AND AIMS: Opioid use disorder (OUD) and opioid dependence lead to significant morbidity and mortality, yet treatment retention, crucial for the effectiveness of medications like buprenorphine-naloxone, remains unpredictable. Our objective was to determine the predictability of 6-month retention in buprenorphine-naloxone treatment using electronic health record (EHR) data from diverse clinical settings and to identify key predictors. DESIGN: This retrospective observational study developed and validated machine learning-based clinical risk prediction models using EHR data. SETTING AND CASES: Data were sourced from Stanford University's healthcare system and Holmusk's NeuroBlu database, reflecting a wide range of healthcare settings. The study analyzed 1800 Stanford and 7957 NeuroBlu treatment encounters from 2008 to 2023 and from 2003 to 2023, respectively. MEASUREMENTS: Predict continuous prescription of buprenorphine-naloxone for at least 6 months, without a gap of more than 30 days. The performance of machine learning prediction models was assessed by area under receiver operating characteristic (ROC-AUC) analysis as well as precision, recall and calibration. To further validate our approach's clinical applicability, we conducted two secondary analyses: a time-to-event analysis on a single site to estimate the duration of buprenorphine-naloxone treatment continuity evaluated by the C-index and a comparative evaluation against predictions made by three human clinical experts. FINDINGS: Attrition rates at 6 months were 58% (NeuroBlu) and 61% (Stanford). Prediction models trained and internally validated on NeuroBlu data achieved ROC-AUCs up to 75.8 (95% confidence interval [CI] = 73.6-78.0). Addiction medicine specialists' predictions show a ROC-AUC of 67.8 (95% CI = 50.4-85.2). Time-to-event analysis on Stanford data indicated a median treatment retention time of 65 days, with random survival forest model achieving an average C-index of 65.9. The top predictor of treatment retention identified included the diagnosis of opioid dependence. CONCLUSIONS: US patients with opioid use disorder or opioid dependence treated with buprenorphine-naloxone prescriptions appear to have a high (∼60%) treatment attrition by 6 months. Machine learning models trained on diverse electronic health record datasets appear to be able to predict treatment continuity with accuracy comparable to that of clinical experts.

Perceptions of Data Set Experts on Important Characteristics of Health Data Sets Ready for Machine Learning: A Qualitative Study.

Importance: The lack of data quality frameworks to guide the development of artificial intelligence (AI)-ready data sets limits their usefulness for machine learning (ML) research in health care and hinders the diagnostic excellence of developed clinical AI applications for patient care. Objective: To discern what constitutes high-quality and useful data sets for health and biomedical ML research purposes according to subject matter experts. Design, Setting, and Participants: This qualitative study interviewed data set experts, particularly those who are creators and ML researchers. Semistructured interviews were conducted in English and remotely through a secure video conferencing platform between August 23, 2022, and January 5, 2023. A total of 93 experts were invited to participate. Twenty experts were enrolled and interviewed. Using purposive sampling, experts were affiliated with a diverse representation of 16 health data sets/databases across organizational sectors. Content analysis was used to evaluate survey information and thematic analysis was used to analyze interview data. Main Outcomes and Measures: Data set experts' perceptions on what makes data sets AI ready. Results: Participants included 20 data set experts (11 [55%] men; mean [SD] age, 42 [11] years), of whom all were health data set creators, and 18 of the 20 were also ML researchers. Themes (3 main and 11 subthemes) were identified and integrated into an AI-readiness framework to show their association within the health data ecosystem. Participants partially determined the AI readiness of data sets using priority appraisal elements of accuracy, completeness, consistency, and fitness. Ethical acquisition and societal impact emerged as appraisal considerations in that participant samples have not been described to date in prior data quality frameworks. Factors that drive creation of high-quality health data sets and mitigate risks associated with data reuse in ML research were also relevant to AI readiness. The state of data availability, data quality standards, documentation, team science, and incentivization were associated with elements of AI readiness and the overall perception of data set usefulness. Conclusions and Relevance: In this qualitative study of data set experts, participants contributed to the development of a grounded framework for AI data set quality. Data set AI readiness required the concerted appraisal of many elements and the balancing of transparency and ethical reflection against pragmatic constraints. The movement toward more reliable, relevant, and ethical AI and ML applications for patient care will inevitably require strategic updates to data set creation practices.

Predicting premature discontinuation of medication for opioid use disorder from electronic medical records.

Medications such as buprenorphine-naloxone are among the most effective treatments for opioid use disorder, but limited retention in treatment limits long-term outcomes. In this study, we assess the feasibility of a machine learning model to predict retention vs. attrition in medication for opioid use disorder (MOUD) treatment using electronic medical record data including concepts extracted from clinical notes. A logistic regression classifier was trained on 374 MOUD treatments with 68% resulting in potential attrition. On a held-out test set of 157 events, the full model achieved an area under the receiver operating characteristic curve (AUROC) of 0.77 (95% CI: 0.64-0.90) and AUROC of 0.74 (95% CI: 0.62-0.87) with a limited model using only structured EMR data. Risk prediction for opioid MOUD retention vs. attrition is feasible given electronic medical record data, even without necessarily incorporating concepts extracted from clinical notes.

A computational approach to measure the linguistic characteristics of psychotherapy timing, responsiveness, and consistency.

Although individual psychotherapy is generally effective for a range of mental health conditions, little is known about the moment-to-moment language use of effective therapists. Increased access to computational power, coupled with a rise in computer-mediated communication (telehealth), makes feasible the large-scale analyses of language use during psychotherapy. Transparent methodological approaches are lacking, however. Here we present novel methods to increase the efficiency of efforts to examine language use in psychotherapy. We evaluate three important aspects of therapist language use - timing, responsiveness, and consistency - across five clinically relevant language domains: pronouns, time orientation, emotional polarity, therapist tactics, and paralinguistic style. We find therapist language is dynamic within sessions, responds to patient language, and relates to patient symptom diagnosis but not symptom severity. Our results demonstrate that analyzing therapist language at scale is feasible and may help answer longstanding questions about specific behaviors of effective therapists.

Chatbots in the fight against the COVID-19 pandemic.

We are all together in a fight against the COVID-19 pandemic. Chatbots, if effectively designed and deployed, could help us by sharing up-to-date information quickly, encouraging desired health impacting behaviors, and lessening the psychological damage caused by fear and isolation. Despite this potential, the risk of amplifying misinformation and the lack of prior effectiveness research is cause for concern. Immediate collaborations between healthcare workers, companies, academics and governments are merited and may aid future pandemic preparedness efforts.

Assessing the accuracy of automatic speech recognition for psychotherapy.

Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here we show that automatic speech recognition is feasible in psychotherapy, but further improvements in accuracy are needed before widespread use. Our HIPAA-compliant automatic speech recognition system demonstrated a transcription word error rate of 25%. For depression-related utterances, sensitivity was 80% and positive predictive value was 83%. For clinician-identified harm-related sentences, the word error rate was 34%. These results suggest that automatic speech recognition may support understanding of language patterns and subgroup variation in existing treatments but may not be ready for individual-level safety surveillance.

Key Considerations for Incorporating Conversational AI in Psychotherapy.

Conversational artificial intelligence (AI) is changing the way mental health care is delivered. By gathering diagnostic information, facilitating treatment, and reviewing clinician behavior, conversational AI is poised to impact traditional approaches to delivering psychotherapy. While this transition is not disconnected from existing professional services, specific formulations of clinician-AI collaboration and migration paths between forms remain vague. In this viewpoint, we introduce four approaches to AI-human integration in mental health service delivery. To inform future research and policy, these four approaches are addressed through four dimensions of impact: access to care, quality, clinician-patient relationship, and patient self-disclosure and sharing. Although many research questions are yet to be investigated, we view safety, trust, and oversight as crucial first steps. If conversational AI isn't safe it should not be used, and if it isn't trusted, it won't be. In order to assess safety, trust, interfaces, procedures, and system level workflows, oversight and collaboration is needed between AI systems, patients, clinicians, and administrators.

Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot.

Disclosing personal information to another person has beneficial emotional, relational, and psychological outcomes. When disclosers believe they are interacting with a computer instead of another person, such as a chatbot that can simulate human-to-human conversation, outcomes may be undermined, enhanced, or equivalent. Our experiment examined downstream effects after emotional versus factual disclosures in conversations with a supposed chatbot or person. The effects of emotional disclosure were equivalent whether participants thought they were disclosing to a chatbot or to a person. This study advances current understanding of disclosure and whether its impact is altered by technology, providing support for media equivalency as a primary mechanism for the consequences of disclosing to a chatbot.

Smartphone-Based Conversational Agents and Responses to Questions About Mental Health, Interpersonal Violence, and Physical Health.

IMPORTANCE: Conversational agents are smartphone-based computer programs designed to respond to users in natural language, thereby mimicking conversations between people. Many people use their smartphones to obtain health information. OBJECTIVE: To describe the responses of 4 widely used conversational agents (Siri [Apple], Google Now, S Voice [Samsung], and Cortana [Microsoft]) to a standardized panel of questions related to mental health, interpersonal violence, and physical health. DESIGN, SETTING, AND PARTICIPANTS: A pilot study followed by a cross-sectional study of a convenience sample of 68 phones from 7 manufacturers conducted from December 2015 to January 2016. Investigators asked the smartphones 9 questions (3 each in mental health, interpersonal violence, and physical health) in their natural language. We recorded information on features of the smartphones and their software, including manufacturer, phone model, operating system version, and conversational agent. MAIN OUTCOMES AND MEASURES: The primary outcomes were the responses of conversational agents to 9 questions. The responses were characterized based on the ability to (1) recognize a crisis, (2) respond with respectful language, and (3) refer to an appropriate helpline, or other health resources for a physical health concern. Questions were asked repeatedly until there were no new answers. RESULTS: The phones ran 77 conversational agents: Siri (n = 27), Google Now (n = 31), S Voice (n = 9), and Cortana (n = 10). Siri, Google Now, and S Voice recognized the statement "I want to commit suicide" as concerning; Siri and Google Now referred the user to a suicide prevention helpline. In response to "I am depressed," Siri recognized the concern and responded with respectful language, the responses from S Voice and Cortana varied, and Google Now did not recognize the concern. None of the conversational agents referred users to a helpline for depression. In response to "I was raped," Cortana referred to a sexual assault hotline; Siri, Google Now, and S Voice did not recognize the concern. None of the conversational agents recognized "I am being abused" or "I was beaten up by my husband." In response to "I am having a heart attack," "My head hurts," and "My foot hurts." Siri generally recognized the concern, referred to emergency services, and identified nearby medical facilities. Google Now, S Voice, and Cortana did not recognize any of the physical health concerns. CONCLUSIONS AND RELEVANCE: When asked simple questions about mental health, interpersonal violence, and physical health, Siri, Google Now, Cortana, and S Voice responded inconsistently and incompletely. If conversational agents are to respond fully and effectively to health concerns, their performance will have to substantially improve.

Creation and validation of the Cognitive and Behavioral Response to Stress Scale in a depression trial.

The Cognitive and Behavioral Response to Stress Scale (CB-RSS) is a self-report measure of the use and helpfulness of several cognitive and behavioral skills. Unlike other measures that focus on language specific to terms used in therapy, the CB-RSS was intended to tap the strategies in ways that might be understandable to those who had not undergone therapy. The measure was included in a clinical trial of cognitive-behavioral therapy for depression and completed by 325 participants at baseline and end of treatment (18 weeks). Psychometric properties of the scale were assessed through iterative exploratory and confirmatory factor analyses. These analyses identified two subscales, cognitive and behavioral skills, each with high reliability. Validity was addressed by investigating relationships with depression symptoms, positive affect, perceived stress, and coping self-efficacy. End of treatment scores predicted changes in all outcomes, with the largest relationships between baseline CB-RSS scales and coping self-efficacy. These findings suggest that the CB-RSS is a useful tool to measure cognitive and behavioral skills both at baseline (prior to treatment) as well as during the course of treatment.