Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data.

AI tools intend to transform mental healthcare by providing remote estimates of depression risk using behavioral data collected by sensors embedded in smartphones. While these tools accurately predict elevated symptoms in small, homogenous populations, recent studies show that these tools are less accurate in larger, more diverse populations. In this work, we show that accuracy is reduced because sensed-behaviors are unreliable predictors of depression across individuals; specifically the sensed-behaviors that predict depression risk are inconsistent across demographic and socioeconomic subgroups. We first identified subgroups where a developed AI tool underperformed by measuring algorithmic bias, where subgroups with depression were incorrectly predicted to be at lower risk than healthier subgroups. We then found inconsistencies between sensed-behaviors predictive of depression across these subgroups. Our findings suggest that researchers developing AI tools predicting mental health from behavior should think critically about the generalizability of these tools, and consider tailored solutions for targeted populations.

Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data.

AI tools intend to transform mental healthcare by providing remote estimates of depression risk using behavioral data collected by sensors embedded in smartphones. While these tools accurately predict elevated depression symptoms in small, homogenous populations, recent studies show that these tools are less accurate in larger, more diverse populations. In this work, we show that accuracy is reduced because sensed-behaviors are unreliable predictors of depression across individuals: sensed-behaviors that predict depression risk are inconsistent across demographic and socioeconomic subgroups. We first identified subgroups where a developed AI tool underperformed by measuring algorithmic bias, where subgroups with depression were incorrectly predicted to be at lower risk than healthier subgroups. We then found inconsistencies between sensed-behaviors predictive of depression across these subgroups. Our findings suggest that researchers developing AI tools predicting mental health from sensed-behaviors should think critically about the generalizability of these tools, and consider tailored solutions for targeted populations.

Understanding Mental Health Clinicians' Perceptions and Concerns Regarding Using Passive Patient-Generated Health Data for Clinical Decision-Making: Qualitative Semistructured Interview Study.

BACKGROUND: Digital health-tracking tools are changing mental health care by giving patients the ability to collect passively measured patient-generated health data (PGHD; ie, data collected from connected devices with little to no patient effort). Although there are existing clinical guidelines for how mental health clinicians should use more traditional, active forms of PGHD for clinical decision-making, there is less clarity on how passive PGHD can be used. OBJECTIVE: We conducted a qualitative study to understand mental health clinicians' perceptions and concerns regarding the use of technology-enabled, passively collected PGHD for clinical decision-making. Our interviews sought to understand participants' current experiences with and visions for using passive PGHD. METHODS: Mental health clinicians providing outpatient services were recruited to participate in semistructured interviews. Interview recordings were deidentified, transcribed, and qualitatively coded to identify overarching themes. RESULTS: Overall, 12 mental health clinicians (n=11, 92% psychiatrists and n=1, 8% clinical psychologist) were interviewed. We identified 4 overarching themes. First, passive PGHD are patient driven-we found that current passive PGHD use was patient driven, not clinician driven; participating clinicians only considered passive PGHD for clinical decision-making when patients brought passive data to clinical encounters. The second theme was active versus passive data as subjective versus objective data-participants viewed the contrast between active and passive PGHD as a contrast between interpretive data on patients' mental health and objective information on behavior. Participants believed that prioritizing passive over self-reported, active PGHD would reduce opportunities for patients to reflect upon their mental health, reducing treatment engagement and raising questions about how passive data can best complement active data for clinical decision-making. Third, passive PGHD must be delivered at appropriate times for action-participants were concerned with the real-time nature of passive PGHD; they believed that it would be infeasible to use passive PGHD for real-time patient monitoring outside clinical encounters and more feasible to use passive PGHD during clinical encounters when clinicians can make treatment decisions. The fourth theme was protecting patient privacy-participating clinicians wanted to protect patient privacy within passive PGHD-sharing programs and discussed opportunities to refine data sharing consent to improve transparency surrounding passive PGHD collection and use. CONCLUSIONS: Although passive PGHD has the potential to enable more contextualized measurement, this study highlights the need for building and disseminating an evidence base describing how and when passive measures should be used for clinical decision-making. This evidence base should clarify how to use passive data alongside more traditional forms of active PGHD, when clinicians should view passive PGHD to make treatment decisions, and how to protect patient privacy within passive data-sharing programs. Clear evidence would more effectively support the uptake and effective use of these novel tools for both patients and their clinicians.

Corrigendum: Personalized digital intervention for depression based on social rhythm principles adds significantly to outpatient treatment.

[This corrects the article DOI: 10.3389/fdgth.2022.870522.].

Personalized digital intervention for depression based on social rhythm principles adds significantly to outpatient treatment.

We conducted a 16-week randomized controlled trial in psychiatric outpatients with a lifetime diagnosis of a mood and/or anxiety disorder to measure the impact of a first-of-its-kind precision digital intervention software solution based on social rhythm regulation principles. The full intent-to-treat (ITT) sample consisted of 133 individuals, aged 18-65. An exploratory sub-sample of interest was those individuals who presented with moderately severe to severe depression at study entry (baseline PHQ-8 score ≥15; N = 28). Cue is a novel digital intervention platform that capitalizes on the smartphone's ability to continuously monitor depression-relevant behavior patterns and use each patient's behavioral data to provide timely, personalized "micro-interventions," making this the first example of a precision digital intervention of which we are aware. Participants were randomly allocated to receive Cue plus care-as-usual or digital monitoring only plus care as usual. Within the full study and depressed-at-entry samples, we fit a mixed effects model to test for group differences in the slope of depressive symptoms over 16 weeks. To account for the non-linear trajectory with more flexibility, we also fit a mixed effects model considering week as a categorical variable and used the resulting estimates to test the group difference in PHQ change from baseline to 16 weeks. In the full sample, the group difference in the slope of PHQ-8 was negligible (Cohen's d = -0.10); however, the Cue group demonstrated significantly greater improvement from baseline to 16 weeks (p = 0.040). In the depressed-at-entry sample, we found evidence for benefit of Cue. The group difference in the slope of PHQ-8 (Cohen's d = -0.72) indicated a meaningfully more rapid rate of improvement in the intervention group than in the control group. The Cue group also demonstrated significantly greater improvement in PHQ-8 from baseline to 16 weeks (p = 0.009). We are encouraged by the size of the intervention effect in those who were acutely ill at baseline, and by the finding that across all participants, 80% of whom were receiving pharmacotherapy, we observed significant benefit of Cue at 16 weeks of treatment. These findings suggest that a social rhythm-focused digital intervention platform may represent a useful and accessible adjunct to antidepressant treatment (https://clinicaltrials.gov/ct2/show/NCT03152864?term=ellen+frank&draw=2&rank=3).

Machine learning for passive mental health symptom prediction: Generalization across different longitudinal mobile sensing studies.

Mobile sensing data processed using machine learning models can passively and remotely assess mental health symptoms from the context of patients' lives. Prior work has trained models using data from single longitudinal studies, collected from demographically homogeneous populations, over short time periods, using a single data collection platform or mobile application. The generalizability of model performance across studies has not been assessed. This study presents a first analysis to understand if models trained using combined longitudinal study data to predict mental health symptoms generalize across current publicly available data. We combined data from the CrossCheck (individuals living with schizophrenia) and StudentLife (university students) studies. In addition to assessing generalizability, we explored if personalizing models to align mobile sensing data, and oversampling less-represented severe symptoms, improved model performance. Leave-one-subject-out cross-validation (LOSO-CV) results were reported. Two symptoms (sleep quality and stress) had similar question-response structures across studies and were used as outcomes to explore cross-dataset prediction. Models trained with combined data were more likely to be predictive (significant improvement over predicting training data mean) than models trained with single-study data. Expected model performance improved if the distance between training and validation feature distributions decreased using combined versus single-study data. Personalization aligned each LOSO-CV participant with training data, but only improved predicting CrossCheck stress. Oversampling significantly improved severe symptom classification sensitivity and positive predictive value, but decreased model specificity. Taken together, these results show that machine learning models trained on combined longitudinal study data may generalize across heterogeneous datasets. We encourage researchers to disseminate collected de-identified mobile sensing and mental health symptom data, and further standardize data types collected across studies to enable better assessment of model generalizability.

A call for open data to develop mental health digital biomarkers.

Digital biomarkers of mental health, created using data extracted from everyday technologies including smartphones, wearable devices, social media and computer interactions, have the opportunity to revolutionise mental health diagnosis and treatment by providing near-continuous unobtrusive and remote measures of behaviours associated with mental health symptoms. Machine learning models process data traces from these technologies to identify digital biomarkers. In this editorial, we caution clinicians against using digital biomarkers in practice until models are assessed for equitable predictions ('model equity') across demographically diverse patients at scale, behaviours over time, and data types extracted from different devices and platforms. We posit that it will be difficult for any individual clinic or large-scale study to assess and ensure model equity and alternatively call for the creation of a repository of open de-identified data for digital biomarker development.

Burnout and the Quantified Workplace: Tensions around Personal Sensing Interventions for Stress in Resident Physicians.

Recent research has explored computational tools to manage workplace stress via personal sensing, a measurement paradigm in which behavioral data streams are collected from technologies including smartphones, wearables, and personal computers. As these tools develop, they invite inquiry into how they can be appropriately implemented towards improving workers' well-being. In this study, we explored this proposition through formative interviews followed by a design provocation centered around measuring burnout in a U.S. resident physician program. Residents and their supervising attending physicians were presented with medium-fidelity mockups of a dashboard providing behavioral data on residents' sleep, activity and time working; self-reported data on residents' levels of burnout; and a free text box where residents could further contextualize their well-being. Our findings uncover tensions around how best to measure workplace well-being, who within a workplace is accountable for worker stress, and how the introduction of such tools remakes the boundaries of appropriate information flows between worker and workplace. We conclude by charting future work confronting these tensions, to ensure personal sensing is leveraged to truly improve worker well-being.

mHealth-Assisted Detection of Precursors to Relapse in Schizophrenia.

Theoretical views and a growing body of empirical evidence suggest that psychiatric relapses in schizophrenia-spectrum disorders (SSDs) have measurable warning signs. However, because they are time- and resource-intensive, existing assessment approaches are not well-suited to detect these warning signs in a timely, scalable fashion. Mobile technologies deploying frequent measurements-i.e., ecological momentary assessment-could be leveraged to detect increases in symptoms that may precede relapses. The present study examined EMA measurements with growth curve models in the 100 days preceding and following 27 relapses (among n = 20 individuals with SSDs) to identify (1) what symptoms changed in the periods gradually preceding, following, and right as relapses occur, (2) how large were these changes, and (3) on what time scale did they occur. Results demonstrated that, on average, participants reported elevations in negative mood (d = 0.34), anxiety (d =0.49), persecutory ideation (d =0.35), and hallucinations (d =0.34) on relapse days relative to their average during the study. These increases emerged gradually on average from significant and steady increases (d = 0.05 per week) in persecutory ideation and hallucinations over the 100-day period preceding relapse. This suggests that brief (i.e., 1-2 item) assessments of psychotic symptoms may detect meaningful signals that precede psychiatric relapses long before they occur. These assessments could increase opportunities for relapse prevention as remote measurement-based care management platforms develop.

Digital biomarker of mental fatigue.

Mental fatigue is an important aspect of alertness and wellbeing. Existing fatigue tests are subjective and/or time-consuming. Here, we show that smartphone-based gaze is significantly impaired with mental fatigue, and tracks the onset and progression of fatigue. A simple model predicts mental fatigue reliably using just a few minutes of gaze data. These results suggest that smartphone-based gaze could provide a scalable, digital biomarker of mental fatigue.

Banbury Forum Consensus Statement on the Path Forward for Digital Mental Health Treatment.

A major obstacle to mental health treatment for many Americans is accessibility: the United States faces a shortage of mental health providers, resulting in federally designated shortage areas. Although digital mental health treatments (DMHTs) are effective interventions for common mental disorders, they have not been widely adopted by the U.S. health care system. National and international expert stakeholders representing health care organizations, insurance companies and payers, employers, patients, researchers, policy makers, health economists, and DMHT companies and the investment community attended two Banbury Forum meetings. The Banbury Forum reviewed the evidence for DMHTs, identified the challenges to successful and sustainable implementation, investigated the factors that contributed to more successful implementation internationally, and developed the following recommendations: guided DMHTs should be offered to all patients experiencing common mental disorders, DMHT products and services should be reimbursable to support integration into the U.S. health care landscape, and an evidence standards framework should be developed to support decision makers in evaluating DMHTs.

Identifying Mobile Sensing Indicators of Stress-Resilience.

Resident physicians (residents) experiencing prolonged workplace stress are at risk of developing mental health symptoms. Creating novel, unobtrusive measures of resilience would provide an accessible approach to evaluate symptom susceptibility without the perceived stigma of formal mental health assessments. In this work, we created a system to find indicators of resilience using passive wearable sensors and smartphone-delivered ecological momentary assessment (EMA). This system identified indicators of resilience during a medical internship, the high stress first-year of a residency program. We then created density estimation approaches to predict these indicators before mental health changes occurred, and validated whether the predicted indicators were also associated with resilience. Our system identified resilience indicators associated with physical activity (step count), sleeping behavior, reduced heart rate, increased mood, and reduced mood variability. Density estimation models were able to replicate a subset of the associations between sleeping behavior, heart rate, and resilience. To the best of our knowledge, this work provides the first methodology to identify and predict indicators of resilience using passive sensing and EMA. Researchers studying resident mental health can apply this approach to design resilience-building interventions and prevent mental health symptom development.

Using Smartphone Sensor Data to Assess Inhibitory Control in the Wild: Longitudinal Study.

BACKGROUND: Inhibitory control, or inhibition, is one of the core executive functions of humans. It contributes to our attention, performance, and physical and mental well-being. Our inhibitory control is modulated by various factors and therefore fluctuates over time. Being able to continuously and unobtrusively assess our inhibitory control and understand the mediating factors may allow us to design intelligent systems that help manage our inhibitory control and ultimately our well-being. OBJECTIVE: The aim of this study is to investigate whether we can assess individuals' inhibitory control using an unobtrusive and scalable approach to identify digital markers that are predictive of changes in inhibitory control. METHODS: We developed InhibiSense, an app that passively collects the following information: users' behaviors based on their phone use and sensor data, the ground truths of their inhibition control measured with stop-signal tasks (SSTs) and ecological momentary assessments (EMAs), and heart rate information transmitted from a wearable heart rate monitor (Polar H10). We conducted a 4-week in-the-wild study, where participants were asked to install InhibiSense on their phone and wear a Polar H10. We used generalized estimating equation (GEE) and gradient boosting tree models fitted with features extracted from participants' phone use and sensor data to predict their stop-signal reaction time (SSRT), an objective metric used to measure an individual's inhibitory control, and identify the predictive digital markers. RESULTS: A total of 12 participants completed the study, and 2189 EMAs and SST responses were collected. The results from the GEE models suggest that the top digital markers positively associated with an individual's SSRT include phone use burstiness (P=.005), the mean duration between 2 consecutive phone use sessions (P=.02), the change rate of battery level when the phone was not charged (P=.04), and the frequency of incoming calls (P=.03). The top digital markers negatively associated with SSRT include the standard deviation of acceleration (P<.001), the frequency of short phone use sessions (P<.001), the mean duration of incoming calls (P<.001), the mean decibel level of ambient noise (P=.007), and the percentage of time in which the phone was connected to the internet through a mobile network (P=.001). No significant correlation between the participants' objective and subjective measurement of inhibitory control was found. CONCLUSIONS: We identified phone-based digital markers that were predictive of changes in inhibitory control and how they were positively or negatively associated with a person's inhibitory control. The results of this study corroborate the findings of previous studies, which suggest that inhibitory control can be assessed continuously and unobtrusively in the wild. We discussed some potential applications of the system and how technological interventions can be designed to help manage inhibitory control.

Assessing the relationship between routine and schizophrenia symptoms with passively sensed measures of behavioral stability.

Increased stability in one's daily routine is associated with well-being in the general population and often a goal of behavioral interventions for people with serious mental illnesses like schizophrenia. Assessing behavioral stability has been limited in clinical research by the use of retrospective scales, which are susceptible to reporting biases and memory inaccuracies. Mobile passive sensors, which are less susceptible to these sources of error, have emerged as tools to assess behavioral patterns in a range of populations. The present study developed and examined a metric of behavioral stability from data generated by a passive sensing system carried by 61 individuals with schizophrenia for one year. This metric-the Stability Index-appeared orthogonal from existing measures drawn from passive sensors and matched the predictive performance of state-of-the-art features. Specifically, greater stability in social activity (e.g., calls and messages) were associated with lower symptoms, and greater stability in physical activity (e.g., being still) appeared associated with elevated symptoms. This study provides additional support for the predictive value of individualized over population-level data in psychiatric populations. The Stability Index offers also a promising tool for generating insights about the impact of behavioral stability in schizophrenia-spectrum disorders.

Using behavioral rhythms and multi-task learning to predict fine-grained symptoms of schizophrenia.

Schizophrenia is a severe and complex psychiatric disorder with heterogeneous and dynamic multi-dimensional symptoms. Behavioral rhythms, such as sleep rhythm, are usually disrupted in people with schizophrenia. As such, behavioral rhythm sensing with smartphones and machine learning can help better understand and predict their symptoms. Our goal is to predict fine-grained symptom changes with interpretable models. We computed rhythm-based features from 61 participants with 6,132 days of data and used multi-task learning to predict their ecological momentary assessment scores for 10 different symptom items. By taking into account both the similarities and differences between different participants and symptoms, our multi-task learning models perform statistically significantly better than the models trained with single-task learning for predicting patients' individual symptom trajectories, such as feeling depressed, social, and calm and hearing voices. We also found different subtypes for each of the symptoms by applying unsupervised clustering to the feature weights in the models. Taken together, compared to the features used in the previous studies, our rhythm features not only improved models' prediction accuracy but also provided better interpretability for how patients' behavioral rhythms and the rhythms of their environments influence their symptom conditions. This will enable both the patients and clinicians to monitor how these factors affect a patient's condition and how to mitigate the influence of these factors. As such, we envision that our solution allows early detection and early intervention before a patient's condition starts deteriorating without requiring extra effort from patients and clinicians.

Predicting Early Warning Signs of Psychotic Relapse From Passive Sensing Data: An Approach Using Encoder-Decoder Neural Networks.

BACKGROUND: Schizophrenia spectrum disorders (SSDs) are chronic conditions, but the severity of symptomatic experiences and functional impairments vacillate over the course of illness. Developing unobtrusive remote monitoring systems to detect early warning signs of impending symptomatic relapses would allow clinicians to intervene before the patient's condition worsens. OBJECTIVE: In this study, we aim to create the first models, exclusively using passive sensing data from a smartphone, to predict behavioral anomalies that could indicate early warning signs of a psychotic relapse. METHODS: Data used to train and test the models were collected during the CrossCheck study. Hourly features derived from smartphone passive sensing data were extracted from 60 patients with SSDs (42 nonrelapse and 18 relapse >1 time throughout the study) and used to train models and test performance. We trained 2 types of encoder-decoder neural network models and a clustering-based local outlier factor model to predict behavioral anomalies that occurred within the 30-day period before a participant's date of relapse (the near relapse period). Models were trained to recreate participant behavior on days of relative health (DRH, outside of the near relapse period), following which a threshold to the recreation error was applied to predict anomalies. The neural network model architecture and the percentage of relapse participant data used to train all models were varied. RESULTS: A total of 20,137 days of collected data were analyzed, with 726 days of data (0.037%) within any 30-day near relapse period. The best performing model used a fully connected neural network autoencoder architecture and achieved a median sensitivity of 0.25 (IQR 0.15-1.00) and specificity of 0.88 (IQR 0.14-0.96; a median 108% increase in behavioral anomalies near relapse). We conducted a post hoc analysis using the best performing model to identify behavioral features that had a medium-to-large effect (Cohen d>0.5) in distinguishing anomalies near relapse from DRH among 4 participants who relapsed multiple times throughout the study. Qualitative validation using clinical notes collected during the original CrossCheck study showed that the identified features from our analysis were presented to clinicians during relapse events. CONCLUSIONS: Our proposed method predicted a higher rate of anomalies in patients with SSDs within the 30-day near relapse period and can be used to uncover individual-level behaviors that change before relapse. This approach will enable technologists and clinicians to build unobtrusive digital mental health tools that can predict incipient relapse in SSDs.

Capturing behavioral indicators of persecutory ideation using mobile technology.

Most existing measures of persecutory ideation (PI) rely on infrequent in-person visits, and this limits their ability to assess rapid changes or real-world functioning. Mobile health (mHealth) technology may address these limitations. Little is known about passively sensed behavioral indicators associated with PI. In the current study, sixty-two participants with schizophrenia spectrum disorders completed momentary assessments of PI on a smartphone that also passively collected behavioral data for one year. Results suggested that PI was prevalent (n = 50, 82% of sample) but had infrequent incidence (25.2% of EMA responses). PI was also associated with changes in several passively sensed variables, including decreases in distance traveled (Mkilometers = -1.20, SD = 18.88), time spent in a vehicle (Mminutes = -4.15, SD = 49.59), length of outgoing phone calls (Mminutes = -0.79, SD = 13.13), time spent proximal to human speech (Mminutes = -6.26, SD = 153.03), and an increase in time sitting still (Mminutes = 4.04, SD = 94.69). The present study suggests changes associated with PI may be detectable by passive sensors, including reductions in moving or traveling, and time spent around others or in self-initiated phone conversations. These constructs might constitute risk for PI.

Relationships between smartphone social behavior and relapse in schizophrenia: A preliminary report.

Social dysfunction is a hallmark of schizophrenia. Social isolation may increase individuals' risk for psychotic symptom exacerbation and relapse. Monitoring and timely detection of shifts in social functioning are hampered by the limitations of traditional clinic-based assessment strategies. Ubiquitous mobile technologies such as smartphones introduce new opportunities to capture objective digital indicators of social behavior. The goal of this study was to evaluate whether smartphone-collected digital measures of social behavior can provide early indication of relapse events among individuals with schizophrenia. Sixty-one individuals with schizophrenia with elevated risk for relapse were given smartphones with the CrossCheck behavioral sensing system for a year of remote monitoring. CrossCheck leveraged the device's microphone, call record, and text messaging log to capture digital socialization data. Relapse events including psychiatric hospitalizations, suicidal ideation, and significant psychiatric symptom exacerbations were recorded by trained assessors. Exploratory mixed effects models examined relationships of social behavior to relapse, finding that reductions in number and duration of outgoing calls, as well as number of text messages were associated with relapses. Number and duration of incoming phone calls and in-person conversations were not. Smartphone enabled social activity may provide an important metric in determining relapse risk in schizophrenia and provide access to sensitive, meaningful and ecologically valid data streams never before available in routine care.

Keppi: A Tangible User Interface for Self-Reporting Pain.

Motivated by the need to support those self-managing chronic pain, we report on the development and evaluation of a novel pressure-based tangible user interface (TUI) for the self-report of scalar values representing pain intensity. Our TUI consists of a conductive foam-based, force-sensitive resistor (FSR) covered in a soft rubber with embedded signal conditioning, an ARM Cortex-M0 microprocessor, and Bluetooth Low Energy (BLE). In-lab usability and feasibility studies with 28 participants found that individuals were able to use the device to make reliable reports with four degrees of freedom as well map squeeze pressure to pain level and visual feedback. Building on insights from these studies, we further redesigned the FSR into a wearable device with multiple form factors, including a necklace, bracelet, and keychain. A usability study with an additional 7 participants from our target population, elderly individuals with chronic pain, found high receptivity to the wearable design, which offered a number of participant-valued characteristics (e.g., discreetness) along with other design implications that serve to inform the continued refinement of tangible devices that support pain self-assessment.