Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data.

AI tools intend to transform mental healthcare by providing remote estimates of depression risk using behavioral data collected by sensors embedded in smartphones. While these tools accurately predict elevated symptoms in small, homogenous populations, recent studies show that these tools are less accurate in larger, more diverse populations. In this work, we show that accuracy is reduced because sensed-behaviors are unreliable predictors of depression across individuals; specifically the sensed-behaviors that predict depression risk are inconsistent across demographic and socioeconomic subgroups. We first identified subgroups where a developed AI tool underperformed by measuring algorithmic bias, where subgroups with depression were incorrectly predicted to be at lower risk than healthier subgroups. We then found inconsistencies between sensed-behaviors predictive of depression across these subgroups. Our findings suggest that researchers developing AI tools predicting mental health from behavior should think critically about the generalizability of these tools, and consider tailored solutions for targeted populations.

Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data.

AI tools intend to transform mental healthcare by providing remote estimates of depression risk using behavioral data collected by sensors embedded in smartphones. While these tools accurately predict elevated depression symptoms in small, homogenous populations, recent studies show that these tools are less accurate in larger, more diverse populations. In this work, we show that accuracy is reduced because sensed-behaviors are unreliable predictors of depression across individuals: sensed-behaviors that predict depression risk are inconsistent across demographic and socioeconomic subgroups. We first identified subgroups where a developed AI tool underperformed by measuring algorithmic bias, where subgroups with depression were incorrectly predicted to be at lower risk than healthier subgroups. We then found inconsistencies between sensed-behaviors predictive of depression across these subgroups. Our findings suggest that researchers developing AI tools predicting mental health from sensed-behaviors should think critically about the generalizability of these tools, and consider tailored solutions for targeted populations.

Understanding Mental Health Clinicians' Perceptions and Concerns Regarding Using Passive Patient-Generated Health Data for Clinical Decision-Making: Qualitative Semistructured Interview Study.

BACKGROUND: Digital health-tracking tools are changing mental health care by giving patients the ability to collect passively measured patient-generated health data (PGHD; ie, data collected from connected devices with little to no patient effort). Although there are existing clinical guidelines for how mental health clinicians should use more traditional, active forms of PGHD for clinical decision-making, there is less clarity on how passive PGHD can be used. OBJECTIVE: We conducted a qualitative study to understand mental health clinicians' perceptions and concerns regarding the use of technology-enabled, passively collected PGHD for clinical decision-making. Our interviews sought to understand participants' current experiences with and visions for using passive PGHD. METHODS: Mental health clinicians providing outpatient services were recruited to participate in semistructured interviews. Interview recordings were deidentified, transcribed, and qualitatively coded to identify overarching themes. RESULTS: Overall, 12 mental health clinicians (n=11, 92% psychiatrists and n=1, 8% clinical psychologist) were interviewed. We identified 4 overarching themes. First, passive PGHD are patient driven-we found that current passive PGHD use was patient driven, not clinician driven; participating clinicians only considered passive PGHD for clinical decision-making when patients brought passive data to clinical encounters. The second theme was active versus passive data as subjective versus objective data-participants viewed the contrast between active and passive PGHD as a contrast between interpretive data on patients' mental health and objective information on behavior. Participants believed that prioritizing passive over self-reported, active PGHD would reduce opportunities for patients to reflect upon their mental health, reducing treatment engagement and raising questions about how passive data can best complement active data for clinical decision-making. Third, passive PGHD must be delivered at appropriate times for action-participants were concerned with the real-time nature of passive PGHD; they believed that it would be infeasible to use passive PGHD for real-time patient monitoring outside clinical encounters and more feasible to use passive PGHD during clinical encounters when clinicians can make treatment decisions. The fourth theme was protecting patient privacy-participating clinicians wanted to protect patient privacy within passive PGHD-sharing programs and discussed opportunities to refine data sharing consent to improve transparency surrounding passive PGHD collection and use. CONCLUSIONS: Although passive PGHD has the potential to enable more contextualized measurement, this study highlights the need for building and disseminating an evidence base describing how and when passive measures should be used for clinical decision-making. This evidence base should clarify how to use passive data alongside more traditional forms of active PGHD, when clinicians should view passive PGHD to make treatment decisions, and how to protect patient privacy within passive data-sharing programs. Clear evidence would more effectively support the uptake and effective use of these novel tools for both patients and their clinicians.

Machine learning for passive mental health symptom prediction: Generalization across different longitudinal mobile sensing studies.

Mobile sensing data processed using machine learning models can passively and remotely assess mental health symptoms from the context of patients' lives. Prior work has trained models using data from single longitudinal studies, collected from demographically homogeneous populations, over short time periods, using a single data collection platform or mobile application. The generalizability of model performance across studies has not been assessed. This study presents a first analysis to understand if models trained using combined longitudinal study data to predict mental health symptoms generalize across current publicly available data. We combined data from the CrossCheck (individuals living with schizophrenia) and StudentLife (university students) studies. In addition to assessing generalizability, we explored if personalizing models to align mobile sensing data, and oversampling less-represented severe symptoms, improved model performance. Leave-one-subject-out cross-validation (LOSO-CV) results were reported. Two symptoms (sleep quality and stress) had similar question-response structures across studies and were used as outcomes to explore cross-dataset prediction. Models trained with combined data were more likely to be predictive (significant improvement over predicting training data mean) than models trained with single-study data. Expected model performance improved if the distance between training and validation feature distributions decreased using combined versus single-study data. Personalization aligned each LOSO-CV participant with training data, but only improved predicting CrossCheck stress. Oversampling significantly improved severe symptom classification sensitivity and positive predictive value, but decreased model specificity. Taken together, these results show that machine learning models trained on combined longitudinal study data may generalize across heterogeneous datasets. We encourage researchers to disseminate collected de-identified mobile sensing and mental health symptom data, and further standardize data types collected across studies to enable better assessment of model generalizability.

A call for open data to develop mental health digital biomarkers.

Digital biomarkers of mental health, created using data extracted from everyday technologies including smartphones, wearable devices, social media and computer interactions, have the opportunity to revolutionise mental health diagnosis and treatment by providing near-continuous unobtrusive and remote measures of behaviours associated with mental health symptoms. Machine learning models process data traces from these technologies to identify digital biomarkers. In this editorial, we caution clinicians against using digital biomarkers in practice until models are assessed for equitable predictions ('model equity') across demographically diverse patients at scale, behaviours over time, and data types extracted from different devices and platforms. We posit that it will be difficult for any individual clinic or large-scale study to assess and ensure model equity and alternatively call for the creation of a repository of open de-identified data for digital biomarker development.

Digital Prompts to Increase Engagement With the Headspace App and for Stress Regulation Among Parents: Feasibility Study.

BACKGROUND: Given the interrelated health of children and parents, strategies to promote stress regulation are critically important in the family context. However, the uptake of preventive mental health is limited among parents owing to competing family demands. OBJECTIVE: In this study, we aim to determine whether it is feasible and acceptable to randomize digital prompts designed to engage parents in real-time brief mindfulness activities guided by a commercially available app. METHODS: We conducted a 30-day pilot microrandomized trial among a sample of parents who used Android smartphones. Each day during a parent-specified time frame, participants had a 50% probability of receiving a prompt with a message encouraging them to engage in a mindfulness activity using a commercial app, Headspace. In the 24 hours following randomization, ecological momentary assessments and passively collected smartphone data were used to assess proximal engagement (yes or no) with the app and any mindfulness activity (with or without the app). These data were combined with baseline and exit surveys to determine feasibility and acceptability. RESULTS: Over 4 months, 83 interested parents were screened, 48 were eligible, 16 were enrolled, and 10 were successfully onboarded. Reasons for nonparticipation included technology barriers, privacy concerns, time constraints, or change of mind. In total, 80% (8/10) of parents who onboarded successfully completed all aspects of the intervention. While it is feasible to randomize prompt delivery, only 60% (6/10) of parents reported that the timing of prompts was helpful despite having control over the delivery window. Across the study period, we observed higher self-reported engagement with Headspace on days with prompts (31/62, 50% of days), as opposed to days without prompts (33/103, 32% of days). This pattern was consistent for most participants in this study (7/8, 87%). The time spent using the app on days with prompts (mean 566, SD 378 seconds) was descriptively higher than on days without prompts (mean 225, SD 276 seconds). App usage was highest during the first week and declined over each of the remaining 3 weeks. However, self-reported engagement in mindfulness activities without the app increased over time. Self-reported engagement with any mindfulness activity was similar on days with (40/62, 65% of days) and without (65/103, 63% of days) prompts. Participants found the Headspace app helpful (10/10, 100%) and would recommend the program to others (9/10, 90%). CONCLUSIONS: Preliminary findings suggest that parents are receptive to using mindfulness apps to support stress management, and prompts are likely to increase engagement with the app. However, we identified several implementation challenges in the current trial, specifically a need to optimize prompt timing and frequency as a strategy to engage users in preventive digital mental health.

Burnout and the Quantified Workplace: Tensions around Personal Sensing Interventions for Stress in Resident Physicians.

Recent research has explored computational tools to manage workplace stress via personal sensing, a measurement paradigm in which behavioral data streams are collected from technologies including smartphones, wearables, and personal computers. As these tools develop, they invite inquiry into how they can be appropriately implemented towards improving workers' well-being. In this study, we explored this proposition through formative interviews followed by a design provocation centered around measuring burnout in a U.S. resident physician program. Residents and their supervising attending physicians were presented with medium-fidelity mockups of a dashboard providing behavioral data on residents' sleep, activity and time working; self-reported data on residents' levels of burnout; and a free text box where residents could further contextualize their well-being. Our findings uncover tensions around how best to measure workplace well-being, who within a workplace is accountable for worker stress, and how the introduction of such tools remakes the boundaries of appropriate information flows between worker and workplace. We conclude by charting future work confronting these tensions, to ensure personal sensing is leveraged to truly improve worker well-being.

Identifying Mobile Sensing Indicators of Stress-Resilience.

Resident physicians (residents) experiencing prolonged workplace stress are at risk of developing mental health symptoms. Creating novel, unobtrusive measures of resilience would provide an accessible approach to evaluate symptom susceptibility without the perceived stigma of formal mental health assessments. In this work, we created a system to find indicators of resilience using passive wearable sensors and smartphone-delivered ecological momentary assessment (EMA). This system identified indicators of resilience during a medical internship, the high stress first-year of a residency program. We then created density estimation approaches to predict these indicators before mental health changes occurred, and validated whether the predicted indicators were also associated with resilience. Our system identified resilience indicators associated with physical activity (step count), sleeping behavior, reduced heart rate, increased mood, and reduced mood variability. Density estimation models were able to replicate a subset of the associations between sleeping behavior, heart rate, and resilience. To the best of our knowledge, this work provides the first methodology to identify and predict indicators of resilience using passive sensing and EMA. Researchers studying resident mental health can apply this approach to design resilience-building interventions and prevent mental health symptom development.

Predicting Early Warning Signs of Psychotic Relapse From Passive Sensing Data: An Approach Using Encoder-Decoder Neural Networks.

BACKGROUND: Schizophrenia spectrum disorders (SSDs) are chronic conditions, but the severity of symptomatic experiences and functional impairments vacillate over the course of illness. Developing unobtrusive remote monitoring systems to detect early warning signs of impending symptomatic relapses would allow clinicians to intervene before the patient's condition worsens. OBJECTIVE: In this study, we aim to create the first models, exclusively using passive sensing data from a smartphone, to predict behavioral anomalies that could indicate early warning signs of a psychotic relapse. METHODS: Data used to train and test the models were collected during the CrossCheck study. Hourly features derived from smartphone passive sensing data were extracted from 60 patients with SSDs (42 nonrelapse and 18 relapse >1 time throughout the study) and used to train models and test performance. We trained 2 types of encoder-decoder neural network models and a clustering-based local outlier factor model to predict behavioral anomalies that occurred within the 30-day period before a participant's date of relapse (the near relapse period). Models were trained to recreate participant behavior on days of relative health (DRH, outside of the near relapse period), following which a threshold to the recreation error was applied to predict anomalies. The neural network model architecture and the percentage of relapse participant data used to train all models were varied. RESULTS: A total of 20,137 days of collected data were analyzed, with 726 days of data (0.037%) within any 30-day near relapse period. The best performing model used a fully connected neural network autoencoder architecture and achieved a median sensitivity of 0.25 (IQR 0.15-1.00) and specificity of 0.88 (IQR 0.14-0.96; a median 108% increase in behavioral anomalies near relapse). We conducted a post hoc analysis using the best performing model to identify behavioral features that had a medium-to-large effect (Cohen d>0.5) in distinguishing anomalies near relapse from DRH among 4 participants who relapsed multiple times throughout the study. Qualitative validation using clinical notes collected during the original CrossCheck study showed that the identified features from our analysis were presented to clinicians during relapse events. CONCLUSIONS: Our proposed method predicted a higher rate of anomalies in patients with SSDs within the 30-day near relapse period and can be used to uncover individual-level behaviors that change before relapse. This approach will enable technologists and clinicians to build unobtrusive digital mental health tools that can predict incipient relapse in SSDs.

Heightened Delta Power during Slow-Wave-Sleep in Patients with Rett Syndrome Associated with Poor Sleep Efficiency.

Sleep problems are commonly reported in Rett syndrome (RTT); however the electroencephalographic (EEG) biomarkers underlying sleep dysfunction are poorly understood. The aim of this study was to analyze the temporal evolution of quantitative EEG (qEEG) biomarkers in overnight EEGs recorded from girls (2-9 yrs. old) diagnosed with RTT using a non-traditional automated protocol. In this study, EEG spectral analysis identified high delta power cycles representing slow wave sleep (SWS) in 8-9h overnight sleep EEGs from the frontal, central and occipital leads (AP axis), comparing age-matched girls with and without RTT. Automated algorithms quantitated the area under the curve (AUC) within identified SWS cycles for each spectral frequency wave form. Both age-matched RTT and control EEGs showed similar increasing trends for recorded delta wave power in the EEG leads along the antero-posterior (AP). RTT EEGs had significantly fewer numbers of SWS sleep cycles; therefore, the overall time spent in SWS was also significantly lower in RTT. In contrast, the AUC for delta power within each SWS cycle was significantly heightened in RTT and remained heightened over consecutive cycles unlike control EEGs that showed an overnight decrement of delta power in consecutive cycles. Gamma wave power associated with these SWS cycles was similar to controls. However, the negative correlation of gamma power with age (r = -.59; p<0.01) detected in controls (2-5 yrs. vs. 6-9 yrs.) was lost in RTT. Poor % SWS (i.e., time spent in SWS overnight) in RTT was also driven by the younger age-group. Incidence of seizures in RTT was associated with significantly lower number of SWS cycles. Therefore, qEEG biomarkers of SWS in RTT evolved temporally and correlated significantly with clinical severity.