Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns.

Objective: The current clinical practice of psychiatric evaluation suffers from subjectivity and bias, and requires highly skilled professionals that are often unavailable or unaffordable. Objective digital biomarkers have shown the potential to address these issues. In this work, we investigated whether behavioral and physiological signals, extracted from remote interviews, provided complimentary information for assessing psychiatric disorders. Methods: Time series of multimodal features were derived from four conceptual modes: facial expression, vocal expression, linguistic expression, and cardiovascular modulation. The features were extracted from simultaneously recorded audio and video of remote interviews using task-specific and foundation models. Averages, standard deviations, and hidden Markov model-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination. Results: Statistically significant feature differences were found between controls and subjects with mental health conditions. Correlations were found between features and self-rated depression and anxiety scores. Visual heart rate dynamics achieved the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities achieved AUROCs of 0.72-0.82. Features from task-specific models outperformed features from foundation models. Conclusion: Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status. Significance: The proposed multimodal approach has the potential to facilitate objective, remote, and low-cost assessment for low-burden automated mental health services.

Using HIPAA (Health Insurance Portability and Accountability Act)-Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study.

BACKGROUND: Automatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription services among different diagnostic groups in a mental health setting. There has also been little research into the types of words ASR transcriptions mistakenly generate or omit. OBJECTIVE: This study compared the WER of 3 ASR transcription services (Amazon Transcribe [Amazon.com, Inc], Zoom-Otter AI [Zoom Video Communications, Inc], and Whisper [OpenAI Inc]) in interviews across 2 different clinical categories (controls and participants experiencing a variety of mental health conditions). These ASR transcription services were also compared with a commercial human transcription service, Rev (Rev.Com, Inc). Words that were either included or excluded by the error in the transcripts were systematically analyzed by their Linguistic Inquiry and Word Count categories. METHODS: Participants completed a 1-time research psychiatric interview, which was recorded on a secure server. Transcriptions created by the research team were used as the gold standard from which WER was calculated. The interviewees were categorized into either the control group (n=18) or the mental health condition group (n=47) using the Mini-International Neuropsychiatric Interview. The total sample included 65 participants. Brunner-Munzel tests were used for comparing independent sets, such as the diagnostic groupings, and Wilcoxon signed rank tests were used for correlated samples when comparing the total sample between different transcription services. RESULTS: There were significant differences between each ASR transcription service's WER (P<.001). Amazon Transcribe's output exhibited significantly lower WERs compared with the Zoom-Otter AI's and Whisper's ASR. ASR performances did not significantly differ across the 2 different clinical categories within each service (P>.05). A comparison between the human transcription service output from Rev and the best-performing ASR (Amazon Transcribe) demonstrated a significant difference (P<.001), with Rev having a slightly lower median WER (7.6%, IQR 5.4%-11.35 vs 8.9%, IQR 6.9%-11.6%). Heat maps and spider plots were used to visualize the most common errors in Linguistic Inquiry and Word Count categories, which were found to be within 3 overarching categories: Conversation, Cognition, and Function. CONCLUSIONS: Overall, consistent with previous literature, our results suggest that the WER between manual and automated transcription services may be narrowing as ASR services advance. These advances, coupled with decreased cost and time in receiving transcriptions, may make ASR transcriptions a more viable option within health care settings. However, more research is required to determine if errors in specific types of words impact the analysis and usability of these transcriptions, particularly for specific applications and in a variety of populations in terms of clinical diagnosis, literacy level, accent, and cultural origin.

Multimodal Assessment of Schizophrenia and Depression Utilizing Video, Acoustic, Locomotor, Electroencephalographic, and Heart Rate Technology: Protocol for an Observational Study.

BACKGROUND: Current standards of psychiatric assessment and diagnostic evaluation rely primarily on the clinical subjective interpretation of a patient's outward manifestations of their internal state. While psychometric tools can help to evaluate these behaviors more systematically, the tools still rely on the clinician's interpretation of what are frequently nuanced speech and behavior patterns. With advances in computing power, increased availability of clinical data, and improving resolution of recording and sensor hardware (including acoustic, video, accelerometer, infrared, and other modalities), researchers have begun to demonstrate the feasibility of cutting-edge technologies in aiding the assessment of psychiatric disorders. OBJECTIVE: We present a research protocol that utilizes facial expression, eye gaze, voice and speech, locomotor, heart rate, and electroencephalography monitoring to assess schizophrenia symptoms and to distinguish patients with schizophrenia from those with other psychiatric disorders and control subjects. METHODS: We plan to recruit three outpatient groups: (1) 50 patients with schizophrenia, (2) 50 patients with unipolar major depressive disorder, and (3) 50 individuals with no psychiatric history. Using an internally developed semistructured interview, psychometrically validated clinical outcome measures, and a multimodal sensing system utilizing video, acoustic, actigraphic, heart rate, and electroencephalographic sensors, we aim to evaluate the system's capacity in classifying subjects (schizophrenia, depression, or control), to evaluate the system's sensitivity to within-group symptom severity, and to determine if such a system can further classify variations in disorder subtypes. RESULTS: Data collection began in July 2020 and is expected to continue through December 2022. CONCLUSIONS: If successful, this study will help advance current progress in developing state-of-the-art technology to aid clinical psychiatric assessment and treatment. If our findings suggest that these technologies are capable of resolving diagnoses and symptoms to the level of current psychometric testing and clinician judgment, we would be among the first to develop a system that can eventually be used by clinicians to more objectively diagnose and assess schizophrenia and depression with the possibility of less risk of bias. Such a tool has the potential to improve accessibility to care; to aid clinicians in objectively evaluating diagnoses, severity of symptoms, and treatment efficacy through time; and to reduce treatment-related morbidity. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/36417.

Issues in the automated classification of multilead ecgs using heterogeneous labels and populations.

Objective.The standard twelve-lead electrocardiogram (ECG) is a widely used tool for monitoring cardiac function and diagnosing cardiac disorders. The development of smaller, lower-cost, and easier-to-use ECG devices may improve access to cardiac care in lower-resource environments, but the diagnostic potential of these devices is unclear. This work explores these issues through a public competition: the 2021 PhysioNet Challenge. In addition, we explore the potential for performance boosting through a meta-learning approach.Approach.We sourced 131,149 twelve-lead ECG recordings from ten international sources. We posted 88,253 annotated recordings as public training data and withheld the remaining recordings as hidden validation and test data. We challenged teams to submit containerized, open-source algorithms for diagnosing cardiac abnormalities using various ECG lead combinations, including the code for training their algorithms. We designed and scored the algorithms using an evaluation metric that captures the risks of different misdiagnoses for 30 conditions. After the Challenge, we implemented a semi-consensus voting model on all working algorithms.Main results.A total of 68 teams submitted 1,056 algorithms during the Challenge, providing a variety of automated approaches from both academia and industry. The performance differences across the different lead combinations were smaller than the performance differences across the different test databases, showing that generalizability posed a larger challenge to the algorithms than the choice of ECG leads. A voting model improved performance by 3.5%.Significance.The use of different ECG lead combinations allowed us to assess the diagnostic potential of reduced-lead ECG recordings, and the use of different data sources allowed us to assess the generalizability of the algorithms to diverse institutions and populations. The submission of working, open-source code for both training and testing and the use of a novel evaluation metric improved the reproducibility, generalizability, and applicability of the research conducted during the Challenge.

Data-driven approach for automatic detection of aortic valve opening: B point detection from impedance cardiogram.

Pre-ejection period (PEP), an indicator of sympathetic nervous system activity, is useful in psychophysiology and cardiovascular studies. Accurate PEP measurement is challenging and relies on robust identification of the timing of aortic valve opening, marked as the B point on impedance cardiogram (ICG) signals. The ICG sensitivity to noise and its waveform's morphological variability makes automated B point detection difficult, requiring inefficient and cumbersome expert visual annotation. In this article, we propose a machine learning-based automated algorithm to detect the aortic valve opening for PEP measurement, which is robust against noise and ICG morphological variations. We analyzed over 60 hr of synchronized ECG and ICG records from 189 subjects. A total of 3657 averaged beats were formed using our recently developed ICG noise removal algorithm. Features such as the averaged ICG waveform, its first and second derivatives, as well as high-level morphological and critical hemodynamic parameters were extracted and fed into the regression algorithms to estimate the B point location. The morphological features were extracted from our proposed "variable" physiologically valid search-window related to diverse B point shapes. A subject-wise nested cross-validation procedure was performed for parameter tuning and model assessment. After examining multiple regression models, Adaboost was selected, which demonstrated superior performance and higher robustness to five state-of-the-art algorithms that were evaluated in terms of low mean absolute error of 3.5 ms, low median absolute error of 0.0 ms, high correlation with experts' estimates (Pearson coefficient = 0.9), and low standard deviation of errors of 9.2 ms. For reproducibility, an open-source toolbox is provided.

Atrial fibrillation detection in outpatient electrocardiogram monitoring: An algorithmic crowdsourcing approach.

BACKGROUND: Atrial fibrillation (AFib) is the most common cardiac arrhythmia associated with stroke, blood clots, heart failure, coronary artery disease, and/or death. Multiple methods have been proposed for AFib detection, with varying performances, but no single approach appears to be optimal. We hypothesized that each state-of-the-art algorithm is appropriate for different subsets of patients and provides some independent information. Therefore, a set of suitably chosen algorithms, combined in a weighted voting framework, will provide a superior performance to any single algorithm. METHODS: We investigate and modify 38 state-of-the-art AFib classification algorithms for a single-lead ambulatory electrocardiogram (ECG) monitoring device. All algorithms are ranked using a random forest classifier and an expert-labeled training dataset of 2,532 recordings. The seven top-ranked algorithms are combined by using an optimized weighting approach. RESULTS: The proposed fusion algorithm, when validated on a separate test dataset consisting of 4,644 recordings, resulted in an area under the receiver operating characteristic (ROC) curve of 0.99. The sensitivity, specificity, positive-predictive-value (PPV), negative-predictive-value (NPV), and F1-score of the proposed algorithm were 0.93, 0.97, 0.87, 0.99, and 0.90, respectively, which were all superior to any single algorithm or any previously published. CONCLUSION: This study demonstrates how a set of well-chosen independent algorithms and a voting mechanism to fuse the outputs of the algorithms, outperforms any single state-of-the-art algorithm for AFib detection. The proposed framework is a case study for the general notion of crowdsourcing between open-source algorithms in healthcare applications. The extension of this framework to similar applications may significantly save time, effort, and resources, by combining readily existing algorithms. It is also a step toward the democratization of artificial intelligence and its application in healthcare.

Classification of 12-lead ECGs: the PhysioNet/Computing in Cardiology Challenge 2020.

OBJECTIVE: Vast 12-lead ECGs repositories provide opportunities to develop new machine learning approaches for creating accurate and automatic diagnostic systems for cardiac abnormalities. However, most 12-lead ECG classification studies are trained, tested, or developed in single, small, or relatively homogeneous datasets. In addition, most algorithms focus on identifying small numbers of cardiac arrhythmias that do not represent the complexity and difficulty of ECG interpretation. This work addresses these issues by providing a standard, multi-institutional database and a novel scoring metric through a public competition: the PhysioNet/Computing in Cardiology Challenge 2020. APPROACH: A total of 66 361 12-lead ECG recordings were sourced from six hospital systems from four countries across three continents; 43 101 recordings were posted publicly with a focus on 27 diagnoses. For the first time in a public competition, we required teams to publish open-source code for both training and testing their algorithms, ensuring full scientific reproducibility. MAIN RESULTS: A total of 217 teams submitted 1395 algorithms during the Challenge, representing a diversity of approaches for identifying cardiac abnormalities from both academia and industry. As with previous Challenges, high-performing algorithms exhibited significant drops ([Formula: see text]10%) in performance on the hidden test data. SIGNIFICANCE: Data from diverse institutions allowed us to assess algorithmic generalizability. A novel evaluation metric considered different misclassification errors for different cardiac abnormalities, capturing the outcomes and risks of different diagnoses. Requiring both trained models and code for training models improved the generalizability of submissions, setting a new bar in reproducibility for public data science competitions.

An open-source automated algorithm for removal of noisy beats for accurate impedance cardiogram analysis.

OBJECTIVE: The impedance cardiogram (ICG) is a non-invasive sensing modality for assessing the mechanical aspects of cardiac function, but is sensitive to artifacts from respiration, speaking, motion, and electrode displacement. Electrocardiogram (ECG)-synchronized ensemble averaging of ICG (conventional ensemble averaging method) partially mitigates these disturbances, as artifacts from intra-subject variability (ISVar) of ICG morphology and event latency remain. This paper describes an automated algorithm for removing noisy beats for improved artifact suppression in ensemble-averaged (EA) ICG beats. APPROACH: Synchronized ECG and ICG signals from 144 male subjects at rest in different psychological conditions were recorded. A 'three-stage EA ICG beat' was formed by passing 60-seconds non-overlapping ECG-synchronized ICG signals through three filtering stages. The amplitude filtering stage removed spikes/noisy beats with amplitudes outside of normal physiological ranges. Cross-correlation was applied to remove noisy beats in coarse and fine filtering stages. The accuracy of the algorithm-detected artifacts was measured with expert-identified artifacts. Agreement between the expert and the algorithm was assessed using intraclass correlation coefficients (ICC) and Bland-Altman plots. The ISVar of the cardiac parameters was evaluated to quantify improvement in these estimates provided by the proposed method. MAIN RESULTS: The proposed algorithm yielded an accuracy of 96.3% and high inter-rater reliability (ICC > 0.997). Bland-Altman plots showed consistently accurate results across values. The ISVar of the cardiac parameters derived using the proposed method was significantly lower than those derived via conventional ensemble averaging method (p < 0.0001). Enhancement in resolution of fiducial points and smoothing of higher-order time derivatives of the EA ICG beats were observed. SIGNIFICANCE: The proposed algorithm provides a robust framework for removal of noisy beats and accurate estimation of ICG-based parameters. Importantly, the methodology reduced the ISVar of cardiac parameters. An open-source toolbox has been provided to enable other researchers to readily reproduce and improve upon this work.