RESUMEN
Synthetic data generation has emerged as a promising solution to overcome the challenges which are posed by data scarcity and privacy concerns, as well as, to address the need for training artificial intelligence (AI) algorithms on unbiased data with sufficient sample size and statistical power. Our review explores the application and efficacy of synthetic data methods in healthcare considering the diversity of medical data. To this end, we systematically searched the PubMed and Scopus databases with a great focus on tabular, imaging, radiomics, time-series, and omics data. Studies involving multi-modal synthetic data generation were also explored. The type of method used for the synthetic data generation process was identified in each study and was categorized into statistical, probabilistic, machine learning, and deep learning. Emphasis was given to the programming languages used for the implementation of each method. Our evaluation revealed that the majority of the studies utilize synthetic data generators to: (i) reduce the cost and time required for clinical trials for rare diseases and conditions, (ii) enhance the predictive power of AI models in personalized medicine, (iii) ensure the delivery of fair treatment recommendations across diverse patient populations, and (iv) enable researchers to access high-quality, representative multimodal datasets without exposing sensitive patient information, among others. We underline the wide use of deep learning based synthetic data generators in 72.6 % of the included studies, with 75.3 % of the generators being implemented in Python. A thorough documentation of open-source repositories is finally provided to accelerate research in the field.
RESUMEN
Monitoring suspended sediment concentration (SSC) in rivers is pivotal for water quality management and sustainable river ecosystem development. However, achieving continuous and precise SSC monitoring is fraught with challenges, including low automation, lengthy measurement processes, and high cost. This study proposes an innovative approach for SSC identification in rivers using multimodal data fusion. We developed a robust model by harnessing colour features from video images, motion characteristics from the Lucas-Kanade (LK) optical flow method, and temperature data. By integrating ResNet with a mixed density network (MDN), our method fused the image and optical flow fields, and temperature data to enhance accuracy and reliability. Validated at a hydropower station in the Xinjiang Uygur Autonomous Region, China, the results demonstrated that while the image field alone offers a baseline level of SSC identification, it experiences local errors under specific conditions. The incorporation of optical flow and water temperature information enhanced model robustness, particularly when coupling the image and optical flow fields, yielding a Nash-Sutcliffe efficiency (NSE) of 0.91. Further enhancement was observed with the combined use of all three data types, attaining an NSE of 0.93. This integrated approach offers a more accurate SSC identification solution, enabling non-contact, low-cost measurements, facilitating remote online monitoring, and supporting water resource management and river water-sediment element monitoring.
Asunto(s)
Monitoreo del Ambiente , Ríos , Temperatura , Ríos/química , Monitoreo del Ambiente/métodos , Sedimentos Geológicos/análisis , China , Calidad del AguaRESUMEN
Background: It is crucial to accurately predict the disease progression of systemic arterial hypertension in order to determine the most effective therapeutic strategy. To achieve this, we have employed a multimodal data-integration approach to predict the longitudinal progression of new-onset systemic arterial hypertension patients with suspected obstructive sleep apnea (OSA) at the individual level. Methods: We developed and validated a predictive nomogram model that utilizes multimodal data, consisting of clinical features, laboratory tests, and sleep monitoring data. We assessed the probabilities of major adverse cardiac and cerebrovascular events (MACCEs) as scores for participants in longitudinal cohorts who have systemic arterial hypertension and suspected OSA. In this cohort study, MACCEs were considered as a composite of cardiac mortality, acute coronary syndrome and nonfatal stroke. The least absolute shrinkage and selection operator (LASSO) regression and multiple Cox regression analyses were performed to identify independent risk factors for MACCEs among these patients. Results: 448 patients were randomly assigned to the training cohort while 189 were assigned to the verification cohort. Four clinical variables were enrolled in the constructed nomogram: age, diabetes mellitus, triglyceride, and apnea-hypopnea index (AHI). This model accurately predicted 2-year and 3-year MACCEs, achieving an impressive area under the receiver operating characteristic (ROC) curve of 0.885 and 0.784 in the training cohort, respectively. In the verification cohort, the performance of the nomogram model had good discriminatory power, with an area under the ROC curve of 0.847 and 0.729 for 2-year and 3-year MACCEs, respectively. The correlation between predicted and actual observed MACCEs was high, provided by a calibration plot, for training and verification cohorts. Conclusions: Our study yielded risk stratification for systemic arterial hypertension patients with suspected OSA, which can be quantified through the integration of multimodal data, thus highlighting OSA as a spectrum of disease. This prediction nomogram could be instrumental in defining the disease state and long-term clinical outcomes.
RESUMEN
BACKGROUND AND OBJECTIVE: The importance of early diagnosis of Alzheimer's Disease (AD) is by no means negligible because no cure has been recognized for it rather than some therapies only lowering the pace of progression. The research gap reveals information on the lack of an automatic non-invasive approach toward the diagnosis of AD, in particular with the help of Virtual Reality (VR) and Artificial Intelligence. Another perspective highlights that current VR studies fail to incorporate a comprehensive range of cognitive tests and consider design notes for elderlies, leading to unreliable results. METHODS: This paper tried to design a VR environment suitable for older adults in which three cognitive assessments namely: ADAS-Cog, Montreal Cognitive Assessment (MoCA), and Mini Mental State Exam (MMSE), are implemented. Moreover, a 3DCNN-ML model was trained based on the corresponding cognitive tests and Magnetic Resonance Imaging (MRI) with different modalities using the Alzheimer's Disease Neuroimaging Initiative 2 (ADNI2) dataset and incorporated into the application to predict if the patient suffers from AD. RESULTS: The model has undergone three experiments with different modalities (Cognitive Scores (CS), MRI images, and CS-MRI). As for the CS-MRI experiment, the trained model achieved 97%, 95%, 95%, 96%, and 94% in terms of precision, recall, F1-score, AUC, and accuracy respectively. The considered design notes were also assessed using a new proposed questionnaire based on existing ones in terms of user experience, user interface, mechanics, in-env assistance, and VR induced symptoms and effects. The designed VR system provided an acceptable level of user experience, with participants reporting an enjoyable and immersive experience. While there were areas for improvement, including graphics and sound quality, as well as comfort issues with prolonged HMD use, the user interface and mechanics of the system were generally well-received. CONCLUSIONS: The reported results state that our method's comprehensive analysis of 3D brain volumes and incorporation of cognitive scores enabled earlier detection of AD progression, potentially allowing for timely interventions and improved patient outcomes. The proposed integrated system provided us with promising insights for improvements in the diagnosis of AD using technologies.
RESUMEN
In various fields, including college admission, medical board certifications, and military recruitment, high-stakes decisions are frequently made based on scores obtained from large-scale assessments. These decisions necessitate precise and reliable scores that enable valid inferences to be drawn about test-takers. However, the ability of such tests to provide reliable, accurate inference on a test-taker's performance could be jeopardized by aberrant test-taking practices, for instance, practicing real items prior to the test. As a result, it is crucial for administrators of such assessments to develop strategies that detect potential aberrant test-takers after data collection. The aim of this study is to explore the implementation of machine learning methods in combination with multimodal data fusion strategies that integrate bio-information technology, such as eye-tracking, and psychometric measures, including response times and item responses, to detect aberrant test-taking behaviors in technology-assisted remote testing settings.
RESUMEN
Research on drug-drug interaction (DDI) prediction, particularly in identifying DDI event types, is crucial for understanding adverse drug reactions and drug combinations. This work introduces a Bidirectional Recurrent Neural Network model for DDI event type prediction (BiRNN-DDI), which simultaneously considers structural relationships and contextual information. Our BiRNN-DDI model constructs drug feature graphs to mine structural relationships. For contextual information, it transforms drug graphs into sequences and employs a two-channel structure, integrating BiRNN, to obtain contextual representations of drug-drug pairs. The model's effectiveness is demonstrated through comparisons with state-of-the-art models on two DDI event-type benchmarks. Extensive experimental results reveal that BiRNN-DDI surpasses other models in accuracy, AUPR, AUC, F1 score, Precision, and Recall metrics on both small and large datasets. Additionally, our model exhibits a lower parameter space, indicating more efficient learning of drug feature representations and prediction of potential DDI event types.
RESUMEN
BACKGROUND: Tuberculosis (TB) kills approximately 1.6 million people yearly despite the fact anti-TB drugs are generally curative. Therefore, TB-case detection and monitoring of therapy, need a comprehensive approach. Automated radiological analysis, combined with clinical, microbiological, and immunological data, by machine learning (ML), can help achieve it. METHODS: Six rhesus macaques were experimentally inoculated with pathogenic Mycobacterium tuberculosis in the lung. Data, including Computed Tomography (CT), were collected at 0, 2, 4, 8, 12, 16, and 20 weeks. RESULTS: Our ML-based CT analysis (TB-Net) efficiently and accurately analyzed disease progression, performing better than standard deep learning model (LLM OpenAI's CLIP Vi4). TB-Net based results were more consistent than, and confirmed independently by, blinded manual disease scoring by two radiologists and exhibited strong correlations with blood biomarkers, TB-lesion volumes, and disease-signs during disease pathogenesis. CONCLUSION: The proposed approach is valuable in early disease detection, monitoring efficacy of therapy, and clinical decision making.
Asunto(s)
Biomarcadores , Aprendizaje Profundo , Macaca mulatta , Mycobacterium tuberculosis , Tomografía Computarizada por Rayos X , Animales , Biomarcadores/sangre , Tomografía Computarizada por Rayos X/veterinaria , Tuberculosis/veterinaria , Tuberculosis/diagnóstico por imagen , Modelos Animales de Enfermedad , Tuberculosis Pulmonar/diagnóstico por imagen , Masculino , Femenino , Pulmón/diagnóstico por imagen , Pulmón/patología , Pulmón/microbiología , Enfermedades de los Monos/diagnóstico por imagen , Enfermedades de los Monos/microbiologíaRESUMEN
Amidst the rapid advancement of Internet of Things (IoT) technology and the burgeoning field of Multimodal Learning Analytics (MMLA), this study employs spatial positioning technology as a case study to investigate the potential of multimodal data in assessing children's social development. This study combines the spatial positioning data of preschool children collected during free play sessions in natural educational settings and the spatial metrics constructed based on observational studies to establish and validate a sociometric status Decision Tree classification model. The findings suggest that the model can overall accurately identify children with three distinct sociometric statuses, albeit with some variability in efficacy across different sociometric groups and age groups. Notably, the model demonstrates a high hitting rate in identifying the potentially neglected children, providing valuable support for educators in understanding and fostering children's developmental needs. This study also highlights the advantages of emerging technology and multimodal data application in child development assessment.
RESUMEN
Textual data often describe events in time but frequently contain little information about their specific timing, whereas complementary structured data streams may have precise timestamps but may omit important contextual information. We investigate the problem in healthcare, where we produce clinician annotations of discharge summaries, with access to either unimodal (text) or multimodal (text and tabular) data, (i) to determine event interval timings and (ii) to train multimodal language models to locate those events in time. We find our annotation procedures, dashboard tools, and annotations result in high-quality timestamps. Specifically, the multimodal approach produces more precise timestamping, with uncertainties of the lower bound, upper bounds, and duration reduced by 42% (95% CI 34-51%), 36% (95% CI 28-44%), and 13% (95% CI 10-17%), respectively. In the classification version of our task, we find that, trained on our annotations, our multimodal BERT model outperforms unimodal BERT model and Llama-2 encoder-decoder models with improvements in F1 scores for upper (10% and 61%, respectively) and lower bounds (8% and 56%, respectively). The code for the annotation tool and the BERT model is available (link).
RESUMEN
Whole-brain functional connectivity networks (connectomes) have been characterized at different scales in humans using EEG and fMRI. Multimodal epileptic networks have also been investigated, but the relationship between EEG and fMRI defined networks on a whole-brain scale is unclear. A unified multimodal connectome description, mapping healthy and pathological networks would close this knowledge gap. Here, we characterize the spatial correlation between the EEG and fMRI connectomes in right and left temporal lobe epilepsy (rTLE/lTLE). From two centers, we acquired resting-state concurrent EEG-fMRI of 35 healthy controls and 34 TLE patients. EEG-fMRI data was projected into the Desikan brain atlas, and functional connectomes from both modalities were correlated. EEG and fMRI connectomes were moderately correlated. This correlation was increased in rTLE when compared to controls for EEG-delta/theta/alpha/beta. Conversely, multimodal correlation in lTLE was decreased in respect to controls for EEG-beta. While the alteration was global in rTLE, in lTLE it was locally linked to the default mode network. The increased multimodal correlation in rTLE and decreased correlation in lTLE suggests a modality-specific lateralized differential reorganization in TLE, which needs to be considered when comparing results from different modalities. Each modality provides distinct information, highlighting the benefit of multimodal assessment in epilepsy.
The relationship between resting-state hemodynamic (fMRI) and electrophysiological (EEG) connectivity has been investigated in healthy subjects, but this relationship is unknown in patients with left and right temporal lobe epilepsies (l/rTLE). Does the magnitude of the relationship differ between healthy subjects and patients? What role does the laterality of the epileptic focus play? What are the spatial contributions to this relationship? Here we use concurrent EEG-fMRI recordings of 65 subjects from two centers (35 controls, 34 TLE patients), to assess the correlation between EEG and fMRI connectivity. For all datasets, frequency-specific changes in cross-modal correlation were seen in lTLE and rTLE. EEG and fMRI connectivities do not measure perfectly overlapping brain networks and provide distinct information on brain networks altered in TLE, highlighting the benefit of multimodal assessment to inform about normal and pathological brain function.
RESUMEN
Herbal medicines, particularly traditional Chinese medicines (TCMs), are a rich source of natural products with significant therapeutic potential. However, understanding their mechanisms of action is challenging due to the complexity of their multi-ingredient compositions. We introduced Herb-CMap, a multimodal fusion framework leveraging protein-protein interactions and herb-perturbed gene expression signatures. Utilizing a network-based heat diffusion algorithm, Herb-CMap creates a connectivity map linking herb perturbations to their therapeutic targets, thereby facilitating the prioritization of active ingredients. As a case study, we applied Herb-CMap to Suhuang antitussive capsule (Suhuang), a TCM formula used for treating cough variant asthma (CVA). Using in vivo rat models, our analysis established the transcriptomic signatures of Suhuang and identified its key compounds, such as quercetin and luteolin, and their target genes, including IL17A, PIK3CB, PIK3CD, AKT1, and TNF. These drug-target interactions inhibit the IL-17 signaling pathway and deactivate PI3K, AKT, and NF-κB, effectively reducing lung inflammation and alleviating CVA. The study demonstrates the efficacy of Herb-CMap in elucidating the molecular mechanisms of herbal medicines, offering valuable insights for advancing drug discovery in TCM.
Asunto(s)
Antitusígenos , Medicamentos Herbarios Chinos , Medicina Tradicional China , Animales , Medicamentos Herbarios Chinos/farmacología , Medicamentos Herbarios Chinos/uso terapéutico , Medicina Tradicional China/métodos , Ratas , Antitusígenos/farmacología , Antitusígenos/uso terapéutico , Mapas de Interacción de Proteínas/efectos de los fármacos , Asma/tratamiento farmacológico , Asma/metabolismo , Asma/genética , Transducción de Señal/efectos de los fármacos , Tos/tratamiento farmacológico , Transcriptoma , HumanosRESUMEN
PURPOSE: To develop and validate a predictive combined model for metastasis in patients with clear cell renal cell carcinoma (ccRCC) by integrating multimodal data. MATERIALS AND METHODS: In this retrospective study, the clinical and imaging data (CT and ultrasound) of patients with ccRCC confirmed by pathology from three tertiary hospitals in different regions were collected from January 2013 to January 2023. We developed three models, including a clinical model, a radiomics model, and a combined model. The performance of the model was determined based on its discriminative power and clinical utility. The evaluation indicators included area under the receiver operating characteristic curve (AUC) value, accuracy, sensitivity, specificity, negative predictive value, positive predictive value and decision curve analysis (DCA) curve. RESULTS: A total of 251 patients were evaluated. Patients (n = 166) from Shandong University Qilu Hospital (Jinan) were divided into the training cohort, of which 50 patients developed metastases; patients (n = 37) from Shandong University Qilu Hospital (Qingdao) were used as internal testing, of which 15 patients developed metastases; patients (n = 48) from Changzhou Second People's Hospital were used as external testing, of which 13 patients developed metastases. In the training set, the combined model showed the highest performance (AUC, 0.924) in predicting lymph node metastasis (LNM), while the clinical and radiomics models both had AUCs of 0.845 and 0.870, respectively. In the internal testing, the combined model had the highest performance (AUC, 0.877) for predicting LNM, while the AUCs of the clinical and radiomics models were 0.726 and 0.836, respectively. In the external testing, the combined model had the highest performance (AUC, 0.849) for predicting LNM, while the AUCs of the clinical and radiomics models were 0.708 and 0.804, respectively. The DCA curve showed that the combined model had a significant prediction probability in predicting the risk of LNM in ccRCC patients compared with the clinical model or the radiomics model. CONCLUSION: The combined model was superior to the clinical and radiomics models in predicting LNM in ccRCC patients.
Asunto(s)
Carcinoma de Células Renales , Neoplasias Renales , Aprendizaje Automático , Tomografía Computarizada por Rayos X , Humanos , Estudios Retrospectivos , Femenino , Masculino , Neoplasias Renales/diagnóstico por imagen , Neoplasias Renales/patología , Carcinoma de Células Renales/diagnóstico por imagen , Carcinoma de Células Renales/patología , Persona de Mediana Edad , Anciano , Tomografía Computarizada por Rayos X/métodos , Valor Predictivo de las Pruebas , Imagen Multimodal/métodos , Ultrasonografía/métodos , Medición de Riesgo , Adulto , Sensibilidad y EspecificidadRESUMEN
Alzheimer's Disease (AD) is a progressive neurodegenerative disorder. Due to the subtlety of symptoms in the early stages of AD, rapid and accurate clinical diagnosis is challenging, leading to a high rate of misdiagnosis. Current research on early diagnosis of AD has not sufficiently focused on tracking the progression of the disease over an extended period in subjects. To address this issue, this paper proposes an ensemble model for assisting early diagnosis of AD that combines structural magnetic resonance imaging (sMRI) data from two time points with clinical information. The model employs a three-dimensional convolutional neural network (3DCNN) and twin neural network modules to extract features from the sMRI data of subjects at two time points, while a multi-layer perceptron (MLP) is used to model the clinical information of the subjects. The objective is to extract AD-related features from the multi-modal data of the subjects as much as possible, thereby enhancing the diagnostic performance of the ensemble model. Experimental results show that based on this model, the classification accuracy rate is 89% for differentiating AD patients from normal controls (NC), 88% for differentiating mild cognitive impairment converting to AD (MCIc) from NC, and 69% for distinguishing non-converting mild cognitive impairment (MCInc) from MCIc, confirming the effectiveness and efficiency of the proposed method for early diagnosis of AD, as well as its potential to play a supportive role in the clinical diagnosis of early Alzheimer's disease.
Asunto(s)
Enfermedad de Alzheimer , Diagnóstico Precoz , Imagen por Resonancia Magnética , Redes Neurales de la Computación , Enfermedad de Alzheimer/diagnóstico por imagen , Enfermedad de Alzheimer/diagnóstico , Humanos , Imagen por Resonancia Magnética/métodos , Progresión de la Enfermedad , AlgoritmosRESUMEN
Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches-Audio Branch, Video Branch, and Text Branch-each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks-reading and interviewing-implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.
Asunto(s)
Depresión , Humanos , Depresión/diagnóstico , Grabación en Video , Emociones/fisiología , Aprendizaje Profundo , Expresión Facial , Femenino , Masculino , Adulto , Redes Neurales de la ComputaciónRESUMEN
The proliferation of online disinformation and fake news, particularly in the context of breaking news events, demands the development of effective detection mechanisms. While textual content remains the predominant medium for disseminating misleading information, the contribution of other modalities is increasingly emerging within online outlets and social media platforms. However, multimodal datasets, which incorporate diverse modalities such as texts and images, are not very common yet, especially in low-resource languages. This study addresses this gap by releasing a dataset tailored for multimodal fake news detection in the Italian language. This dataset was originally employed in a shared task on the Italian language. The dataset is divided into two data subsets, each corresponding to a distinct sub-task. In sub-task 1, the goal is to assess the effectiveness of multimodal fake news detection systems. Sub-task 2 aims to delve into the interplay between text and images, specifically analyzing how these modalities mutually influence the interpretation of content when distinguishing between fake and real news. Both sub-tasks were managed as classification problems. The dataset consists of social media posts and news articles. After collecting it, it was labeled via crowdsourcing. Annotators were provided with external knowledge about the topic of the news to be labeled, enhancing their ability to discriminate between fake and real news. The data subsets for sub-task 1 and sub-task 2 consist of 913 and 1350 items, respectively, encompassing newspaper articles and tweets.
RESUMEN
This study presents a pioneering approach that leverages advanced sensing technologies and data processing techniques to enhance the process of clinical documentation generation during medical consultations. By employing sophisticated sensors to capture and interpret various cues such as speech patterns, intonations, or pauses, the system aims to accurately perceive and understand patient-doctor interactions in real time. This sensing capability allows for the automation of transcription and summarization tasks, facilitating the creation of concise and informative clinical documents. Through the integration of automatic speech recognition sensors, spoken dialogue is seamlessly converted into text, enabling efficient data capture. Additionally, deep models such as Transformer models are utilized to extract and analyze crucial information from the dialogue, ensuring that the generated summaries encapsulate the essence of the consultations accurately. Despite encountering challenges during development, experimentation with these sensing technologies has yielded promising results. The system achieved a maximum ROUGE-1 metric score of 0.57, demonstrating its effectiveness in summarizing complex medical discussions. This sensor-based approach aims to alleviate the administrative burden on healthcare professionals by automating documentation tasks and safeguarding important patient information. Ultimately, by enhancing the efficiency and reliability of clinical documentation, this innovative method contributes to improving overall healthcare outcomes.
Asunto(s)
Aprendizaje Profundo , Humanos , Software de Reconocimiento del HablaRESUMEN
In the big data era, integrating diverse data modalities poses significant challenges, particularly in complex fields like healthcare. This paper introduces a new process model for multimodal Data Fusion for Data Mining, integrating embeddings and the Cross-Industry Standard Process for Data Mining with the existing Data Fusion Information Group model. Our model aims to decrease computational costs, complexity, and bias while improving efficiency and reliability. We also propose "disentangled dense fusion," a novel embedding fusion method designed to optimize mutual information and facilitate dense inter-modality feature interaction, thereby minimizing redundant information. We demonstrate the model's efficacy through three use cases: predicting diabetic retinopathy using retinal images and patient metadata, domestic violence prediction employing satellite imagery, internet, and census data, and identifying clinical and demographic features from radiography images and clinical notes. The model achieved a Macro F1 score of 0.92 in diabetic retinopathy prediction, an R-squared of 0.854 and sMAPE of 24.868 in domestic violence prediction, and a macro AUC of 0.92 and 0.99 for disease prediction and sex classification, respectively, in radiological analysis. These results underscore the Data Fusion for Data Mining model's potential to significantly impact multimodal data processing, promoting its adoption in diverse, resource-constrained settings.
RESUMEN
A late post-traumatic seizure (LPTS), a consequence of traumatic brain injury (TBI), can potentially evolve into a lifelong condition known as post-traumatic epilepsy (PTE). Presently, the mechanism that triggers epileptogenesis in TBI patients remains elusive, inspiring the epilepsy community to devise ways to predict which TBI patients will develop PTE and to identify potential biomarkers. In response to this need, our study collected comprehensive, longitudinal multimodal data from 48 TBI patients across multiple participating institutions. A supervised binary classification task was created, contrasting data from LPTS patients with those without LPTS. To accommodate missing modalities in some subjects, we took a two-pronged approach. Firstly, we extended a graphical model-based Bayesian estimator to directly classify subjects with incomplete modality. Secondly, we explored conventional imputation techniques. The imputed multimodal information was then combined, following several fusion and dimensionality reduction techniques found in the literature, and subsequently fitted to a kernel- or a tree-based classifier. For this fusion, we proposed two new algorithms: recursive elimination of correlated components (RECC) that filters information based on the correlation between the already selected features, and information decomposition and selective fusion (IDSF), which effectively recombines information from decomposed multimodal features. Our cross-validation findings showed that the proposed IDSF algorithm delivers superior performance based on the area under the curve (AUC) score. Ultimately, after rigorous statistical comparisons and interpretable machine learning examination using Shapley values of the most frequently selected features, we recommend the two following magnetic resonance imaging (MRI) abnormalities as potential biomarkers: the left anterior limb of internal capsule in diffusion MRI (dMRI), and the right middle temporal gyrus in functional MRI (fMRI).
Asunto(s)
Biomarcadores , Lesiones Traumáticas del Encéfalo , Aprendizaje Automático , Neuroimagen , Humanos , Lesiones Traumáticas del Encéfalo/diagnóstico por imagen , Lesiones Traumáticas del Encéfalo/complicaciones , Neuroimagen/métodos , Masculino , Femenino , Imagen por Resonancia Magnética/métodos , Adulto , Algoritmos , Epilepsia Postraumática/diagnóstico por imagen , Epilepsia Postraumática/etiología , Imagen Multimodal/métodos , Convulsiones/diagnóstico por imagen , Teorema de Bayes , Persona de Mediana EdadRESUMEN
Integration of multiple data sources presents a challenge for accurate prediction of molecular patho-phenotypic features in automated analysis of data from human model systems. Here, we applied a machine learning-based data integration to distinguish patho-phenotypic features at the subcellular level for dilated cardiomyopathy (DCM). We employed a human induced pluripotent stem cell-derived cardiomyocyte (iPSC-CM) model of a DCM mutation in the sarcomere protein troponin T (TnT), TnT-R141W, compared to isogenic healthy (WT) control iPSC-CMs. We established a multimodal data fusion (MDF)-based analysis to integrate source datasets for Ca2+ transients, force measurements, and contractility recordings. Data were acquired for three additional layer types, single cells, cell monolayers, and 3D spheroid iPSC-CM models. For data analysis, numerical conversion as well as fusion of data from Ca2+ transients, force measurements, and contractility recordings, a non-negative blind deconvolution (NNBD)-based method was applied. Using an XGBoost algorithm, we found a high prediction accuracy for fused single cell, monolayer, and 3D spheroid iPSC-CM models (≥92 ± 0.08â¯%), as well as for fused Ca2+ transient, beating force, and contractility models (>96 ± 0.04â¯%). Integrating MDF and XGBoost provides a highly effective analysis tool for prediction of patho-phenotypic features in complex human disease models such as DCM iPSC-CMs.
Asunto(s)
Cardiomiopatía Dilatada , Células Madre Pluripotentes Inducidas , Aprendizaje Automático , Células Madre Pluripotentes Inducidas/metabolismo , Células Madre Pluripotentes Inducidas/citología , Células Madre Pluripotentes Inducidas/patología , Cardiomiopatía Dilatada/patología , Cardiomiopatía Dilatada/metabolismo , Humanos , Fenotipo , Miocitos Cardíacos/metabolismo , Miocitos Cardíacos/patología , Troponina T/metabolismo , Calcio/metabolismoRESUMEN
Reinforcement learning has achieved promising results on robotic control tasks but struggles to leverage information effectively from multiple sensory modalities that differ in many characteristics. Recent works construct auxiliary losses based on reconstruction or mutual information to extract joint representations from multiple sensory inputs to improve the sample efficiency and performance of reinforcement learning algorithms. However, the representations learned by these methods could capture information irrelevant to learning a policy and may degrade the performance. We argue that compressing information in the learned joint representations about raw multimodal observations is helpful, and propose a multimodal information bottleneck model to learn task-relevant joint representations from egocentric images and proprioception. Our model compresses and retains the predictive information in multimodal observations for learning a compressed joint representation, which fuses complementary information from visual and proprioceptive feedback and meanwhile filters out task-irrelevant information in raw multimodal observations. We propose to minimize the upper bound of our multimodal information bottleneck objective for computationally tractable optimization. Experimental evaluations on several challenging locomotion tasks with egocentric images and proprioception show that our method achieves better sample efficiency and zero-shot robustness to unseen white noise than leading baselines. We also empirically demonstrate that leveraging information from egocentric images and proprioception is more helpful for learning policies on locomotion tasks than solely using one single modality.