SynergyX: a multi-modality mutual attention network for interpretable drug synergy prediction.

Discovering effective anti-tumor drug combinations is crucial for advancing cancer therapy. Taking full account of intricate biological interactions is highly important in accurately predicting drug synergy. However, the extremely limited prior knowledge poses great challenges in developing current computational methods. To address this, we introduce SynergyX, a multi-modality mutual attention network to improve anti-tumor drug synergy prediction. It dynamically captures cross-modal interactions, allowing for the modeling of complex biological networks and drug interactions. A convolution-augmented attention structure is adopted to integrate multi-omic data in this framework effectively. Compared with other state-of-the-art models, SynergyX demonstrates superior predictive accuracy in both the General Test and Blind Test and cross-dataset validation. By exhaustively screening combinations of approved drugs, SynergyX reveals its ability to identify promising drug combination candidates for potential lung cancer treatment. Another notable advantage lies in its multidimensional interpretability. Taking Sorafenib and Vorinostat as an example, SynergyX serves as a powerful tool for uncovering drug-gene interactions and deciphering cell selectivity mechanisms. In summary, SynergyX provides an illuminating and interpretable framework, poised to catalyze the expedition of drug synergy discovery and deepen our comprehension of rational combination therapy.

Complementary multi-modality molecular self-supervised learning via non-overlapping masking for property prediction.

Self-supervised learning plays an important role in molecular representation learning because labeled molecular data are usually limited in many tasks, such as chemical property prediction and virtual screening. However, most existing molecular pre-training methods focus on one modality of molecular data, and the complementary information of two important modalities, SMILES and graph, is not fully explored. In this study, we propose an effective multi-modality self-supervised learning framework for molecular SMILES and graph. Specifically, SMILES data and graph data are first tokenized so that they can be processed by a unified Transformer-based backbone network, which is trained by a masked reconstruction strategy. In addition, we introduce a specialized non-overlapping masking strategy to encourage fine-grained interaction between these two modalities. Experimental results show that our framework achieves state-of-the-art performance in a series of molecular property prediction tasks, and a detailed ablation study demonstrates efficacy of the multi-modality framework and the masking strategy.

Radiomics features for the discrimination of tuberculomas from high grade gliomas and metastasis: a multimodal study.

BACKGROUND: Tuberculomas are prevalent in developing countries and demonstrate variable signals on MRI resulting in the overlap of the conventional imaging phenotype with other entities including glioma and brain metastasis. An accurate MRI diagnosis is important for the early institution of anti-tubercular therapy, decreased patient morbidity, mortality, and prevents unnecessary neurosurgical excision. This study aims to assess the potential of radiomics features of regular contrast images including T1W, T2W, T2W FLAIR, T1W post contrast images, and ADC maps, to differentiate between tuberculomas, high-grade-gliomas and metastasis, the commonest intra parenchymal mass lesions encountered in the clinical practice. METHODS: This retrospective study includes 185 subjects. Images were resampled, co-registered, skull-stripped, and zscore-normalized. Automated lesion segmentation was performed followed by radiomics feature extraction, train-test split, and features reduction. All machine learning algorithms that natively support multiclass classification were trained and assessed on features extracted from individual modalities as well as combined modalities. Model explainability of the best performing model was calculated using the summary plot obtained by SHAP values. RESULTS: Extra tree classifier trained on the features from ADC maps was the best classifier for the discrimination of tuberculoma from high-grade-glioma and metastasis with AUC-score of 0.96, accuracy-score of 0.923, Brier-score of 0.23. CONCLUSION: This study demonstrates that radiomics features are effective in discriminating between tuberculoma, metastasis, and high-grade-glioma with notable accuracy and AUC scores. Features extracted from the ADC maps surfaced as the most robust predictors of the target variable.

Nitric oxide nano-reactor DNMF/PLGA enables tumor vascular microenvironment and chemo-hyperthermia synergetic therapy.

BACKGROUND: Breast cancer ranks first among malignant tumors, of which triple-negative breast cancer (TNBC) is characterized by its highly invasive behavior and the worst prognosis. Timely diagnosis and precise treatment of TNBC are substantially challenging. Abnormal tumor vessels play a crucial role in TNBC progression and treatment. Nitric oxide (NO) regulates angiogenesis and maintains vascular homeostasis, while effective NO delivery can normalize the tumor vasculature. Accordingly, we have proposed here a tumor vascular microenvironment remodeling strategy based on NO-induced vessel normalization and extracellular matrix collagen degradation with multimodality imaging-guided nanoparticles against TNBC called DNMF/PLGA. RESULTS: Nanoparticles were synthesized using a chemotherapeutic agent doxorubicin (DOX), a NO donor L-arginine (L-Arg), ultrasmall spinel ferrites (MnFe2O4), and a poly (lactic-co-glycolic acid) (PLGA) shell. Nanoparticle distribution in the tumor was accurately monitored in real-time through highly enhanced magnetic resonance imaging and photoacoustic imaging. Near-infrared irradiation of tumor cells revealed that MnFe2O4 catalyzes the production of a large amount of reactive oxygen species (ROS) from H2O2, resulting in a cascade catalysis of L-Arg to trigger NO production in the presence of ROS. In addition, DOX activates niacinamide adenine dinucleotide phosphate oxidase to generate and supply H2O2. The generated NO improves the vascular endothelial cell integrity and pericellular contractility to promote vessel normalization and induces the activation of endogenous matrix metalloproteinases (mainly MMP-1 and MMP-2) so as to promote extravascular collagen degradation, thereby providing an auxiliary mechanism for efficient nanoparticle delivery and DOX penetration. Moreover, the chemotherapeutic effect of DOX and the photothermal effect of MnFe2O4 served as a chemo-hyperthermia synergistic therapy against TNBC. CONCLUSION: The two therapeutic mechanisms, along with an auxiliary mechanism, were perfectly combined to enhance the therapeutic effects. Briefly, multimodality image-guided nanoparticles provide a reliable strategy for the potential application in the fight against TNBC.

Moderate aortic stenosis: Navigating the uncharted.

Aortic stenosis (AS) stands as the most common valvular heart disease in developed countries and is characterized by progressive narrowing of the aortic valve orifice resulting in elevated transvalvular flow resistance, left ventricular hypertrophy, and progressive increased risk of heart failure and sudden death. This narrative review explores clinical challenges and evolving perspectives in moderate AS, where discrepancies between aortic valve area and pressure gradient measurements may pose diagnostic and therapeutic quandaries. Transthoracic echocardiography is the first-line imaging modality for AS evaluation, yet cases of discordance may require the application of ancillary noninvasive diagnostic modalities. This review underscores the importance of accurate grading of AS severity, especially in low-gradient phenotypes, emphasizing the need for vigilant follow-up. Current clinical guidelines primarily recommend aortic valve replacement for severe AS, potentially overlooking latent risks in moderate disease stages. The noninvasive multimodality imaging approach-including echocardiography, cardiac magnetic resonance, computed tomography, and nuclear techniques-provides unique insights into adaptive and maladaptive cardiac remodeling in AS and offers a promising avenue to deliver precise indications and exact timing for intervention in moderate AS phenotypes and asymptomatic patients, potentially improving long-term outcomes. Nevertheless, what we may have gleaned from a large amount of observational data is still insufficient to build a robust framework for clinical decision-making in moderate AS. Future research will prioritize randomized clinical trials designed to weigh the benefits and risks of preemptive aortic valve replacement in the management of moderate AS, as directed by specific imaging and nonimaging biomarkers.

MRD-YOLO: A Multispectral Object Detection Algorithm for Complex Road Scenes.

Object detection is one of the core technologies for autonomous driving. Current road object detection mainly relies on visible light, which is prone to missed detections and false alarms in rainy, night-time, and foggy scenes. Multispectral object detection based on the fusion of RGB and infrared images can effectively address the challenges of complex and changing road scenes, improving the detection performance of current algorithms in complex scenarios. However, previous multispectral detection algorithms suffer from issues such as poor fusion of dual-mode information, poor detection performance for multi-scale objects, and inadequate utilization of semantic information. To address these challenges and enhance the detection performance in complex road scenes, this paper proposes a novel multispectral object detection algorithm called MRD-YOLO. In MRD-YOLO, we utilize interaction-based feature extraction to effectively fuse information and introduce the BIC-Fusion module with attention guidance to fuse different modal information. We also incorporate the SAConv module to improve the model's detection performance for multi-scale objects and utilize the AIFI structure to enhance the utilization of semantic information. Finally, we conduct experiments on two major public datasets, FLIR_Aligned and M3FD. The experimental results demonstrate that compared to other algorithms, the proposed algorithm achieves superior detection performance in complex road scenes.

Rehabilitation Assessment System for Stroke Patients Based on Fusion-Type Optoelectronic Plethysmography Device and Multi-Modality Fusion Model: Design and Validation.

This study aimed to propose a portable and intelligent rehabilitation evaluation system for digital stroke-patient rehabilitation assessment. Specifically, the study designed and developed a fusion device capable of emitting red, green, and infrared lights simultaneously for photoplethysmography (PPG) acquisition. Leveraging the different penetration depths and tissue reflection characteristics of these light wavelengths, the device can provide richer and more comprehensive physiological information. Furthermore, a Multi-Channel Convolutional Neural Network-Long Short-Term Memory-Attention (MCNN-LSTM-Attention) evaluation model was developed. This model, constructed based on multiple convolutional channels, facilitates the feature extraction and fusion of collected multi-modality data. Additionally, it incorporated an attention mechanism module capable of dynamically adjusting the importance weights of input information, thereby enhancing the accuracy of rehabilitation assessment. To validate the effectiveness of the proposed system, sixteen volunteers were recruited for clinical data collection and validation, comprising eight stroke patients and eight healthy subjects. Experimental results demonstrated the system's promising performance metrics (accuracy: 0.9125, precision: 0.8980, recall: 0.8970, F1 score: 0.8949, and loss function: 0.1261). This rehabilitation evaluation system holds the potential for stroke diagnosis and identification, laying a solid foundation for wearable-based stroke risk assessment and stroke rehabilitation assistance.

Transcatheter Mitral Valve Intervention: Current and Future Role of Multimodality Imaging for Device Selection and Periprocedural Guidance.

Mitral regurgitation (MR) is a broadly diffuse valvular heart disease (VHD) with a significant impact on the healthcare system and patient prognosis. Transcatheter mitral valve interventions (TMVI) are now well-established techniques included in the therapeutic armamentarium for managing patients with mitral regurgitation, either primary or functional MR. Even if the guidelines give indications regarding the correct management of this VHD, the wide heterogeneity of patients' clinical backgrounds and valvular and heart anatomies make each patient a unique case, in which the appropriate device's selection requires a multimodal imaging evaluation and a multidisciplinary discussion. Proper pre-procedural evaluation plays a pivotal role in judging the feasibility of TMVI, while a cooperative work between imagers and interventionalist is also crucial for procedural success. This manuscript aims to provide an exhaustive overview of the main parameters that need to be evaluated for appropriate device selection, pre-procedural planning, intra-procedural guidance and post-operative assessment in the setting of TMVI. In addition, it tries to give some insights about future perspectives for structural cardiovascular imaging.

Multi-Modality Cardiac Imaging for Pericardial Diseases: A Contemporary Review.

Advances in multi-modality cardiac imaging have aided the evaluation, surveillance and treatment guidance of pericardial diseases, which have traditionally been a challenging group of conditions to manage. Although echocardiography remains the first-line imaging modality to assess the pericardium, both computed tomography (CT) and magnetic resonance imaging (MRI) have valuable complimentary roles. It is critical for clinicians to have a clear understanding of the utilities, advantages and disadvantages of these cardiac imaging modalities in pericardial pathologies. This contemporary review provides an update regarding the applications of multi-modality cardiac imaging in the evaluation of pericardial syndromes including acute/recurrent pericarditis, effusion/tamponade, constriction, masses and congenital anomalies.

Development and first implementation of a novel multi-modality cardiac motion and dosimetry phantom for radiotherapy applications.

BACKGROUND: Cardiac applications in radiation therapy are rapidly expanding including magnetic resonance guided radiation therapy (MRgRT) for real-time gating for targeting and avoidance near the heart or treating ventricular tachycardia (VT). PURPOSE: This work describes the development and implementation of a novel multi-modality and magnetic resonance (MR)-compatible cardiac phantom. METHODS: The patient-informed 3D model was derived from manual contouring of a contrast-enhanced Coronary Computed Tomography Angiography scan, exported as a Stereolithography model, then post-processed to simulate female heart with an average volume. The model was 3D-printed using Elastic50A to provide MR contrast to water background. Two rigid acrylic modules containing cardiac structures were designed and assembled, retrofitting to an MR-safe programmable motor to supply cardiac and respiratory motion in superior-inferior directions. One module contained a cavity for an ion chamber (IC), and the other was equipped with multiple interchangeable cavities for plastic scintillation detectors (PSDs). Images were acquired on a 0.35 T MR-linac for validation of phantom geometry, motion, and simulated online treatment planning and delivery. Three motion profiles were prescribed: patient-derived cardiac (sine waveform, 4.3 mm peak-to-peak, 60 beats/min), respiratory (cos4 waveform, 30 mm peak-to-peak, 12 breaths/min), and a superposition of cardiac (sine waveform, 4 mm peak-to-peak, 70 beats/min) and respiratory (cos4 waveform, 24 mm peak-to-peak, 12 breaths/min). The amplitude of the motion profiles was evaluated from sagittal cine images at eight frames/s with a resolution of 2.4 mm × 2.4 mm. Gated dosimetry experiments were performed using the two module configurations for calculating dose relative to stationary. A CT-based VT treatment plan was delivered twice under cone-beam CT guidance and cumulative stationary doses to multi-point PSDs were evaluated. RESULTS: No artifacts were observed on any images acquired during phantom operation. Phantom excursions measured 49.3 ± 25.8%/66.9 ± 14.0%, 97.0 ± 2.2%/96.4 ± 1.7%, and 90.4 ± 4.8%/89.3 ± 3.5% of prescription for cardiac, respiratory, and cardio-respiratory motion profiles for the 2-chamber (PSD) and 12-substructure (IC) phantom modules respectively. In the gated experiments, the cumulative dose was <2% from expected using the IC module. Real-time dose measured for the PSDs at 10 Hz acquisition rate demonstrated the ability to detect the dosimetric consequences of cardiac, respiratory, and cardio-respiratory motion when sampling of different locations during a single delivery, and the stability of our phantom dosimetric results over repeated cycles for the high dose and high gradient regions. For the VT delivery, high dose PSD was <1% from expected (5-6 cGy deviation of 5.9 Gy/fraction) and high gradient/low dose regions had deviations <3.6% (6.3 cGy less than expected 1.73 Gy/fraction). CONCLUSIONS: A novel multi-modality modular heart phantom was designed, constructed, and used for gated radiotherapy experiments on a 0.35 T MR-linac. Our phantom was capable of mimicking cardiac, cardio-respiratory, and respiratory motion while performing dosimetric evaluations of gated procedures using IC and PSD configurations. Time-resolved PSDs with small sensitive volumes appear promising for low-amplitude/high-frequency motion and multi-point data acquisition for advanced dosimetric capabilities. Illustrating VT planning and delivery further expands our phantom to address the unmet needs of cardiac applications in radiotherapy.

The Radiographic Appearances of Bilateral Ureteroceles and Their Management in an Adult Patient Presenting With Recurrent Urinary Tract Infections.

A ureterocele is a congenital abnormality in which there is abnormal dilatation of the distalmost portion of the ureter, as it enters the urinary bladder. Patients present with frequent urinary tract infections, urinary retention, cyclical abdominal pains, failure to thrive, and hematuria. Ureteroceles are often diagnosed on antenatal ultrasound and sometimes postnatally on ultrasounds done in the setting of a urinary tract infection. This case describes a 51-year-old female who presented with recurrent urinary tract infections. Subsequent imaging with ultrasound, intravenous urogram, and computed tomography demonstrated features typical for bilateral ureteroceles.

3D-MRI super-resolution reconstruction using multi-modality based on multi-resolution CNN.

BACKGROUND AND OBJECTIVE: High-resolution (HR) MR images provide rich structural detail to assist physicians in clinical diagnosis and treatment plan. However, it is arduous to acquire HR MRI due to equipment limitations, scanning time or patient comfort. Instead, HR MRI could be obtained through a number of computer assisted post-processing methods that have proven to be effective and reliable. This paper aims to develop a convolutional neural network (CNN) based super-resolution reconstruction framework for low-resolution (LR) T2w images. METHOD: In this paper, we propose a novel multi-modal HR MRI generation framework based on deep learning techniques. Specifically, we construct a CNN based on multi-resolution analysis to learn an end-to-end mapping between LR T2w and HR T2w, where HR T1w is fed into the network to offer detailed a priori information to help generate HR T2w. Furthermore, a low-frequency filtering module is introduced to filter out the interference from HR-T1w during high-frequency information extraction. Based on the idea of multi-resolution analysis, detailed features extracted from HR T1w and LR T2w are fused at two scales in the network and then HR T2w is reconstructed by upsampling and dense connectivity module. RESULTS: Extensive quantitative and qualitative evaluations demonstrate that the proposed method enhances the recovered HR T2w details and outperforms other state-of-the-art methods. In addition, the experimental results also suggest that our network has a lightweight structure and favorable generalization performance. CONCLUSION: The results show that the proposed method is capable of reconstructing HR T2w with higher accuracy. Meanwhile, the super-resolution reconstruction results on other dataset illustrate the excellent generalization ability of the method.

GIT-Mol: A multi-modal large language model for molecular science with graph, image, and text.

Large language models have made significant strides in natural language processing, enabling innovative applications in molecular science by processing textual representations of molecules. However, most existing language models cannot capture the rich information with complex molecular structures or images. In this paper, we introduce GIT-Mol, a multi-modal large language model that integrates the Graph, Image, and Text information. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space. We achieve a 5%-10% accuracy increase in properties prediction and a 20.2% boost in molecule generation validity compared to the baselines. With the any-to-language molecular translation strategy, our model has the potential to perform more downstream tasks, such as compound name recognition and chemical reaction prediction.

Automatic segmentation of medial temporal lobe subregions in multi-scanner, multi-modality MRI of variable quality.

Background: Volumetry of subregions in the medial temporal lobe (MTL) computed from automatic segmentation in MRI can track neurodegeneration in Alzheimer's disease. However, image quality may vary in MRI. Poor quality MR images can lead to unreliable segmentation of MTL subregions. Considering that different MRI contrast mechanisms and field strengths (jointly referred to as "modalities" here) offer distinct advantages in imaging different parts of the MTL, we developed a muti-modality segmentation model using both 7 tesla (7T) and 3 tesla (3T) structural MRI to obtain robust segmentation in poor-quality images. Method: MRI modalities including 3T T1-weighted, 3T T2-weighted, 7T T1-weighted and 7T T2-weighted (7T-T2w) of 197 participants were collected from a longitudinal aging study at the Penn Alzheimer's Disease Research Center. Among them, 7T-T2w was used as the primary modality, and all other modalities were rigidly registered to the 7T-T2w. A model derived from nnU-Net took these registered modalities as input and outputted subregion segmentation in 7T-T2w space. 7T-T2w images most of which had high quality from 25 selected training participants were manually segmented to train the multi-modality model. Modality augmentation, which randomly replaced certain modalities with Gaussian noise, was applied during training to guide the model to extract information from all modalities. To compare our proposed model with a baseline single-modality model in the full dataset with mixed high/poor image quality, we evaluated the ability of derived volume/thickness measures to discriminate Amyloid+ mild cognitive impairment (A+MCI) and Amyloid- cognitively unimpaired (A-CU) groups, as well as the stability of these measurements in longitudinal data. Results: The multi-modality model delivered good performance regardless of 7T-T2w quality, while the single-modality model under-segmented subregions in poor-quality images. The multi-modality model generally demonstrated stronger discrimination of A+MCI versus A-CU. Intra-class correlation and Bland-Altman plots demonstrate that the multi-modality model had higher longitudinal segmentation consistency in all subregions while the single-modality model had low consistency in poor-quality images. Conclusion: The multi-modality MRI segmentation model provides an improved biomarker for neurodegeneration in the MTL that is robust to image quality. It also provides a framework for other studies which may benefit from multimodal imaging.

Advancing surgical VQA with scene graph knowledge.

PURPOSE: The modern operating room is becoming increasingly complex, requiring innovative intra-operative support systems. While the focus of surgical data science has largely been on video analysis, integrating surgical computer vision with natural language capabilities is emerging as a necessity. Our work aims to advance visual question answering (VQA) in the surgical context with scene graph knowledge, addressing two main challenges in the current surgical VQA systems: removing question-condition bias in the surgical VQA dataset and incorporating scene-aware reasoning in the surgical VQA model design. METHODS: First, we propose a surgical scene graph-based dataset, SSG-VQA, generated by employing segmentation and detection models on publicly available datasets. We build surgical scene graphs using spatial and action information of instruments and anatomies. These graphs are fed into a question engine, generating diverse QA pairs. We then propose SSG-VQA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module, which integrates geometric scene knowledge in the VQA model design by employing cross-attention between the textual and the scene features. RESULTS: Our comprehensive analysis shows that our SSG-VQA dataset provides a more complex, diverse, geometrically grounded, unbiased and surgical action-oriented dataset compared to existing surgical VQA datasets and SSG-VQA-Net outperforms existing methods across different question types and complexities. We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex queries. CONCLUSION: We present a novel surgical VQA dataset and model and show that results can be significantly improved by incorporating geometric scene features in the VQA model design. We point out that the bottleneck of the current surgical visual question-answer model lies in learning the encoded representation rather than decoding the sequence. Our SSG-VQA dataset provides a diagnostic benchmark to test the scene understanding and reasoning capabilities of the model. The source code and the dataset will be made publicly available at: https://github.com/CAMMA-public/SSG-VQA .

Assessing the Knowledge, Attitude, and Practice of Orthopedicians for Pain Management by Multimodal Approach: A Prospective, Cross-Sectional, and Observational Survey.

BACKGROUND: The routine use of multimodal analgesic modality results in lower pain scores with minimum side effects and opioid utilization. MATERIALS AND METHODS:  A prospective, cross-sectional, observational study was conducted among orthopedicians practicing across India to assess the professional opinions on using analgesics to manage orthopedic pain effectively. RESULTS:  A total of 530 orthopedicians participated in this survey. Over 50% of the participants responded that tramadol with or without paracetamol was the choice of therapy for acute pain. Nearly 50% of the participants mentioned that multimodal interventions can sometimes help to manage pain. A total of 55.6% of participants mentioned that using Non-steroidal anti-inflammatory drugs was the most common in their clinical practice, while 25.7% of participants mentioned that they used tramadol more commonly in their clinical practice. As per clinical efficacy ranking, the combination of tramadol plus paracetamol (44.3%) was ranked first among analgesic combinations, followed by aceclofenac plus paracetamol (40.0%). The severity of pain (62.6%) followed by age (60.6%) and duration of therapy (52.6%) were the most common factors that should be considered while prescribing tramadol plus paracetamol combination. Gastrointestinal and renal are reported as the most common safety concerns encountered with analgesics. CONCLUSION:  The combination of tramadol and paracetamol was identified as the most preferred choice of analgesics for prolonged orthopedic pain management.

Confidence-aware multi-modality learning for eye disease screening.

Multi-modal ophthalmic image classification plays a key role in diagnosing eye diseases, as it integrates information from different sources to complement their respective performances. However, recent improvements have mainly focused on accuracy, often neglecting the importance of confidence and robustness in predictions for diverse modalities. In this study, we propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information using a multi-distribution fusion perspective. Specifically, our method first utilizes normal inverse gamma prior distributions over pre-trained models to learn both aleatoric and epistemic uncertainty for uni-modality. Then, the normal inverse gamma distribution is analyzed as the Student's t distribution. Furthermore, within a confidence-aware fusion framework, we propose a mixture of Student's t distributions to effectively integrate different modalities, imparting the model with heavy-tailed properties and enhancing its robustness and reliability. More importantly, the confidence-aware multi-modality ranking regularization term induces the model to more reasonably rank the noisy single-modal and fused-modal confidence, leading to improved reliability and accuracy. Experimental results on both public and internal datasets demonstrate that our model excels in robustness, particularly in challenging scenarios involving Gaussian noise and modality missing conditions. Moreover, our model exhibits strong generalization capabilities to out-of-distribution data, underscoring its potential as a promising solution for multimodal eye disease screening.

The challenging approach of a young patient with a primary intimal sarcoma of the heart: a case report.

Background: Primary intimal sarcomas of the heart are extremely rare and have a dismal prognosis. Their management represents a complex clinical challenge since complete surgical resection is the only reliable possibility of cure but is only possible in 50% of patients. In non-resectable disease, anthracycline-based therapy is the most effective treatment, but pazopanib may be used in patients unfit to receive anthracyclines. Case summary: A 38-year-old man presented with acute right heart failure symptoms due to a primary intimal sarcoma of the heart. A definite diagnosis was made after cardiac surgery. Multi-modality cardiac imaging showed early recurrence of disease with mitral valve and pulmonary veins' invasion, and the patient was deemed inoperable. Due to chronic kidney disease and previous heart failure symptoms, he was started on first-line pazopanib palliative treatment. After 11 months of chemotherapy, there was good clinical tolerance and no evidence of disease progression, which occurred after 13 months. Discussion: This case highlights the value of a multi-modality imaging approach for cardiac masses. Most importantly, it reports the successful treatment of a young patient with a primary intimal sarcoma of the heart who was started on palliative pazopanib, with a significantly higher progression-free survival than is reported in the literature. This finding may support pazopanib as a good alternative as first-line treatment when there is contraindication for anthracycline-based chemotherapy.

Evidence modeling for reliability learning and interpretable decision-making under multi-modality medical image segmentation.

Reliability learning and interpretable decision-making are crucial for multi-modality medical image segmentation. Although many works have attempted multi-modality medical image segmentation, they rarely explore how much reliability is provided by each modality for segmentation. Moreover, the existing approach of decision-making such as the softmax function lacks the interpretability for multi-modality fusion. In this study, we proposed a novel approach named contextual discounted evidential network (CDE-Net) for reliability learning and interpretable decision-making under multi-modality medical image segmentation. Specifically, the CDE-Net first models the semantic evidence by uncertainty measurement using the proposed evidential decision-making module. Then, it leverages the contextual discounted fusion layer to learn the reliability provided by each modality. Finally, a multi-level loss function is deployed for the optimization of evidence modeling and reliability learning. Moreover, this study elaborates on the framework interpretability by discussing the consistency between pixel attribution maps and the learned reliability coefficients. Extensive experiments are conducted on both multi-modality brain and liver datasets. The CDE-Net gains high performance with an average Dice score of 0.914 for brain tumor segmentation and 0.913 for liver tumor segmentation, which proves CDE-Net has great potential to facilitate the interpretation of artificial intelligence-based multi-modality medical image fusion.

Prediction model of early recurrence of multimodal hepatocellular carcinoma with tensor fusion.

Objective.In oncology, clinical decision-making relies on a multitude of data modalities, including histopathological, radiological, and clinical factors. Despite the emergence of computer-aided multimodal decision-making systems for predicting hepatocellular carcinoma (HCC) recurrence post-hepatectomy, existing models often employ simplistic feature-level concatenation, leading to redundancy and suboptimal performance. Moreover, these models frequently lack effective integration with clinically relevant data and encounter challenges in integrating diverse scales and dimensions, as well as incorporating the liver background, which holds clinical significance but has been previously overlooked.Approach.To address these limitations, we propose two approaches. Firstly, we introduce the tensor fusion method to our model, which offers distinct advantages in handling multi-scale and multi-dimensional data fusion, potentially enhancing overall performance. Secondly, we pioneer the consideration of the liver background's impact, integrating it into the feature extraction process using a deep learning segmentation-based algorithm. This innovative inclusion aligns the model more closely with real-world clinical scenarios, as the liver background may contain crucial information related to postoperative recurrence.Main results.We collected radiomics (MRI) and histopathological images from 176 cases diagnosed by experienced clinicians across two independent centers. Our proposed network underwent training and 5-fold cross-validation on this dataset before validation on an external test dataset comprising 40 cases. Ultimately, our model demonstrated outstanding performance in predicting early recurrence of HCC postoperatively, achieving an AUC of 0.883.Significance.These findings signify significant progress in addressing challenges related to multimodal data fusion and hold promise for more accurate clinical outcome predictions. In this study, we exploited global 3D liver background into modelling which is crucial to to the prognosis assessment and analyzed the whole liver background in addition to the tumor region. Both MRI images and histopathological images of HCC were fused at high-dimensional feature space using tensor techniques to solve cross-scale data integration issue.