Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging.

Explainable artificial intelligence (XAI) has experienced a vast increase in recognition over the last few years. While the technical developments are manifold, less focus has been placed on the clinical applicability and usability of systems. Moreover, not much attention has been given to XAI systems that can handle multimodal and longitudinal data, which we postulate are important features in many clinical workflows. In this study, we review, from a clinical perspective, the current state of XAI for multimodal and longitudinal datasets and highlight the challenges thereof. Additionally, we propose the XAI orchestrator, an instance that aims to help clinicians with the synopsis of multimodal and longitudinal data, the resulting AI predictions, and the corresponding explainability output. We propose several desirable properties of the XAI orchestrator, such as being adaptive, hierarchical, interactive, and uncertainty-aware.

Automated liver segmental volume ratio quantification on non-contrast T1-Vibe Dixon liver MRI using deep learning.

PURPOSE: To evaluate the effectiveness of automated liver segmental volume quantification and calculation of the liver segmental volume ratio (LSVR) on a non-contrast T1-vibe Dixon liver MRI sequence using a deep learning segmentation pipeline. METHOD: A dataset of 200 liver MRI with a non-contrast 3 mm T1-vibe Dixon sequence was manually labeledslice-by-sliceby an expert for Couinaud liver segments, while portal and hepatic veins were labeled separately. A convolutional neural networkwas trainedusing 170 liver MRI for training and 30 for evaluation. Liver segmental volumes without liver vessels were retrieved and LSVR was calculated as the liver segmental volumes I-III divided by the liver segmental volumes IV-VIII. LSVR was compared with the expert manual LSVR calculation and the LSVR calculated on CT scans in 30 patients with CT and MRI within 6 months. RESULTS: Theconvolutional neural networkclassified the Couinaud segments I-VIII with an average Dice score of 0.770 ± 0.03, ranging between 0.726 ± 0.13 (segment IVb) and 0.810 ± 0.09 (segment V). The calculated mean LSVR with liver MRI unseen by the model was 0.32 ± 0.14, as compared with manually quantified LSVR of 0.33 ± 0.15, resulting in a mean absolute error (MAE) of 0.02. A comparable LSVR of 0.35 ± 0.14 with a MAE of 0.04 resulted with the LSRV retrieved from the CT scans. The automated LSVR showed significant correlation with the manual MRI LSVR (Spearman r = 0.97, p < 0.001) and CT LSVR (Spearman r = 0.95, p < 0.001). CONCLUSIONS: A convolutional neural network allowed for accurate automated liver segmental volume quantification and calculation of LSVR based on a non-contrast T1-vibe Dixon sequence.

Evaluating automated longitudinal tumor measurements for glioblastoma response assessment.

Automated tumor segmentation tools for glioblastoma show promising performance. To apply these tools for automated response assessment, longitudinal segmentation, and tumor measurement, consistency is critical. This study aimed to determine whether BraTumIA and HD-GLIO are suited for this task. We evaluated two segmentation tools with respect to automated response assessment on the single-center retrospective LUMIERE dataset with 80 patients and a total of 502 post-operative time points. Volumetry and automated bi-dimensional measurements were compared with expert measurements following the Response Assessment in Neuro-Oncology (RANO) guidelines. The longitudinal trend agreement between the expert and methods was evaluated, and the RANO progression thresholds were tested against the expert-derived time-to-progression (TTP). The TTP and overall survival (OS) correlation was used to check the progression thresholds. We evaluated the automated detection and influence of non-measurable lesions. The tumor volume trend agreement calculated between segmentation volumes and the expert bi-dimensional measurements was high (HD-GLIO: 81.1%, BraTumIA: 79.7%). BraTumIA achieved the closest match to the expert TTP using the recommended RANO progression threshold. HD-GLIO-derived tumor volumes reached the highest correlation between TTP and OS (0.55). Both tools failed at an accurate lesion count across time. Manual false-positive removal and restricting to a maximum number of measurable lesions had no beneficial effect. Expert supervision and manual corrections are still necessary when applying the tested automated segmentation tools for automated response assessment. The longitudinal consistency of current segmentation tools needs further improvement. Validation of volumetric and bi-dimensional progression thresholds with multi-center studies is required to move toward volumetry-based response assessment.

A comprehensive dataset of annotated brain metastasis MR images with clinical and radiomic data.

Brain metastasis (BM) is one of the main complications of many cancers, and the most frequent malignancy of the central nervous system. Imaging studies of BMs are routinely used for diagnosis of disease, treatment planning and follow-up. Artificial Intelligence (AI) has great potential to provide automated tools to assist in the management of disease. However, AI methods require large datasets for training and validation, and to date there have been just one publicly available imaging dataset of 156 BMs. This paper publishes 637 high-resolution imaging studies of 75 patients harboring 260 BM lesions, and their respective clinical data. It also includes semi-automatic segmentations of 593 BMs, including pre- and post-treatment T1-weighted cases, and a set of morphological and radiomic features for the cases segmented. This data-sharing initiative is expected to enable research into and performance evaluation of automatic BM detection, lesion segmentation, disease status evaluation and treatment planning methods for BMs, as well as the development and validation of predictive and prognostic tools with clinical applicability.

PyRaDiSe: A Python package for DICOM-RT-based auto-segmentation pipeline construction and DICOM-RT data conversion.

BACKGROUND AND OBJECTIVE: Despite fast evolution cycles in deep learning methodologies for medical imaging in radiotherapy, auto-segmentation solutions rarely run in clinics due to the lack of open-source frameworks feasible for processing DICOM RT Structure Sets. Besides this shortage, available open-source DICOM RT Structure Set converters rely exclusively on 2D reconstruction approaches leading to pixelated contours with potentially low acceptance by healthcare professionals. PyRaDiSe, an open-source, deep learning framework independent Python package, addresses these issues by providing a framework for building auto-segmentation solutions feasible to operate directly on DICOM data. In addition, PyRaDiSe provides profound DICOM RT Structure Set conversion and processing capabilities; thus, it applies also to auto-segmentation-related tasks, such as dataset construction for deep learning model training. METHODS: The PyRaDiSe package follows a holistic approach and provides DICOM data handling, deep learning model inference, pre-processing, and post-processing functionalities. The DICOM data handling allows for highly automated and flexible handling of DICOM image series, DICOM RT Structure Sets, and DICOM registrations, including 2D-based and 3D-based conversion from and to DICOM RT Structure Sets. For deep learning model inference, extending given skeleton classes is straightforwardly achieved, allowing for employing any deep learning framework. Furthermore, a profound set of pre-processing and post-processing routines is included that incorporate partial invertibility for restoring spatial properties, such as image origin or orientation. RESULTS: The PyRaDiSe package, characterized by its flexibility and automated routines, allows for fast deployment and prototyping, reducing efforts for auto-segmentation pipeline implementation. Furthermore, while deep learning model inference is independent of the deep learning framework, it can easily be integrated into famous deep learning frameworks such as PyTorch or Tensorflow. The developed package has successfully demonstrated its capabilities in a research project at our institution for organs-at-risk segmentation in brain tumor patients. Furthermore, PyRaDiSe has shown its conversion performance for dataset construction. CONCLUSIONS: The PyRaDiSe package closes the gap between data science and clinical radiotherapy by enabling deep learning segmentation models to be easily transferred into clinical research practice. PyRaDiSe is available on https://github.com/ubern-mia/pyradise and can be installed directly from the Python Package Index using pip install pyradise.

The LUMIERE dataset: Longitudinal Glioblastoma MRI with expert RANO evaluation.

Publicly available Glioblastoma (GBM) datasets predominantly include pre-operative Magnetic Resonance Imaging (MRI) or contain few follow-up images for each patient. Access to fully longitudinal datasets is critical to advance the refinement of treatment response assessment. We release a single-center longitudinal GBM MRI dataset with expert ratings of selected follow-up studies according to the response assessment in neuro-oncology criteria (RANO). The expert rating includes details about the rationale of the ratings. For a subset of patients, we provide pathology information regarding methylation of the O6-methylguanine-DNA methyltransferase (MGMT) promoter status and isocitrate dehydrogenase 1 (IDH1), as well as the overall survival time. The data includes T1-weighted pre- and post-contrast, T2-weighted, and fluid-attenuated inversion recovery (FLAIR) MRI. Segmentations from state-of-the-art automated segmentation tools, as well as radiomic features, complement the data. Possible applications of this dataset are radiomics research, the development and validation of automated segmentation methods, and studies on response assessment. This collection includes MRI data of 91 GBM patients with a total of 638 study dates and 2487 images.

Convolutional neural network for automated segmentation of the liver and its vessels on non-contrast T1 vibe Dixon acquisitions.

We evaluated the effectiveness of automated segmentation of the liver and its vessels with a convolutional neural network on non-contrast T1 vibe Dixon acquisitions. A dataset of non-contrast T1 vibe Dixon liver magnetic resonance images was labelled slice-by-slice for the outer liver border, portal, and hepatic veins by an expert. A 3D U-Net convolutional neural network was trained with different combinations of Dixon in-phase, opposed-phase, water, and fat reconstructions. The neural network trained with the single-modal in-phase reconstructions achieved a high performance for liver parenchyma (Dice 0.936 ± 0.02), portal veins (0.634 ± 0.09), and hepatic veins (0.532 ± 0.12) segmentation. No benefit of using multi-modal input was observed (p = 1.0 for all experiments), combining in-phase, opposed-phase, fat, and water reconstruction. Accuracy for differentiation between portal and hepatic veins was 99% for portal veins and 97% for hepatic veins in the central region and slightly lower in the peripheral region (91% for portal veins, 80% for hepatic veins). In conclusion, deep learning-based automated segmentation of the liver and its vessels on non-contrast T1 vibe Dixon was highly effective. The single-modal in-phase input achieved the best performance in segmentation and differentiation between portal and hepatic veins.

Radiomics for glioblastoma survival analysis in pre-operative MRI: exploring feature robustness, class boundaries, and machine learning techniques.

BACKGROUND: This study aims to identify robust radiomic features for Magnetic Resonance Imaging (MRI), assess feature selection and machine learning methods for overall survival classification of Glioblastoma multiforme patients, and to robustify models trained on single-center data when applied to multi-center data. METHODS: Tumor regions were automatically segmented on MRI data, and 8327 radiomic features extracted from these regions. Single-center data was perturbed to assess radiomic feature robustness, with over 16 million tests of typical perturbations. Robust features were selected based on the Intraclass Correlation Coefficient to measure agreement across perturbations. Feature selectors and machine learning methods were compared to classify overall survival. Models trained on single-center data (63 patients) were tested on multi-center data (76 patients). Priors using feature robustness and clinical knowledge were evaluated. RESULTS: We observed a very large performance drop when applying models trained on single-center on unseen multi-center data, e.g. a decrease of the area under the receiver operating curve (AUC) of 0.56 for the overall survival classification boundary at 1 year. By using robust features alongside priors for two overall survival classes, the AUC drop could be reduced by 21.2%. In contrast, sensitivity was 12.19% lower when applying a prior. CONCLUSIONS: Our experiments show that it is possible to attain improved levels of robustness and accuracy when models need to be applied to unseen multi-center data. The performance on multi-center data of models trained on single-center data can be increased by using robust features and introducing prior knowledge. For successful model robustification, tailoring perturbations for robustness testing to the target dataset is key.

Brain Morphometry Estimation: From Hours to Seconds Using Deep Learning.

Motivation: Brain morphometry from magnetic resonance imaging (MRI) is a promising neuroimaging biomarker for the non-invasive diagnosis and monitoring of neurodegenerative and neurological disorders. Current tools for brain morphometry often come with a high computational burden, making them hard to use in clinical routine, where time is often an issue. We propose a deep learning-based approach to predict the volumes of anatomically delineated subcortical regions of interest (ROI), and mean thicknesses and curvatures of cortical parcellations directly from T1-weighted MRI. Advantages are the timely availability of results while maintaining a clinically relevant accuracy. Materials and Methods: An anonymized dataset of 574 subjects (443 healthy controls and 131 patients with epilepsy) was used for the supervised training of a convolutional neural network (CNN). A silver-standard ground truth was generated with FreeSurfer 6.0. Results: The CNN predicts a total of 165 morphometric measures directly from raw MR images. Analysis of the results using intraclass correlation coefficients showed, in general, good correlation with FreeSurfer generated ground truth data, with some of the regions nearly reaching human inter-rater performance (ICC > 0.75). Cortical thicknesses predicted by the CNN showed cross-sectional annual age-related gray matter atrophy rates both globally (thickness change of -0.004 mm/year) and regionally in agreement with the literature. A statistical test to dichotomize patients with epilepsy from healthy controls revealed similar effect sizes for structures affecting all subtypes as reported in a large-scale epilepsy study. Conclusions: We demonstrate the general feasibility of using deep learning to estimate human brain morphometry directly from T1-weighted MRI within seconds. A comparison of the results to other publications shows accuracies of comparable magnitudes for the subcortical volumes and cortical thicknesses.

Automated white matter fiber tract identification in patients with brain tumors.

We propose a method for the automated identification of key white matter fiber tracts for neurosurgical planning, and we apply the method in a retrospective study of 18 consecutive neurosurgical patients with brain tumors. Our method is designed to be relatively robust to challenges in neurosurgical tractography, which include peritumoral edema, displacement, and mass effect caused by mass lesions. The proposed method has two parts. First, we learn a data-driven white matter parcellation or fiber cluster atlas using groupwise registration and spectral clustering of multi-fiber tractography from healthy controls. Key fiber tract clusters are identified in the atlas. Next, patient-specific fiber tracts are automatically identified using tractography-based registration to the atlas and spectral embedding of patient tractography. Results indicate good generalization of the data-driven atlas to patients: 80% of the 800 fiber clusters were identified in all 18 patients, and 94% of the 800 fiber clusters were found in 16 or more of the 18 patients. Automated subject-specific tract identification was evaluated by quantitative comparison to subject-specific motor and language functional MRI, focusing on the arcuate fasciculus (language) and corticospinal tracts (motor), which were identified in all patients. Results indicate good colocalization: 89 of 95, or 94%, of patient-specific language and motor activations were intersected by the corresponding identified tract. All patient-specific activations were within 3mm of the corresponding language or motor tract. Overall, our results indicate the potential of an automated method for identifying fiber tracts of interest for neurosurgical planning, even in patients with mass lesions.

Effects of CPAP therapy withdrawal on exhaled breath pattern in obstructive sleep apnoea.

BACKGROUND: Obstructive sleep apnoea (OSA) is highly prevalent and associated with cardiovascular and metabolic changes. OSA is usually diagnosed by polysomnography which is time-consuming and provides little information on the patient's phenotype thus limiting a personalised treatment approach. Exhaled breath contains information on metabolism which can be analysed by mass spectrometry within minutes. The objective of this study was to identify a breath profile in OSA recurrence by use of secondary-electrospray-ionization-mass spectrometry (SESI-MS). METHODS: Patients with OSA effectively treated with CPAP were randomised to either withdraw treatment (subtherapeutic CPAP) or continue therapeutic CPAP for 2 weeks. Exhaled breath analysis by untargeted SESI-MS was performed at baseline and 2 weeks after randomisation. The primary outcome was the change in exhaled molecular breath pattern. RESULTS: 30 patients with OSA were randomised and 26 completed the trial according to the protocol. CPAP withdrawal led to a recurrence of OSA (mean difference in change of oxygen desaturation index between groups +30.3/h; 95% CI 19.8/h,40.7/h, p<0.001) which was accompanied by a significant change in 62 exhaled features (16 metabolites identified). The panel of discriminating mass-spectral features allowed differentiation between treated and untreated OSA with a sensitivity of 92.9% and a specificity of 84.6%. CONCLUSION: Exhaled breath analysis by SESI-MS allows rapid and accurate detection of OSA recurrence. The technique has the potential to characterise an individual's metabolic response to OSA and thus makes a comprehensible phenotyping of OSA possible. TRIAL REGISTRATION NUMBER: NCT02050425 (registered at ClinicalTrials.gov).

Differentiation of oral bacteria in in vitro cultures and human saliva by secondary electrospray ionization - mass spectrometry.

The detection of bacterial-specific volatile metabolites may be a valuable tool to predict infection. Here we applied a real-time mass spectrometric technique to investigate differences in volatile metabolic profiles of oral bacteria that cause periodontitis. We coupled a secondary electrospray ionization (SESI) source to a commercial high-resolution mass spectrometer to interrogate the headspace from bacterial cultures and human saliva. We identified 120 potential markers characteristic for periodontal pathogens Aggregatibacter actinomycetemcomitans (n = 13), Porphyromonas gingivalis (n = 70), Tanerella forsythia (n = 30) and Treponema denticola (n = 7) in in vitro cultures. In a second proof-of-principle phase, we found 18 (P. gingivalis, T. forsythia and T. denticola) of the 120 in vitro compounds in the saliva from a periodontitis patient with confirmed infection with P. gingivalis, T. forsythia and T. denticola with enhanced ion intensity compared to two healthy controls. In conclusion, this method has the ability to identify individual metabolites of microbial pathogens in a complex medium such as saliva.