Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Information extraction from weakly structured radiological reports with natural language queries.

OBJECTIVES: Provide physicians and researchers an efficient way to extract information from weakly structured radiology reports with natural language processing (NLP) machine learning models. METHODS: We evaluate seven different German bidirectional encoder representations from transformers (BERT) models on a dataset of 857,783 unlabeled radiology reports and an annotated reading comprehension dataset in the format of SQuAD 2.0 based on 1223 additional reports. RESULTS: Continued pre-training of a BERT model on the radiology dataset and a medical online encyclopedia resulted in the most accurate model with an F1-score of 83.97% and an exact match score of 71.63% for answerable questions and 96.01% accuracy in detecting unanswerable questions. Fine-tuning a non-medical model without further pre-training led to the lowest-performing model. The final model proved stable against variation in the formulations of questions and in dealing with questions on topics excluded from the training set. CONCLUSIONS: General domain BERT models further pre-trained on radiological data achieve high accuracy in answering questions on radiology reports. We propose to integrate our approach into the workflow of medical practitioners and researchers to extract information from radiology reports. CLINICAL RELEVANCE STATEMENT: By reducing the need for manual searches of radiology reports, radiologists' resources are freed up, which indirectly benefits patients. KEY POINTS: • BERT models pre-trained on general domain datasets and radiology reports achieve high accuracy (83.97% F1-score) on question-answering for radiology reports. • The best performing model achieves an F1-score of 83.97% for answerable questions and 96.01% accuracy for questions without an answer. • Additional radiology-specific pretraining of all investigated BERT models improves their performance.

Back to the Roots: Reconstructing Large and Complex Cranial Defects using an Image-based Statistical Shape Model.

Designing implants for large and complex cranial defects is a challenging task, even for professional designers. Current efforts on automating the design process focused mainly on convolutional neural networks (CNN), which have produced state-of-the-art results on reconstructing synthetic defects. However, existing CNN-based methods have been difficult to translate to clinical practice in cranioplasty, as their performance on large and complex cranial defects remains unsatisfactory. In this paper, we present a statistical shape model (SSM) built directly on the segmentation masks of the skulls represented as binary voxel occupancy grids and evaluate it on several cranial implant design datasets. Results show that, while CNN-based approaches outperform the SSM on synthetic defects, they are inferior to SSM when it comes to large, complex and real-world defects. Experienced neurosurgeons evaluate the implants generated by the SSM to be feasible for clinical use after minor manual corrections. Datasets and the SSM model are publicly available at https://github.com/Jianningli/ssm .

Evaluation of thresholding methods for the quantification of [68Ga]Ga-PSMA-11 PET molecular tumor volume and their effect on survival prediction in patients with advanced prostate cancer undergoing [177Lu]Lu-PSMA-617 radioligand therapy.

PURPOSE: The aim of this study was to systematically evaluate the effect of thresholding algorithms used in computer vision for the quantification of prostate-specific membrane antigen positron emission tomography (PET) derived tumor volume (PSMA-TV) in patients with advanced prostate cancer. The results were validated with respect to the prognostication of overall survival in patients with advanced-stage prostate cancer. MATERIALS AND METHODS: A total of 78 patients who underwent [177Lu]Lu-PSMA-617 radionuclide therapy from January 2018 to December 2020 were retrospectively included in this study. [68Ga]Ga-PSMA-11 PET images, acquired prior to radionuclide therapy, were used for the analysis of thresholding algorithms. All PET images were first analyzed semi-automatically using a pre-evaluated, proprietary software solution as the baseline method. Subsequently, five histogram-based thresholding methods and two local adaptive thresholding methods that are well established in computer vision were applied to quantify molecular tumor volume. The resulting whole-body molecular tumor volumes were validated with respect to the prognostication of overall patient survival as well as their statistical correlation to the baseline methods and their performance on standardized phantom scans. RESULTS: The whole-body PSMA-TVs, quantified using different thresholding methods, demonstrate a high positive correlation with the baseline methods. We observed the highest correlation with generalized histogram thresholding (GHT) (Pearson r (r), p value (p): r = 0.977, p < 0.001) and Sauvola thresholding (r = 0.974, p < 0.001) and the lowest correlation with Multiotsu (r = 0.877, p < 0.001) and Yen thresholding methods (r = 0.878, p < 0.001). The median survival time of all patients was 9.87 months (95% CI [9.3 to 10.13]). Stratification by median whole-body PSMA-TV resulted in a median survival time from 11.8 to 13.5 months for the patient group with lower tumor burden and 6.5 to 6.6 months for the patient group with higher tumor burden. The patient group with lower tumor burden had significantly higher probability of survival (p < 0.00625) in eight out of nine thresholding methods (Fig. 2); those methods were SUVmax50 (p = 0.0038), SUV ≥3 (p = 0.0034), Multiotsu (p = 0.0015), Yen (p = 0.0015), Niblack (p = 0.001), Sauvola (p = 0.0001), Otsu (p = 0.0053), and Li thresholding (p = 0.0053). CONCLUSION: Thresholding methods commonly used in computer vision are promising tools for the semiautomatic quantification of whole-body PSMA-TV in [68Ga]Ga-PSMA-11-PET. The proposed algorithm-driven thresholding strategy is less arbitrary and less prone to biases than thresholding with predefined values, potentially improving the application of whole-body PSMA-TV as an imaging biomarker.

Low bone mineral density is a prognostic factor for elderly patients with HCC undergoing TACE: results from a multicenter study.

OBJECTIVES: Low bone mineral density (BMD) was recently identified as a novel risk factor for patients with hepatocellular carcinoma (HCC). In this multicenter study, we aimed to validate the role of BMD as a prognostic factor for patients with HCC undergoing transarterial chemoembolization (TACE). METHODS: This retrospective multicenter trial included 908 treatment-naïve patients with HCC who were undergoing TACE as a first-line treatment, at six tertiary care centers, between 2010 and 2020. BMD was assessed by measuring the mean Hounsfield units (HUs) in the midvertebral core of the 11th thoracic vertebra, on contrast-enhanced computer tomography performed before treatment. We assessed the influence of BMD on median overall survival (OS) and performed multivariate analysis including established estimates for survival. RESULTS: The median BMD was 145 HU (IQR, 115-175 HU). Patients with a high BMD (≥ 114 HU) had a median OS of 22.2 months, while patients with a low BMD (< 114 HU) had a lower median OS of only 16.2 months (p < .001). Besides albumin, bilirubin, tumor number, and tumor diameter, BMD remained an independent prognostic factor in multivariate analysis. CONCLUSIONS: BMD is an independent predictive factor for survival in elderly patients with HCC undergoing TACE. The integration of BMD into novel scoring systems could potentially improve survival prediction and clinical decision-making. KEY POINTS: • Bone mineral density can be easily assessed in routinely acquired pre-interventional computed tomography scans. • Bone mineral density is an independent predictive factor for survival in elderly patients with HCC undergoing TACE. • Thus, bone mineral density is a novel imaging biomarker for prognosis prediction in elderly patients with HCC undergoing TACE.

"A net for everyone": fully personalized and unsupervised neural networks trained with longitudinal data from a single patient.

BACKGROUND: With the rise in importance of personalized medicine and deep learning, we combine the two to create personalized neural networks. The aim of the study is to show a proof of concept that data from just one patient can be used to train deep neural networks to detect tumor progression in longitudinal datasets. METHODS: Two datasets with 64 scans from 32 patients with glioblastoma multiforme (GBM) were evaluated in this study. The contrast-enhanced T1w sequences of brain magnetic resonance imaging (MRI) images were used. We trained a neural network for each patient using just two scans from different timepoints to map the difference between the images. The change in tumor volume can be calculated with this map. The neural networks were a form of a Wasserstein-GAN (generative adversarial network), an unsupervised learning architecture. The combination of data augmentation and the network architecture allowed us to skip the co-registration of the images. Furthermore, no additional training data, pre-training of the networks or any (manual) annotations are necessary. RESULTS: The model achieved an AUC-score of 0.87 for tumor change. We also introduced a modified RANO criteria, for which an accuracy of 66% can be achieved. CONCLUSIONS: We show a novel approach to deep learning in using data from just one patient to train deep neural networks to monitor tumor change. Using two different datasets to evaluate the results shows the potential to generalize the method.

CT Angiography Clot Burden Score from Data Mining of Structured Reports for Pulmonary Embolism.

Background Many studies emphasize the role of structured reports (SRs) because they are readily accessible for further automated analyses. However, using SR data obtained in clinical routine for research purposes is not yet well represented in literature. Purpose To compare the performance of the Qanadli scoring system with a clot burden score mined from structured pulmonary embolism (PE) reports from CT angiography. Materials and Methods In this retrospective study, a rule-based text mining pipeline was developed to extract descriptors of PE and right heart strain from SR of patients with suspected PE between March 2017 and February 2020. From standardized PE reporting, a pulmonary artery obstruction index (PAOI) clot burden score (PAOICBS) was derived and compared with the Qanadli score (PAOIQ). Scoring time and confidence from two independent readings were compared. Interobserver and interscore agreement was tested by using the intraclass correlation coefficient (ICC) and Bland-Altman analysis. To assess conformity and diagnostic performance of both scores, areas under the receiver operating characteristic curve (AUCs) were calculated to predict right heart strain incidence, as were optimal cutoff values for maximum sensitivity and specificity. Results SR content authored by 67 residents and signed off by 32 consultants from 1248 patients (mean age, 63 years ± 17 [standard deviation]; 639 men) was extracted accurately and allowed for PAOICBS calculation in 304 of 357 (85.2%) PE-positive reports. The PAOICBS strongly correlated with the PAOIQ (r = 0.94; P < .001). Use of PAOICBS yielded overall time savings (1.3 minutes ± 0.5 vs 3.0 minutes ± 1.7), higher confidence levels (4.2 ± 0.6 vs 3.6 ± 1.0), and a higher ICC (ICC, 0.99 vs 0.95), respectively, compared with PAOIQ (each, P < .001). AUCs were similar for PAOICBS (AUC, 0.75; 95% CI: 0.70, 0.81) and PAOIQ (AUC, 0.77; 95% CI: 0.72, 0.83; P = .68), with cutoff values of 27.5% for both scores. Conclusion Data mining of structured reports enabled the development of a CT angiography scoring system that simplified the Qanadli score as a semiquantitative estimate of thrombus burden in patients with pulmonary embolism. © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Hunsaker in this issue.

Artificial intelligence guided enhancement of digital PET: scans as fast as CT?

PURPOSE: Both digital positron emission tomography (PET) detector technologies and artificial intelligence based image post-reconstruction methods allow to reduce the PET acquisition time while maintaining diagnostic quality. The aim of this study was to acquire ultra-low-count fluorodeoxyglucose (FDG) ExtremePET images on a digital PET/computed tomography (CT) scanner at an acquisition time comparable to a CT scan and to generate synthetic full-dose PET images using an artificial neural network. METHODS: This is a prospective, single-arm, single-center phase I/II imaging study. A total of 587 patients were included. For each patient, a standard and an ultra-low-count FDG PET/CT scan (whole-body acquisition time about 30 s) were acquired. A modified pix2pixHD deep-learning network was trained employing 387 data sets as training and 200 as test cohort. Three models (PET-only and PET/CT with or without group convolution) were compared. Detectability and quantification were evaluated. RESULTS: The PET/CT input model with group convolution performed best regarding lesion signal recovery and was selected for detailed evaluation. Synthetic PET images were of high visual image quality; mean absolute lesion SUVmax (maximum standardized uptake value) difference was 1.5. Patient-based sensitivity and specificity for lesion detection were 79% and 100%, respectively. Not-detected lesions were of lower tracer uptake and lesion volume. In a matched-pair comparison, patient-based (lesion-based) detection rate was 89% (78%) for PERCIST (PET response criteria in solid tumors)-measurable and 36% (22%) for non PERCIST-measurable lesions. CONCLUSION: Lesion detectability and lesion quantification were promising in the context of extremely fast acquisition times. Possible application scenarios might include re-staging of late-stage cancer patients, in whom assessment of total tumor burden can be of higher relevance than detailed evaluation of small and low-uptake lesions.

Deep Learning-driven classification of external DICOM studies for PACS archiving.

OBJECTIVES: Over the course of their treatment, patients often switch hospitals, requiring staff at the new hospital to import external imaging studies to their local database. In this study, the authors present MOdality Mapping and Orchestration (MOMO), a Deep Learning-based approach to automate this mapping process by combining metadata analysis and a neural network ensemble. METHODS: A set of 11,934 imaging series with existing anatomical labels was retrieved from the PACS database of the local hospital to train an ensemble of neural networks (DenseNet-161 and ResNet-152), which process radiological images and predict the type of study they belong to. We developed an algorithm that automatically extracts relevant metadata from imaging studies, regardless of their structure, and combines it with the neural network ensemble, forming a powerful classifier. A set of 843 anonymized external studies from 321 hospitals was hand-labeled to assess performance. We tested several variations of this algorithm. RESULTS: MOMO achieves 92.71% accuracy and 2.63% minor errors (at 99.29% predictive power) on the external study classification task, outperforming both a commercial product (82.86% accuracy, 1.36% minor errors, 96.20% predictive power) and a pure neural network ensemble (72.69% accuracy, 10.3% minor errors, 99.05% predictive power) performing the same task. We find that the highest performance is achieved by an algorithm that combines all information into one vote-based classifier. CONCLUSION: Deep Learning combined with metadata matching is a promising and flexible approach for the automated classification of external DICOM studies for PACS archiving. KEY POINTS: • The algorithm can successfully identify 76 medical study types across seven modalities (CT, X-ray angiography, radiographs, MRI, PET (+CT/MRI), ultrasound, and mammograms). • The algorithm outperforms a commercial product performing the same task by a significant margin (> 9% accuracy gain). • The performance of the algorithm increases through the application of Deep Learning techniques.

Quantification and reduction of cross-vendor variation in multicenter DWI MR imaging: results of the Cancer Core Europe imaging task force.

OBJECTIVES: In the Cancer Core Europe Consortium (CCE), standardized biomarkers are required for therapy monitoring oncologic multicenter clinical trials. Multiparametric functional MRI and particularly diffusion-weighted MRI offer evident advantages for noninvasive characterization of tumor viability compared to CT and RECIST. A quantification of the inter- and intraindividual variation occurring in this setting using different hardware is missing. In this study, the MRI protocol including DWI was standardized and the residual variability of measurement parameters quantified. METHODS: Phantom and volunteer measurements (single-shot T2w and DW-EPI) were performed at the seven CCE sites using the MR hardware produced by three different vendors. Repeated measurements were performed at the sites and across the sites including a traveling volunteer, comparing qualitative and quantitative ROI-based results including an explorative radiomics analysis. RESULTS: For DWI/ADC phantom measurements using a central post-processing algorithm, the maximum deviation could be decreased to 2%. However, there is no significant difference compared to a decentralized ADC value calculation at the respective MRI devices. In volunteers, the measurement variation in 2 repeated scans did not exceed 11% for ADC and is below 20% for single-shot T2w in systematic liver ROIs. The measurement variation between sites amounted to 20% for ADC and < 25% for single-shot T2w. Explorative radiomics classification experiments yield better results for ADC than for single-shot T2w. CONCLUSION: Harmonization of MR acquisition and post-processing parameters results in acceptable standard deviations for MR/DW imaging. MRI could be the tool in oncologic multicenter trials to overcome the limitations of RECIST-based response evaluation. KEY POINTS: • Harmonizing acquisition parameters and post-processing homogenization, standardized protocols result in acceptable standard deviations for multicenter MR-DWI studies. • Total measurement variation does not to exceed 11% for ADC in repeated measurements in repeated MR acquisitions, and below 20% for an identical volunteer travelling between sites. • Radiomic classification experiments were able to identify stable features allowing for reliable discrimination of different physiological tissue samples, even when using heterogeneous imaging data.

Vesseg: An Open-Source Tool for Deep Learning-Based Atherosclerotic Plaque Quantification in Histopathology Images-Brief Report.

Objective: Manual plaque segmentation in microscopy images is a time-consuming process in atherosclerosis research and potentially subject to unacceptable user-to-user variability and observer bias. We address this by releasing Vesseg a tool that includes state-of-the-art deep learning models for atherosclerotic plaque segmentation. Approach and Results: Vesseg is a containerized, extensible, open-source, and user-oriented tool. It includes 2 models, trained and tested on 1089 hematoxylin-eosin stained mouse model atherosclerotic brachiocephalic artery sections. The models were compared to 3 human raters. Vesseg can be accessed at https://vesseg .online or downloaded. The models show mean Soerensen-Dice scores of 0.91+/-0.15 for plaque and 0.97+/-0.08 for lumen pixels. The mean accuracy is 0.98+/-0.05. Vesseg is already in active use, generating time savings of >10 minutes per slide. Conclusions: Vesseg brings state-of-the-art deep learning methods to atherosclerosis research, providing drastic time savings, while allowing for continuous improvement of models and the underlying pipeline.

PSMA PET total tumor volume predicts outcome of patients with advanced prostate cancer receiving [177Lu]Lu-PSMA-617 radioligand therapy in a bicentric analysis.

INTRODUCTION: [177Lu]Lu-PSMA-617 (Lu-PSMA) radioligand therapy is an emerging treatment option for patients with end-stage prostate cancer. However, response to Lu-PSMA therapy is only achieved in approximately half of patients. It is clinically important to identify patients at risk of poor outcome. Therefore, the aim of this study was to evaluate pretherapeutic PSMA PET derived total tumor volume and related metrics as prognosticators of overall survival in patients receiving Lu-PSMA therapy. METHODS: A total number of 110 patients form the Departments of Nuclear Medicine Münster and Essen were included in this retrospective analysis. Baseline PSMA PET-CT was available for all patients. Employing a previously published approach, all tumor lesions were semi-automatically delineated in PSMA PET-CT acquisitions. Total lesion number, total tumor volume (PSMA-TV), total lesion uptake (PSMA-TLU = PSMA-TV * SUVmean), and total lesion quotient (PSMA-TLQ = PSMA-TV / SUVmean) were quantified for each patient. Log2 transformation was used for regressions. RESULTS: Lesion number, PSMA-TV, and PSMA-TLQ were prognosticators of overall survival (HR = 1.255, p = 0.009; HR = 1.299, p = 0.005; HR = 1.326, p = 0.002). In a stepwise backward Cox regression including lesion number, PSMA-TV, PSA, LDH, and PSMA-TLQ, only the latter two remained independent and statistically significant negative prognosticators of overall survival (HR = 1.632, p = 0.011; HR = 1.239, p = 0.024). PSMA-TLQ and LDH were significant negative prognosticators in multivariate Cox regression in contrast to PSA value. CONCLUSION: PSMA-TV was a statistically significant negative prognosticator of overall survival in patients receiving Lu-PSMA therapy. PSMA-TLQ was an independent and superior prognosticator of overall survival compared with PSMA-TV.

[Artificial intelligence in oncological radiology : A (p)review]. / Künstliche Intelligenz in der onkologischen Radiologie : Ein (P)review.

BACKGROUND: Artificial intelligence (AI) has the potential to fundamentally change medicine within the coming decades. Radiological imaging is one of the primary fields of its clinical application. OBJECTIVES: In this article, we summarize previous AI developments with a focus on oncological radiology. Based on selected examples, we derive scenarios for developments in the next 10 years. MATERIALS AND METHODS: This work is based on a review of various literature and product databases, publications by regulatory authorities, reports, and press releases. CONCLUSIONS: The clinical use of AI applications is still in an early stage of development. The large number of research publications shows the potential of the field. Several certified products have already become available to users. However, for a widespread adoption of AI applications in clinical routine, several fundamental prerequisites are still awaited. These include the generation of evidence justifying the use of algorithms through representative clinical studies, adjustments to the framework for approval processes and dedicated education and teaching resources for its users. It is expected that use of AI methods will increase, thus, creating new opportunities for improved diagnostics, therapy, and more efficient workflows.

[Artificial intelligence in hybrid imaging]. / Künstliche Intelligenz in der Hybridbildgebung.

CLINICAL ISSUE: Hybrid imaging enables the precise visualization of cellular metabolism by combining anatomical and metabolic information. Advances in artificial intelligence (AI) offer new methods for processing and evaluating this data. METHODOLOGICAL INNOVATIONS: This review summarizes current developments and applications of AI methods in hybrid imaging. Applications in image processing as well as methods for disease-related evaluation are presented and discussed. MATERIALS AND METHODS: This article is based on a selective literature search with the search engines PubMed and arXiv. ASSESSMENT: Currently, there are only a few AI applications using hybrid imaging data and no applications are established in clinical routine yet. Although the first promising approaches are emerging, they still need to be evaluated prospectively. In the future, AI applications will support radiologists and nuclear medicine radiologists in diagnosis and therapy.

[A primer on machine learning]. / Wie funktioniert maschinelles Lernen?

BACKGROUND: The methods of machine learning and artificial intelligence are slowly but surely being introduced in everyday medical practice. In the future, they will support us in diagnosis and therapy and thus improve treatment for the benefit of the individual patient. It is therefore important to deal with this topic and to develop a basic understanding of it. OBJECTIVES: This article gives an overview of the exciting and dynamic field of machine learning and serves as an introduction to some methods primarily from the realm of supervised learning. In addition to definitions and simple examples, limitations are discussed. CONCLUSIONS: The basic principles behind the methods are simple. Nevertheless, due to their high dimensional nature, the factors influencing the results are often difficult or impossible to understand by humans. In order to build confidence in the new technologies and to guarantee their safe application, we need explainable algorithms and prospective effectiveness studies.

[A primer on radiomics]. / Wie funktioniert Radiomics?

CLINICAL ISSUE: The reproducible and exhaustive extraction of information from radiological images is a central task in the practice of radiology. Dynamic developments in the fields of artificial intelligence (AI) and machine learning are introducing new methods for this task. Radiomics is one such method and offers new opportunities and challenges for the future of radiology. METHODOLOGICAL INNOVATIONS: Radiomics describes the quantitative evaluation, interpretation, and clinical assessment of imaging markers in radiological data. Components of a radiomics analysis are data acquisition, data preprocessing, data management, segmentation of regions of interest, computation and selection of imaging markers, as well as the development of a radiomics model used for diagnosis and prognosis. This article explains these components and aims at providing an introduction to the field of radiomics while highlighting existing limitations. MATERIALS AND METHODS: This article is based on a selective literature search with the PubMed search engine. ASSESSMENT: Even though radiomics applications have yet to arrive in routine clinical practice, the quantification of radiological data in terms of radiomics is underway and will increase in the future. This holds the potential for lasting change in the discipline of radiology. Through the successful extraction and interpretation of all the information encoded in radiological images the next step in the direction of a more personalized, future-oriented form of medicine can be taken.

[Artificial intelligence and machine learning in oncologic imaging]. / Künstliche Intelligenz und maschinelles Lernen in der onkologischen Bildgebung.

Machine learning (ML) is entering many areas of society, including medicine. This transformation has the potential to drastically change medicine and medical practice. These aspects become particularly clear when considering the different stages of oncologic patient care and the involved interdisciplinary and intermodality interactions. In recent publications, computers-in collaboration with humans or alone-have been outperforming humans regarding tumor identification, tumor classification, estimating prognoses, and evaluation of treatments. In addition, ML algorithms, e.g., artificial neural networks (ANNs), which constitute the drivers behind many of the latest achievements in ML, can deliver this level of performance in a reproducible, fast, and inexpensive manner. In the future, artificial intelligence applications will become an integral part of the medical profession and offer advantages for oncologic diagnostics and treatment.

Deep MRI brain extraction: A 3D convolutional neural network for skull stripping.

Brain extraction from magnetic resonance imaging (MRI) is crucial for many neuroimaging workflows. Current methods demonstrate good results on non-enhanced T1-weighted images, but struggle when confronted with other modalities and pathologically altered tissue. In this paper we present a 3D convolutional deep learning architecture to address these shortcomings. In contrast to existing methods, we are not limited to non-enhanced T1w images. When trained appropriately, our approach handles an arbitrary number of modalities including contrast-enhanced scans. Its applicability to MRI data, comprising four channels: non-enhanced and contrast-enhanced T1w, T2w and FLAIR contrasts, is demonstrated on a challenging clinical data set containing brain tumors (N=53), where our approach significantly outperforms six commonly used tools with a mean Dice score of 95.19. Further, the proposed method at least matches state-of-the-art performance as demonstrated on three publicly available data sets: IBSR, LPBA40 and OASIS, totaling N=135 volumes. For the IBSR (96.32) and LPBA40 (96.96) data set the convolutional neuronal network (CNN) obtains the highest average Dice scores, albeit not being significantly different from the second best performing method. For the OASIS data the second best Dice (95.02) results are achieved, with no statistical difference in comparison to the best performing tool. For all data sets the highest average specificity measures are evaluated, whereas the sensitivity displays about average results. Adjusting the cut-off threshold for generating the binary masks from the CNN's probability output can be used to increase the sensitivity of the method. Of course, this comes at the cost of a decreased specificity and has to be decided application specific. Using an optimized GPU implementation predictions can be achieved in less than one minute. The proposed method may prove useful for large-scale studies and clinical trials.

Multisensory Extended Reality Applications Offer Benefits for Volumetric Biomedical Image Analysis in Research and Medicine.

3D data from high-resolution volumetric imaging is a central resource for diagnosis and treatment in modern medicine. While the fast development of AI enhances imaging and analysis, commonly used visualization methods lag far behind. Recent research used extended reality (XR) for perceiving 3D images with visual depth perception and touch but used restrictive haptic devices. While unrestricted touch benefits volumetric data examination, implementing natural haptic interaction with XR is challenging. The research question is whether a multisensory XR application with intuitive haptic interaction adds value and should be pursued. In a study, 24 experts for biomedical images in research and medicine explored 3D medical shapes with 3 applications: a multisensory virtual reality (VR) prototype using haptic gloves, a simple VR prototype using controllers, and a standard PC application. Results of standardized questionnaires showed no significant differences between all application types regarding usability and no significant difference between both VR applications regarding presence. Participants agreed to statements that VR visualizations provide better depth information, using the hands instead of controllers simplifies data exploration, the multisensory VR prototype allows intuitive data exploration, and it is beneficial over traditional data examination methods. While most participants mentioned manual interaction as the best aspect, they also found it the most improvable. We conclude that a multisensory XR application with improved manual interaction adds value for volumetric biomedical data examination. We will proceed with our open-source research project ISH3DE (Intuitive Stereoptic Haptic 3D Data Exploration) to serve medical education, therapeutic decisions, surgery preparations, or research data analysis.