Predicting the HER2 status in oesophageal cancer from tissue microarrays using convolutional neural networks.

BACKGROUND: Fast and accurate diagnostics are key for personalised medicine. Particularly in cancer, precise diagnosis is a prerequisite for targeted therapies, which can prolong lives. In this work, we focus on the automatic identification of gastroesophageal adenocarcinoma (GEA) patients that qualify for a personalised therapy targeting epidermal growth factor receptor 2 (HER2). We present a deep-learning method for scoring microscopy images of GEA for the presence of HER2 overexpression. METHODS: Our method is based on convolutional neural networks (CNNs) trained on a rich dataset of 1602 patient samples and tested on an independent set of 307 patient samples. We additionally verified the CNN's generalisation capabilities with an independent dataset with 653 samples from a separate clinical centre. We incorporated an attention mechanism in the network architecture to identify the tissue regions, which are important for the prediction outcome. Our solution allows for direct automated detection of HER2 in immunohistochemistry-stained tissue slides without the need for manual assessment and additional costly in situ hybridisation (ISH) tests. RESULTS: We show accuracy of 0.94, precision of 0.97, and recall of 0.95. Importantly, our approach offers accurate predictions in cases that pathologists cannot resolve and that require additional ISH testing. We confirmed our findings in an independent dataset collected in a different clinical centre. The attention-based CNN exploits morphological information in microscopy images and is superior to a predictive model based on the staining intensity only. CONCLUSIONS: We demonstrate that our approach not only automates an important diagnostic process for GEA patients but also paves the way for the discovery of new morphological features that were previously unknown for GEA pathology.

Auxiliary Tools for Enhanced Depth Perception in Vascular Structures.

This chapter discusses the concept of Auxiliary Tools in depth perception. Four recent techniques are considered, that apply the concept in the domain of liver vasculature visualization. While an improvement is evident, the evaluations and conducted studies are found to be biased and not general enough to provide a convincing assessment. The chapter provides background information about human visual perception and a brief history on vascular visualization. Then four state-of-the-art methods are discussed. Finally, a comparative discussion points out objectives for future follow-up work.

Perceptually Uniform Construction of Illustrative Textures.

Illustrative textures, such as stippling or hatching, were predominantly used as an alternative to conventional Phong rendering. Recently, the potential of encoding information on surfaces or maps using different densities has also been recognized. This has the significant advantage that additional color can be used as another visual channel and the illustrative textures can then be overlaid. Effectively, it is thus possible to display multiple information, such as two different scalar fields on surfaces simultaneously. In previous work, these textures were manually generated and the choice of density was unempirically determined. Here, we first want to determine and understand the perceptual space of illustrative textures. We chose a succession of simplices with increasing dimensions as primitives for our textures: Dots, lines, and triangles. Thus, we explore the texture types of stippling, hatching, and triangles. We create a range of textures by sampling the density space uniformly. Then, we conduct three perceptual studies in which the participants performed pairwise comparisons for each texture type. We use multidimensional scaling (MDS) to analyze the perceptual spaces per category. The perception of stippling and triangles seems relatively similar. Both are adequately described by a 1D manifold in 2D space. The perceptual space of hatching consists of two main clusters: Crosshatched textures, and textures with only one hatching direction. However, the perception of hatching textures with only one hatching direction is similar to the perception of stippling and triangles. Based on our findings, we construct perceptually uniform illustrative textures. Afterwards, we provide concrete application examples for the constructed textures.

Longitudinal wall shear stress evaluation using centerline projection approach in the numerical simulations of the patient-based carotid artery.

In this numerical study, areas of the carotid bifurcation and of a distal stenosis in the internal carotid artery are closely observed to evaluate the patient's current risks of ischemic stroke. An indicator for the vessel wall defects is the stress exerted by blood on the vessel tissue, typically expressed by the amplitude of the wall shear stress vector (WSS) and its oscillatory shear index. To detect negative shear stresses corresponding with reversal flow, we perform orientation-based shear evaluation. We investigate the longitudinal component of the wall shear vector, where tangential vectors aligned longitudinally with the vessel are necessary. However, resulting from imaging segmentation resolution of patients' computed tomography angiography scans and stenotic regions, the geometry model's mesh is non-smooth on its surface areas and the automatically generated tangential vector field is discontinuous and multi-directional, making an interpretation of our orientation-based risk indicators unreliable. We improve the evaluation of longitudinal shear stress by applying the projection of the vessel's centerline to the surface to construct smooth tangential field aligned longitudinally with the vessel. We validate our approach for the longitudinal WSS component and the corresponding oscillatory index by comparing them to results obtained using automatically generated tangents in both rigid and elastic vessel modeling and to amplitude-based indicators. We present the major benefit of our longitudinal WSS evaluation based on its directionality for the cardiovascular risk assessment, which is the detection of negative WSS indicating persistent reversal or transverse flow. This is impossible in the case of the amplitude-based WSS.

Perception of Line Attributes for Visualization.

Line attributes such as width and dashing are commonly used to encode information. However, many questions on the perception of line attributes remain, such as how many levels of attribute variation can be distinguished or which line attributes are the preferred choices for which tasks. We conducted three studies to develop guidelines for using stylized lines to encode scalar data. In our first study, participants drew stylized lines to encode uncertainty information. Uncertainty is usually visualized alongside other data. Therefore, alternative visual channels are important for the visualization of uncertainty. Additionally, uncertainty-e.g., in weather forecasts-is a familiar topic to most people. Thus, we picked it for our visualization scenarios in study 1. We used the results of our study to determine the most common line attributes for drawing uncertainty: Dashing, luminance, wave amplitude, and width. While those line attributes were especially common for drawing uncertainty, they are also commonly used in other areas. In studies 2 and 3, we investigated the discriminability of the line attributes determined in study 1. Studies 2 and 3 did not require specific application areas; thus, their results apply to visualizing any scalar data in line attributes. We evaluated the just-noticeable differences (JND) and derived recommendations for perceptually distinct line levels. We found that participants could discriminate considerably more levels for the line attribute width than for wave amplitude, dashing, or luminance.

Fabrication, characterization and numerical validation of a novel thin-wall hydrogel vessel model for cardiovascular research based on a patient-specific stenotic carotid artery bifurcation.

In vitro vascular models, primarily made of silicone, have been utilized for decades for studying hemodynamics and supporting the development of implants for catheter-based treatments of diseases such as stenoses and aneurysms. Hydrogels have emerged as prominent materials in tissue-engineering applications, offering distinct advantages over silicone models for fabricating vascular models owing to their viscoelasticity, low friction, and tunable mechanical properties. Our study evaluated the feasibility of fabricating thin-wall, anatomical vessel models made of polyvinyl alcohol hydrogel (PVA-H) based on a patient-specific carotid artery bifurcation using a combination of 3D printing and molding technologies. The model's geometry, elastic modulus, volumetric compliance, and diameter distensibility were characterized experimentally and numerically simulated. Moreover, a comparison with silicone models with the same anatomy was performed. A PVA-H vessel model was integrated into a mock circulatory loop for a preliminary ultrasound-based assessment of fluid dynamics. The vascular model's geometry was successfully replicated, and the elastic moduli amounted to 0.31 ± 0.007 MPa and 0.29 ± 0.007 MPa for PVA-H and silicone, respectively. Both materials exhibited nearly identical volumetric compliance (0.346 and 0.342% mmHg-1), which was higher compared to numerical simulation (0.248 and 0.290% mmHg-1). The diameter distensibility ranged from 0.09 to 0.20% mmHg-1 in the experiments and between 0.10 and 0.18% mmHg-1 in the numerical model at different positions along the vessel model, highlighting the influence of vessel geometry on local deformation. In conclusion, our study presents a method and provides insights into the manufacturing and mechanical characterization of hydrogel-based thin-wall vessel models, potentially allowing for a combination of fluid dynamics and tissue engineering studies in future cardio- and neurovascular research.

GRay: Ray Casting for Visualization and Interactive Data Exploration of Gaussian Mixture Models.

The Gaussian mixture model (GMM) describes the distribution of random variables from several different populations. GMMs have widespread applications in probability theory, statistics, machine learning for unsupervised cluster analysis and topic modeling, as well as in deep learning pipelines. So far, few efforts have been made to explore the underlying point distribution in combination with the GMMs, in particular when the data becomes high-dimensional and when the GMMs are composed of many Gaussians. We present an analysis tool comprising various GPU-based visualization techniques to explore such complex GMMs. To facilitate the exploration of high-dimensional data, we provide a novel navigation system to analyze the underlying data. Instead of projecting the data to 2D, we utilize interactive 3D views to better support users in understanding the spatial arrangements of the Gaussian distributions. The interactive system is composed of two parts: (1) raycasting-based views that visualize cluster memberships, spatial arrangements, and support the discovery of new modes. (2) overview visualizations that enable the comparison of Gaussians with each other, as well as small multiples of different choices of basis vectors. Users are supported in their exploration with customization tools and smooth camera navigations. Our tool was developed and assessed by five domain experts, and its usefulness was evaluated with 23 participants. To demonstrate the effectiveness, we identify interesting features in several data sets.

GUCCI - Guided Cardiac Cohort Investigation of Blood Flow Data.

We present the framework GUCCI (Guided Cardiac Cohort Investigation), which provides a guided visual analytics workflow to analyze cohort-based measured blood flow data in the aorta. In the past, many specialized techniques have been developed for the visual exploration of such data sets for a better understanding of the influence of morphological and hemodynamic conditions on cardiovascular diseases. However, there is a lack of dedicated techniques that allow visual comparison of multiple data sets and defined cohorts, which is essential to characterize pathologies. GUCCI offers visual analytics techniques and novel visualization methods to guide the user through the comparison of predefined cohorts, such as healthy volunteers and patients with a pathologically altered aorta. The combination of overview and glyph-based depictions together with statistical cohort-specific information allows investigating differences and similarities of the time-dependent data. Our framework was evaluated in a qualitative user study with three radiologists specialized in cardiac imaging and two experts in medical blood flow visualization. They were able to discover cohort-specific characteristics, which supports the derivation of standard values as well as the assessment of pathology-related severity and the need for treatment.

Visualization of Human Spine Biomechanics for Spinal Surgery.

We propose a visualization application, designed for the exploration of human spine simulation data. Our goal is to support research in biomechanical spine simulation and advance efforts to implement simulation-backed analysis in surgical applications. Biomechanical simulation is a state-of-the-art technique for analyzing load distributions of spinal structures. Through the inclusion of patient-specific data, such simulations may facilitate personalized treatment and customized surgical interventions. Difficulties in spine modelling and simulation can be partly attributed to poor result representation, which may also be a hindrance when introducing such techniques into a clinical environment. Comparisons of measurements across multiple similar anatomical structures and the integration of temporal data make commonly available diagrams and charts insufficient for an intuitive and systematic display of results. Therefore, we facilitate methods such as multiple coordinated views, abstraction and focus and context to display simulation outcomes in a dedicated tool. By linking the result data with patient-specific anatomy, we make relevant parameters tangible for clinicians. Furthermore, we introduce new concepts to show the directions of impact force vectors, which were not accessible before. We integrated our toolset into a spine segmentation and simulation pipeline and evaluated our methods with both surgeons and biomechanical researchers. When comparing our methods against standard representations that are currently in use, we found increases in accuracy and speed in data exploration tasks. in a qualitative review, domain experts deemed the tool highly useful when dealing with simulation result data, which typically combines time-dependent patient movement and the resulting force distributions on spinal structures.

Intuitive assessment of modeled lumbar spinal motion by clustering and visualization of finite helical axes.

A variety of medical imaging procedures, cadaver experiments, and computer models have been utilized to capture, depict, and understand the motion of the human lumbar spine. Particular interest lies in assessing the relative movement between two adjacent vertebrae, which can be represented by a temporal evolution of finite helical axes (FHA). Mathematically, this FHA evolution constitutes a seven-dimensional quantity: one dimension for the time, two for the (normalized) direction vector, another two for the (unique) position vector, as well as one for each the angle of rotation around and the amount of translation along the axis. Predominantly in the literature, however, movements are assumed to take place in certain physiological planes on which FHA are projected. The resulting three-dimensional quantity - the so-called centrode - is easily presentable but leaves out substantial pieces of available data. Here, we investigate and assess several possibilities to visualize subsets of FHA data of increasing dimensionality. Finally, we utilize an agglomerative hierarchical clustering algorithm and propose a novel visualization technique, namely the quiver principal axis plot (QPAP), to depict the entirety of information inherent to hundreds or thousands of FHA. The QPAP method is applied to flexion-extension, lateral bending, and axial rotation movements of a lumbar spine within both a reduced model as well as a complex upper body system.

Comparison of Augmented Reality Display Techniques to Support Medical Needle Insertion.

Augmented reality (AR) may be a useful technique to overcome issues of conventionally used navigation systems supporting medical needle insertions, like increased mental workload and complicated hand-eye coordination. Previous research primarily focused on the development of AR navigation systems designed for specific displaying devices, but differences between employed methods have not been investigated before. To this end, a user study involving a needle insertion task was conducted comparing different AR display techniques with a monitor-based approach as baseline condition for the visualization of navigation information. A video see-through stationary display, an optical see-through head-mounted display and a spatial AR projector-camera-system were investigated in this comparison. Results suggest advantages of using projected navigation information in terms of lower task completion time, lower angular deviation and affirmative subjective participant feedback. Techniques requiring the intermediate view on screens, i.e. the stationary display and the baseline condition, showed less favorable results. Thus, benefits of providing AR navigation information compared to a conventionally used method could be identified. Significant objective measures results, as well as an identification of advantages and disadvantages of individual display techniques contribute to the development and design of improved needle navigation systems.

Temporal Views of Flattened Mitral Valve Geometries.

The mitral valve, one of the four valves in the human heart, controls the bloodflow between the left atrium and ventricle and may suffer from various pathologies. Malfunctioning valves can be treated by reconstructive surgeries, which have to be carefully planned and evaluated. While current research focuses on the modeling and segmentation of the valve, we base our work on existing segmentations of patient-specific mitral valves, that are also time-resolved ( 3D+t) over the cardiac cycle. The interpretation of the data can be ambiguous, due to the complex surface of the valve and multiple time steps. We therefore propose a software prototype to analyze such 3D+t data, by extracting pathophysiological parameters and presenting them via dimensionally reduced visualizations. For this, we rely on an existing algorithm to unroll the convoluted valve surface towards a flattened 2D representation. In this paper, we show that the 3D+t data can be transferred to 3D or 2D representations in a way that allows the domain expert to faithfully grasp important aspects of the cardiac cycle. In this course, we not only consider common pathophysiological parameters, but also introduce new observations that are derived from landmarks within the segmentation model. Our analysis techniques were developed in collaboration with domain experts and a survey showed that the insights have the potential to support mitral valve diagnosis and the comparison of the pre- and post-operative condition of a patient.

Mitral valve flattening and parameter mapping for patient-specific valve diagnosis.

PURPOSE: Intensive planning and analysis from echocardiography are a crucial step before reconstructive surgeries are applied to malfunctioning mitral valves. Volume visualizations of echocardiographic data are often used in clinical routine. However, they lack a clear visualization of the crucial factors for decision making. METHODS: We build upon patient-specific mitral valve surface models segmented from echocardiography that represent the valve's geometry, but suffer from self-occlusions due to complex 3D shape. We transfer these to 2D maps by unfolding their geometry, resulting in a novel 2D representation that maintains anatomical resemblance to the 3D geometry. It can be visualized together with color mappings and presented to physicians to diagnose the pathology in one gaze without the need for further scene interaction. Furthermore, it facilitates the computation of a Pathology Score, which can be used for diagnosis support. RESULTS: Quality and effectiveness of the proposed methods were evaluated through a user survey conducted with domain experts. We assessed pathology detection accuracy using 3D valve models in comparison with the novel visualizations. Classification accuracy increased by 5.3% across all tested valves and by 10.0% for prolapsed valves. Further, the participants' understanding of the relation between 3D and 2D views was evaluated. The Pathology Score is found to have potential to support discriminating pathologic valves from normal valves. CONCLUSIONS: In summary, our survey shows that pathology detection can be improved in comparison with simple 3D surface visualizations of the mitral valve. The correspondence between the 2D and 3D representations is comprehensible, and color-coded pathophysiological magnitudes further support the clinical assessment.

Comparison of Projective Augmented Reality Concepts to Support Medical Needle Insertion.

Augmented reality (AR) is a promising tool to improve instrument navigation in needle-based interventions. Limited research has been conducted regarding suitable navigation visualizations. In this work, three navigation concepts based on existing approaches were compared in a user study using a projective AR setup. Each concept was implemented with three different scales for accuracy-to-color mapping and two methods of navigation indicator scaling. Participants were asked to perform simulated needle insertion tasks with each of the resulting 18 prototypes. Insertion angle and insertion depth accuracies were measured and analyzed, as well as task completion time and participants' subjectively perceived task difficulty. Results show a clear ranking of visualization concepts across variables. Less consistent results were obtained for the color and indicator scaling factors. Results suggest that logarithmic indicator scaling achieved better accuracy, but participants perceived it to be more difficult than linear scaling. With specific results for angle and depth accuracy, our study contributes to the future composition of improved navigation support and systems for precise needle insertion or similar applications.

Classification of Blood Flow Patterns in Cerebral Aneurysms.

We present a Cerebral Aneurysm Vortex Classification (CAVOCLA) that allows to classify blood flow in cerebral aneurysms. Medical studies assume a strong relation between the progression and rupture of aneurysms and flow patterns. To understand how flow patterns impact the vessel morphology, they are manually classified according to predefined classes. However, manual classifications are time-consuming and exhibit a high inter-observer variability. In contrast, our approach is more objective and faster than manual methods. The classification of integral lines, representing steady or unsteady blood flow, is based on a mapping of the aneurysm surface to a hemisphere by calculating polar-based coordinates. The lines are clustered and for each cluster a representative is calculated. Then, the polar-based coordinates are transformed to the representative as basis for the classification. Classes are based on the flow complexity. The classification results are presented by a detail-on-demand approach using a visual transition from the representative over an enclosing surface to the associated lines. Based on seven representative datasets, we conduct an informal interview with five domain experts to evaluate the system. They confirmed that CAVOCLA allows for a robust classification of intra-aneurysmal flow patterns. The detail-on-demand visualization enables an efficient exploration and interpretation of flow patterns.

Visual Analysis of Aneurysm Data using Statistical Graphics.

This paper presents a framework to explore multi-field data of aneurysms occurring at intracranial and cardiac arteries by using statistical graphics. The rupture of an aneurysm is often a fatal scenario, whereas during treatment serious complications for the patient can occur. Whether an aneurysm ruptures or whether a treatment is successful depends on the interaction of different morphological such as wall deformation and thickness, and hemodynamic attributes like wall shear stress and pressure. Therefore, medical researchers are very interested in better understanding these relationships. However, the required analysis is a time-consuming process, where suspicious wall regions are difficult to detect due to the time-dependent behavior of the data. Our proposed visualization framework enables medical researchers to efficiently assess aneurysm risk and treatment options. This comprises a powerful set of views including 2D and 3D depictions of the aneurysm morphology as well as statistical plots of different scalar fields. Brushing and linking aids the user to identify interesting wall regions and to understand the influence of different attributes on the aneurysm's state. Moreover, a visual comparison of pre- and post-treatment as well as different treatment options is provided. Our analysis techniques are designed in collaboration with domain experts, e.g., physicians, and we provide details about the evaluation.

Management of Cerebral Aneurysm Descriptors based on an Automatic Ostium Extraction.

We present a framework to manage cerebral aneurysms. Rupture risk evaluation is based on manually extracted descriptors, which is time-consuming. Thus, we provide an automatic solution by considering several questions: How can expert knowledge be integrated? How should meta data be defined? Which interaction techniques are needed for data exploration.

Concepts for augmented reality visualisation to support needle guidance inside the MRI.

During MRI-guided interventions, navigation support is often separated from the operating field on displays, which impedes the interpretation of positions and orientations of instruments inside the patient's body as well as hand-eye coordination. To overcome these issues projector-based augmented reality can be used to support needle guidance inside the MRI bore directly in the operating field. The authors present two visualisation concepts for needle navigation aids which were compared in an accuracy and usability study with eight participants, four of whom were experienced radiologists. The results show that both concepts are equally accurate ( 2.0 ± 0.6 and 1.7 ± 0.5 mm ), useful and easy to use, with clear visual feedback about the state and success of the needle puncture. For easier clinical applicability, a dynamic projection on moving surfaces and organ movement tracking are needed. For now, tests with patients with respiratory arrest are feasible.

Visualization and Extraction of Carvings for Heritage Conservation.

We present novel techniques for visualizing, illustrating, analyzing, and generating carvings in surfaces. In particular, we consider the carvings in the plaster of the cloister of the Magdeburg cathedral, which dates to the 13th century. Due to aging and weathering, the carvings have flattened. Historians and restorers are highly interested in using digitalization techniques to analyze carvings in historic artifacts and monuments and to get impressions and illustrations of their original shape and appearance. Moreover, museums and churches are interested in such illustrations for presenting them to visitors. The techniques that we propose allow for detecting, selecting, and visualizing carving structures. In addition, we introduce an example-based method for generating carvings. The resulting tool, which integrates all techniques, was evaluated by three experienced restorers to assess the usefulness and applicability. Furthermore, we compared our approach with exaggerated shading and other state-of-the-art methods.

Combined Visualization of Vessel Deformation and Hemodynamics in Cerebral Aneurysms.

We present the first visualization tool that combines patient-specific hemodynamics with information about the vessel wall deformation and wall thickness in cerebral aneurysms. Such aneurysms bear the risk of rupture, whereas their treatment also carries considerable risks for the patient. For the patient-specific rupture risk evaluation and treatment analysis, both morphological and hemodynamic data have to be investigated. Medical researchers emphasize the importance of analyzing correlations between wall properties such as the wall deformation and thickness, and hemodynamic attributes like the Wall Shear Stress and near-wall flow. Our method uses a linked 2.5D and 3D depiction of the aneurysm together with blood flow information that enables the simultaneous exploration of wall characteristics and hemodynamic attributes during the cardiac cycle. We thus offer medical researchers an effective visual exploration tool for aneurysm treatment risk assessment. The 2.5D view serves as an overview that comprises a projection of the vessel surface to a 2D map, providing an occlusion-free surface visualization combined with a glyph-based depiction of the local wall thickness. The 3D view represents the focus upon which the data exploration takes place. To support the time-dependent parameter exploration and expert collaboration, a camera path is calculated automatically, where the user can place landmarks for further exploration of the properties. We developed a GPU-based implementation of our visualizations with a flexible interactive data exploration mechanism. We designed our techniques in collaboration with domain experts, and provide details about the evaluation.