Multi-scale Feature Alignment for Continual Learning of Unlabeled Domains.

Methods for unsupervised domain adaptation (UDA) help to improve the performance of deep neural networks on unseen domains without any labeled data. Especially in medical disciplines such as histopathology, this is crucial since large datasets with detailed annotations are scarce. While the majority of existing UDA methods focus on the adaptation from a labeled source to a single unlabeled target domain, many real-world applications with a long life cycle involve more than one target domain. Thus, the ability to sequentially adapt to multiple target domains becomes essential. In settings where the data from previously seen domains cannot be stored, e.g., due to data protection regulations, the above becomes a challenging continual learning problem. To this end, we propose to use generative feature-driven image replay in conjunction with a dual-purpose discriminator that not only enables the generation of images with realistic features for replay, but also promotes feature alignment during domain adaptation. We evaluate our approach extensively on a sequence of three histopathological datasets for tissue-type classification, achieving state-of-the-art results. We present detailed ablation experiments studying our proposed method components and demonstrate a possible use-case of our continual UDA method for an unsupervised patch-based segmentation task given high-resolution tissue images. Our code is available at: https://github.com/histocartography/multi-scale-feature-alignment.

Longevity interventions modulate mechanotransduction and extracellular matrix homeostasis in C. elegans.

Dysfunctional extracellular matrices (ECM) contribute to aging and disease. Repairing dysfunctional ECM could potentially prevent age-related pathologies. Interventions promoting longevity also impact ECM gene expression. However, the role of ECM composition changes in healthy aging remains unclear. Here we perform proteomics and in-vivo monitoring to systematically investigate ECM composition (matreotype) during aging in C. elegans revealing three distinct collagen dynamics. Longevity interventions slow age-related collagen stiffening and prolong the expression of collagens that are turned over. These prolonged collagen dynamics are mediated by a mechanical feedback loop of hemidesmosome-containing structures that span from the exoskeletal ECM through the hypodermis, basement membrane ECM, to the muscles, coupling mechanical forces to adjust ECM gene expression and longevity via the transcriptional co-activator YAP-1 across tissues. Our results provide in-vivo evidence that coordinated ECM remodeling through mechanotransduction is required and sufficient to promote longevity, offering potential avenues for interventions targeting ECM dynamics.

Weakly supervised joint whole-slide segmentation and classification in prostate cancer.

The identification and segmentation of histological regions of interest can provide significant support to pathologists in their diagnostic tasks. However, segmentation methods are constrained by the difficulty in obtaining pixel-level annotations, which are tedious and expensive to collect for whole-slide images (WSI). Though several methods have been developed to exploit image-level weak-supervision for WSI classification, the task of segmentation using WSI-level labels has received very little attention. The research in this direction typically require additional supervision beyond image labels, which are difficult to obtain in real-world practice. In this study, we propose WholeSIGHT, a weakly-supervised method that can simultaneously segment and classify WSIs of arbitrary shapes and sizes. Formally, WholeSIGHT first constructs a tissue-graph representation of WSI, where the nodes and edges depict tissue regions and their interactions, respectively. During training, a graph classification head classifies the WSI and produces node-level pseudo-labels via post-hoc feature attribution. These pseudo-labels are then used to train a node classification head for WSI segmentation. During testing, both heads simultaneously render segmentation and class prediction for an input WSI. We evaluate the performance of WholeSIGHT on three public prostate cancer WSI datasets. Our method achieves state-of-the-art weakly-supervised segmentation performance on all datasets while resulting in better or comparable classification with respect to state-of-the-art weakly-supervised WSI classification methods. Additionally, we assess the generalization capability of our method in terms of segmentation and classification performance, uncertainty estimation, and model calibration. Our code is available at: https://github.com/histocartography/wholesight.

Robust Imaging of Speed of Sound Using Virtual Source Transmission.

Speed of sound (SoS) is a novel imaging biomarker for assessing the biomechanical characteristics of soft tissues. SoS imaging in the pulse-echo mode using conventional ultrasound (US) systems with hand-held transducers has the potential to enable new clinical uses. Recent work demonstrated that diverging waves (DWs) from a single element (SE) transmit to outperform plane-wave sequences. However, SE transmits have severely limited power and hence produce a low signal-to-noise ratio (SNR) in echo data. We herein propose Walsh-Hadamard (WH) coded and virtual-source (VS) transmit sequences for the improved SNR in SoS imaging. We additionally present an iterative method of estimating beamforming (BF) SoS in the medium, which otherwise confounds SoS reconstructions due to beamforming inaccuracies in the images used for reconstruction. Through numerical simulations, phantom experiments, and in vivo imaging data, we show that WH is not robust against motion, which is often unavoidable in clinical imaging scenarios. Our proposed VS sequence is shown to provide the highest SoS reconstruction performance, especially robust to motion artifacts. In phantom experiments, despite having a comparable SoS root-mean-square error (RMSE) of 17.5-18.0 m/s at rest, with a minor axial probe motion of ≈ 0.67 mm/s the RMSE for SE, WH, and VS already deteriorate to 20.2, 105.4, and 19.0 m/s, respectively, showing that WH produces unacceptable results, not robust to motion. In the clinical data, the high SNR and motion resilience of VS sequences are seen to yield superior contrast compared to SE and WH sequences.

Generative appearance replay for continual unsupervised domain adaptation.

Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To that end, we introduce GarDA (Generative Appearance Replay for continual Domain Adaptation), a generative-replay based approach that can adapt a segmentation model sequentially to new domains with unlabeled data. In contrast to single-step unsupervised domain adaptation (UDA), continual adaptation to a sequence of domains enables leveraging and consolidation of information from multiple domains. Unlike previous approaches in incremental UDA, our method does not require access to previously seen data, making it applicable in many practical scenarios. We evaluate GarDA on three datasets with different organs and modalities, where it substantially outperforms existing techniques. Our code is available at: https://github.com/histocartography/generative-appearance-replay.

Analytical estimation of beamforming speed-of-sound using transmission geometry.

Most ultrasound imaging techniques necessitate the fundamental step of converting temporal signals received from transducer elements into a spatial echogenecity map. This beamforming (BF) step requires the knowledge of speed-of-sound (SoS) value in the imaged medium. An incorrect assumption of BF SoS leads to aberration artifacts, not only deteriorating the quality and resolution of conventional brightness mode (B-mode) images, hence limiting their clinical usability, but also impairing other ultrasound modalities such as elastography and spatial SoS reconstructions, which rely on faithfully beamformed images as their input. In this work, we propose an analytical method for estimating BF SoS. We show that pixel-wise relative shifts between frames beamformed with an assumed SoS is a function of geometric disparities of the transmission paths and the error in such SoS assumption. Using this relation, we devise an analytical model, the closed form solution of which yields the difference between the assumed and the true SoS in the medium. Based on this, we correct the BF SoS, which can also be applied iteratively. Both in simulations and experiments, lateral B-mode resolution is shown to be improved by ≈25% compared to that with an initial SoS assumption error of 3.3% (50m/s), while localization artifacts from beamforming are also corrected. After 5 iterations, our method achieves BF SoS errors of under 0.6m/s in simulations. Residual time-delay errors in beamforming 32 numerical phantoms are shown to reduce down to 0.07µs, with average improvements of up to 21 folds compared to initial inaccurate assumptions. We additionally show the utility of the proposed method in imaging local SoS maps, where using our correction method reduces reconstruction root-mean-square errors substantially, down to their lower-bound with actual BF SoS.

Probabilistic spatial analysis in quantitative microscopy with uncertainty-aware cell detection using deep Bayesian regression.

The investigation of biological systems with three-dimensional microscopy demands automatic cell identification methods that not only are accurate but also can imply the uncertainty in their predictions. The use of deep learning to regress density maps is a popular successful approach for extracting cell coordinates from local peaks in a postprocessing step, which then, however, hinders any meaningful probabilistic output. We propose a framework that can operate on large microscopy images and output probabilistic predictions (i) by integrating deep Bayesian learning for the regression of uncertainty-aware density maps, where peak detection algorithms generate cell proposals, and (ii) by learning a mapping from prediction proposals to a probabilistic space that accurately represents the chances of a successful prediction. Using these calibrated predictions, we propose a probabilistic spatial analysis with Monte Carlo sampling. We demonstrate this in a bone marrow dataset, where our proposed methods reveal spatial patterns that are otherwise undetectable.

Computational analysis of subscapularis tears and pectoralis major transfers on muscular activity.

BACKGROUND: Pectoralis major is the most common muscle transfer procedure to restore joint function after subscapularis tears. Limited information is available on how the neuromuscular system adjusts to the new configuration, which could explain the mixed outcomes of the procedure. The purpose of this study is to assess how muscles activation patterns change after pectoralis major transfers and report their biomechanical implications. METHODS: We compare how muscle activation change with subscapularis tears and after its treatment by pectoralis major transfers of the clavicular, sternal, or both these segments, during three activities of daily living and a computational musculoskeletal model of the shoulder. FINDINGS: Our results indicate that subscapularis tears require a compensatory activation of the supraspinatus and is accompanied by a reduced co-contraction of the infraspinatus, both of which can be partially recovered after transfer. Furthermore, although the pectoralis major acts asynchronously to the subscapularis before the transfer, its activation pattern changes significantly after the transfer. INTERPRETATION: The capability of a transferred muscle segment to activate similarly to the intact subscapularis is found to be dependent on the given motion. Differences in the activation patterns between intact subscapularis and the segments of pectoralis major may explain the difficulty in adapting psycho-motor patterns during the rehabilitation period. Thereby, rehabilitation programs could benefit from targeted training on specific motion and biofeedback programs. Finally, the condition of the anterior deltoid should be considered to improve joint function.

Nonlinear Characterization of Tissue Viscoelasticity With Acoustoelastic Attenuation of Shear Waves.

Shear-wave elastography (SWE) measures shear-wave speed (SWS), which is related to the underlying shear modulus of soft tissue. SWS in soft tissue changes depending on the amount of external strain that soft tissue is subjected to due to the acoustoelastic (AE) phenomenon. In the literature, variations of SWS as a function of applied uniaxial strain were used for nonlinear characterization, assuming soft tissues to be elastic, although soft tissues are indeed viscoelastic in nature. Hence, nonlinear characterization using SWS alone is insufficient. In this work, we use SWS together with shear-wave attenuation (SWA) during incremental quasi-static compressions in order to derive biomechanical characterization based on the AE theory in terms of well-defined storage and loss moduli. As part of this study, we also quantify the effect of applied strain on measurements of SWS and SWA since such confounding effects need to be taken into account when using SWS and/or SWA, e.g., for staging a disease state, while such effects can also serve as an additional imaging biomarker. Our results from tissue-mimicking phantoms with varying oil percentages and ex vivo porcine liver experiments demonstrate the feasibility of our proposed methods. In both experiments, SWA was observed to decrease with applied strain. For 10% compression in ex vivo livers, shear-wave attenuation decreased, on average, by 28% (93 Np/m), while SWS increased, on average, by 20% (0.26 m/s).

Hierarchical graph representations in digital pathology.

Cancer diagnosis, prognosis, and therapy response predictions from tissue specimens highly depend on the phenotype and topological distribution of constituting histological entities. Thus, adequate tissue representations for encoding histological entities is imperative for computer aided cancer patient care. To this end, several approaches have leveraged cell-graphs, capturing the cell-microenvironment, to depict the tissue. These allow for utilizing graph theory and machine learning to map the tissue representation to tissue functionality, and quantify their relationship. Though cellular information is crucial, it is incomplete alone to comprehensively characterize complex tissue structure. We herein treat the tissue as a hierarchical composition of multiple types of histological entities from fine to coarse level, capturing multivariate tissue information at multiple levels. We propose a novel multi-level hierarchical entity-graph representation of tissue specimens to model the hierarchical compositions that encode histological entities as well as their intra- and inter-entity level interactions. Subsequently, a hierarchical graph neural network is proposed to operate on the hierarchical entity-graph and map the tissue structure to tissue functionality. Specifically, for input histology images, we utilize well-defined cells and tissue regions to build HierArchical Cell-to-Tissue (HACT) graph representations, and devise HACT-Net, a message passing graph neural network, to classify the HACT representations. As part of this work, we introduce the BReAst Carcinoma Subtyping (BRACS) dataset, a large cohort of Haematoxylin & Eosin stained breast tumor regions-of-interest, to evaluate and benchmark our proposed methodology against pathologists and state-of-the-art computer-aided diagnostic approaches. Through comparative assessment and ablation studies, our proposed method is demonstrated to yield superior classification results compared to alternative methods as well as individual pathologists. The code, data, and models can be accessed at https://github.com/histocartography/hact-net.

Influence of rotator cuff integrity on loading and kinematics before and after reverse shoulder arthroplasty.

Reverse Shoulder Arthroplasty has become a very common procedure for shoulder joint replacement, even for scenarios where an anatomical reconstruction would traditionally be used. Our hypothesis is that implanting a reverse prosthesis with a functional rotator cuff may lead to higher joint reaction force (JRF) and have a negative impact on the prosthesis. Available motion capture data during anterior flexion was input to a finite-element musculoskeletal shoulder model, and muscle activations were computed using inverse dynamics. Simulations were carried out for the intact joint as well as for various types of rotator cuff tears: superior (supraspinatus), superior-anterior (supraspinatus and subscapularis), and superior-posterior (supraspinatus, infraspinatus and teres minor). Each rotator cuff tear condition was repeated after shifting the humerus and the glenohumeral joint center of rotation to represent the effect of a reverse prosthesis. Changes in compressive, shear, and total JRF were analyzed. The model compared favorably to in vivo JRF measurements, and existing clinical and biomechanical knowledge. Implanting a reverse prosthesis with a functional rotator cuff or with an isolated supraspinatus tear led to more than 2 times higher compressive JRF than with massive rotator cuff tears (superior-anterior or superior-posterior), while the shear force remained comparable. The total JRF increased more than 1.5 times. While a lower shear to compressive ratio may reduce the risk of glenosphere loosening, higher JRF might increase the risk for other failure modes such as fracture or polyethylene wear of the reverse prosthesis.

Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy.

Fluorescence microscopy allows for a detailed inspection of cells, cellular networks, and anatomical landmarks by staining with a variety of carefully-selected markers visualized as color channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers, and therefore applicable to a very restricted number of experimental settings. We herein propose Marker Sampling and Excite - a neural network approach with a modality sampling strategy and a novel attention module that together enable (i) flexible training with heterogeneous datasets with combinations of markers and (ii) successful utility of learned models on arbitrary subsets of markers prospectively. We show that our single neural network solution performs comparably to an upper bound scenario where an ensemble of many networks is naïvely trained for each possible marker combination separately. In addition, we demonstrate the feasibility of this framework in high-throughput biological analysis by revising a recent quantitative characterization of bone marrow vasculature in 3D confocal microscopy datasets and further confirm the validity of our approach on an additional, significantly different dataset of microvessels in fetal liver tissues. Not only can our work substantially ameliorate the use of deep learning in fluorescence microscopy analysis, but it can also be utilized in other fields with incomplete data acquisitions and missing modalities.

Speed-of-sound imaging using diverging waves.

PURPOSE: Due to its safe, low-cost, portable, and real-time nature, ultrasound is a prominent imaging method in computer-assisted interventions. However, typical B-mode ultrasound images have limited contrast and tissue differentiation capability for several clinical applications. METHODS: Recent introduction of imaging speed-of-sound (SoS) in soft tissues using conventional ultrasound systems and transducers has great potential in clinical translation providing additional imaging contrast, e.g., in intervention planning, navigation, and guidance applications. However, current pulse-echo SoS imaging methods relying on plane wave (PW) sequences are highly prone to aberration effects, therefore suboptimal in image quality. In this paper we propose using diverging waves (DW) for SoS imaging and study this comparatively to PW. RESULTS: We demonstrate wavefront aberration and its effects on the key step of displacement tracking in the SoS reconstruction pipeline, comparatively between PW and DW on a synthetic example. We then present the parameterization sensitivity of both approaches on a set of simulated phantoms. Analyzing SoS imaging performance comparatively indicates that using DW instead of PW, the reconstruction accuracy improves by over 20% in root-mean-square-error (RMSE) and by 42% in contrast-to-noise ratio (CNR). We then demonstrate SoS reconstructions with actual US acquisitions of a breast phantom. With our proposed DW, CNR for a high contrast tumor-representative inclusion is improved by 42%, while for a low contrast cyst-representative inclusion a 2.8-fold improvement is achieved. CONCLUSION: SoS imaging, so far only studied using a plane wave transmission scheme, can be made more reliable and accurate using DW. The high imaging contrast of DW-based SoS imaging will thus facilitate the clinical translation of the method and utilization in computer-assisted interventions such as ultrasound-guided biopsies, where B-Mode contrast is often to low to detect potential lesions.

Hemodynamic response to sensory stimulation in mice: Comparison between functional ultrasound and optoacoustic imaging.

Intense efforts are underway to develop functional imaging modalities for capturing brain activity at the whole organ scale with high spatial and temporal resolution. Functional optoacoustic (fOA) imaging is emerging as a new tool to monitor multiple hemodynamic parameters across the mouse brain, but its sound validation against other neuroimaging modalities is often lacking. Here we investigate mouse brain responses to peripheral sensory stimulation using both fOA and functional ultrasound (fUS) imaging. The two modalities operate under similar spatio-temporal resolution regime, with a potential to provide synergistic and complementary hemodynamic readouts. Specific contralateral activation was observed with sub-millimeter spatial resolution with both methods. Sensitivity to hemodynamic activity was found to be on comparable levels, with the strongest responses obtained in the oxygenated hemoglobin channel of fOA. While the techniques attained highly correlated hemodynamic responses, the differential fOA readings of oxygenated and deoxygenated haemoglobin provided complementary information to the blood flow contrast of fUS. The multi-modal approach may thus emerge as a powerful tool providing new insights into brain function, complementing our current knowledge generated with well-established neuroimaging methods.

Learning ultrasound rendering from cross-sectional model slices for simulated training.

PURPOSE: Given the high level of expertise required for navigation and interpretation of ultrasound images, computational simulations can facilitate the training of such skills in virtual reality. With ray-tracing based simulations, realistic ultrasound images can be generated. However, due to computational constraints for interactivity, image quality typically needs to be compromised. METHODS: We propose herein to bypass any rendering and simulation process at interactive time, by conducting such simulations during a non-time-critical offline stage and then learning image translation from cross-sectional model slices to such simulated frames. We use a generative adversarial framework with a dedicated generator architecture and input feeding scheme, which both substantially improve image quality without increase in network parameters. Integral attenuation maps derived from cross-sectional model slices, texture-friendly strided convolutions, providing stochastic noise and input maps to intermediate layers in order to preserve locality are all shown herein to greatly facilitate such translation task. RESULTS: Given several quality metrics, the proposed method with only tissue maps as input is shown to provide comparable or superior results to a state-of-the-art that uses additional images of low-quality ultrasound renderings. An extensive ablation study shows the need and benefits from the individual contributions utilized in this work, based on qualitative examples and quantitative ultrasound similarity metrics. To that end, a local histogram statistics based error metric is proposed and demonstrated for visualization of local dissimilarities between ultrasound images. CONCLUSION: A deep-learning based direct transformation from interactive tissue slices to likeness of high quality renderings allow to obviate any complex rendering process in real-time, which could enable extremely realistic ultrasound simulations on consumer-hardware by moving the time-intensive processes to a one-time, offline, preprocessing data preparation stage that can be performed on dedicated high-end hardware.

Quantification of immobilization-induced changes in human calf muscle using speed-of-sound ultrasound: An observational pilot study.

ABSTRACT: Short-term immobilization leads to fatty muscular degeneration, which is associated with various negative health effects. Based on literature showing very high correlations between MRI Dixon fat fraction and Speed-of-Sound (SoS), we hypothesized that we can detect short-term-immobilization-induced differences in SoS.Both calves of 10 patients with a calf cast on one side for a mean duration of 41 ±â€Š26 days were examined in relaxed position using a standard ultrasound machine. Calf perimeters were measured for both sides. A flat Plexiglas-reflector, placed vertically on the opposite side of the probe with the calf in-between, was used as a timing reference for SoS. SoS was both manually annotated by two readers and assessed by an automatic annotation algorithm. The thickness values of the subcutaneous fat and muscle layers were manually read from the B-mode images. Differences between the cast and non-cast calves were calculated with a paired t test. Correlation analysis of SoS and calf perimeter was performed using Pearson's correlation coefficient.Paired t test showed significant differences between the cast and non-cast side for both SoS (P < .01) and leg perimeter (P < .001). SoS was reduced with the number of days after cast installment (r = -0.553, P = .097). No significant differences were found for muscle layer thickness, subcutaneous fat layer thickness, mean fat echo intensity, or mean muscle echo intensity.Short-term-immobilization led to a significant reduction in SoS in the cast calf compared to the healthy calf, indicating a potential role of SoS as a biomarker in detecting immobilization-induced fatty muscular degeneration not visible on B-mode ultrasound.

Reducing annotation effort in digital pathology: A Co-Representation learning framework for classification tasks.

Classification of digital pathology images is imperative in cancer diagnosis and prognosis. Recent advancements in deep learning and computer vision have greatly benefited the pathology workflow by developing automated solutions for classification tasks. However, the cost and time for acquiring high quality task-specific large annotated training data are subject to intra- and inter-observer variability, thus challenging the adoption of such tools. To address these challenges, we propose a classification framework via co-representation learning to maximize the learning capability of deep neural networks while using a reduced amount of training data. The framework captures the class-label information and the local spatial distribution information by jointly optimizing a categorical cross-entropy objective and a deep metric learning objective respectively. A deep metric learning objective is incorporated to enhance the classification, especially in the low training data regime. Further, a neighborhood-aware multiple similarity sampling strategy, and a soft-multi-pair objective that optimizes interactions between multiple informative sample pairs, is proposed to accelerate deep metric learning. We evaluate the proposed framework on five benchmark datasets from three digital pathology tasks, i.e., nuclei classification, mitosis detection, and tissue type classification. For all the datasets, our framework achieves state-of-the-art performance when using approximately only 50% of the training data. On using complete training data, the proposed framework outperforms the state-of-the-art on all the five datasets.

Frequency-dependent attenuation reconstruction with an acoustic reflector.

Attenuation of ultrasound waves varies with tissue composition, hence its estimation offers great potential for tissue characterization and diagnosis and staging of pathology. We recently proposed a method that allows to spatially reconstruct the distribution of the overall ultrasound attenuation in tissue based on computed tomography, using reflections from a passive acoustic reflector. This requires a standard ultrasound transducer operating in pulse-echo mode and a calibration protocol using water measurements, thus it can be implemented on conventional ultrasound systems with minor adaptations. Herein, we extend this method by additionally estimating and imaging the frequency-dependent nature of local ultrasound attenuation for the first time. Spatial distributions of attenuation coefficient and exponent are reconstructed, enabling an elaborate and expressive tissue-specific characterization. With simulations, we demonstrate that our proposed method yields a low reconstruction error of 0.04 dB/cm at 1 MHz for attenuation coefficient and 0.08 for the frequency exponent. With tissue-mimicking phantoms and ex-vivo bovine muscle samples, a high reconstruction contrast as well as reproducibility are demonstrated. Attenuation exponents of a gelatin-cellulose mixture and an ex-vivo bovine muscle sample were found to be, respectively, 1.4 and 0.5 on average, consistently from different images of their heterogeneous compositions. Such frequency-dependent parametrization could enable novel imaging and diagnostic techniques, as well as facilitate attenuation compensation of other ultrasound-based imaging techniques.