WaveNet: Wavelet Network With Knowledge Distillation for RGB-T Salient Object Detection.

In recent years, various neural network architectures for computer vision have been devised, such as the visual transformer and multilayer perceptron (MLP). A transformer based on an attention mechanism can outperform a traditional convolutional neural network. Compared with the convolutional neural network and transformer, the MLP introduces less inductive bias and achieves stronger generalization. In addition, a transformer shows an exponential increase in the inference, training, and debugging times. Considering a wave function representation, we propose the WaveNet architecture that adopts a novel vision task-oriented wavelet-based MLP for feature extraction to perform salient object detection in RGB (red-green-blue)-thermal infrared images. In addition, we apply knowledge distillation to a transformer as an advanced teacher network to acquire rich semantic and geometric information and guide WaveNet learning with this information. Following the shortest-path concept, we adopt the Kullback-Leibler distance as a regularization term for the RGB features to be as similar to the thermal infrared features as possible. The discrete wavelet transform allows for the examination of frequency-domain features in a local time domain and time-domain features in a local frequency domain. We apply this representation ability to perform cross-modality feature fusion. Specifically, we introduce a progressively cascaded sine-cosine module for cross-layer feature fusion and use low-level features to obtain clear boundaries of salient objects through the MLP. Results from extensive experiments indicate that the proposed WaveNet achieves impressive performance on benchmark RGB-thermal infrared datasets. The results and code are publicly available at https://github.com/nowander/WaveNet.

PET and MRI image fusion based on a dense convolutional network with dual attention.

The fusion techniques of different modalities in medical images, e.g., Positron Emission Tomography (PET) and Magnetic Resonance Imaging (MRI), are increasingly significant in many clinical applications by integrating the complementary information from different medical images. In this paper, we propose a novel fusion model based on a dense convolutional network with dual attention (CSpA-DN) for PET and MRI images. In our framework, an encoder composed of the densely connected neural network is constructed to extract features from source images, and a decoder network is employed to generate the fused image from these features. Simultaneously, a dual-attention module is introduced in the encoder and decoder to further integrate local features along with their global dependencies adaptively. In the dual-attention module, a spatial attention block is leveraged to extract features of each point from encoder network by a weighted sum of feature information at all positions. Meanwhile, the interdependent correlation of all image features is aggregated via a module of channel attention. In addition, we design a specific loss function including image loss, structural loss, gradient loss and perception loss to preserve more structural and detail information and sharpen the edges of targets. Our approach facilitates the fused images to not only preserve abundant functional information from PET images but also retain rich detail structures of MRI images. Experimental results on publicly available datasets illustrate the superiorities of CSpA-DN model compared with state-of-the-art methods according to both qualitative observation and objective assessment.

GMNet: Graded-Feature Multilabel-Learning Network for RGB-Thermal Urban Scene Semantic Segmentation.

Semantic segmentation is a fundamental task in computer vision, and it has various applications in fields such as robotic sensing, video surveillance, and autonomous driving. A major research topic in urban road semantic segmentation is the proper integration and use of cross-modal information for fusion. Here, we attempt to leverage inherent multimodal information and acquire graded features to develop a novel multilabel-learning network for RGB-thermal urban scene semantic segmentation. Specifically, we propose a strategy for graded-feature extraction to split multilevel features into junior, intermediate, and senior levels. Then, we integrate RGB and thermal modalities with two distinct fusion modules, namely a shallow feature fusion module and deep feature fusion module for junior and senior features. Finally, we use multilabel supervision to optimize the network in terms of semantic, binary, and boundary characteristics. Experimental results confirm that the proposed architecture, the graded-feature multilabel-learning network, outperforms state-of-the-art methods for urban scene semantic segmentation, and it can be generalized to depth data.

IRFR-Net: Interactive Recursive Feature-Reshaping Network for Detecting Salient Objects in RGB-D Images.

Using attention mechanisms in saliency detection networks enables effective feature extraction, and using linear methods can promote proper feature fusion, as verified in numerous existing models. Current networks usually combine depth maps with red-green-blue (RGB) images for salient object detection (SOD). However, fully leveraging depth information complementary to RGB information by accurately highlighting salient objects deserves further study. We combine a gated attention mechanism and a linear fusion method to construct a dual-stream interactive recursive feature-reshaping network (IRFR-Net). The streams for RGB and depth data communicate through a backbone encoder to thoroughly extract complementary information. First, we design a context extraction module (CEM) to obtain low-level depth foreground information. Subsequently, the gated attention fusion module (GAFM) is applied to the RGB depth (RGB-D) information to obtain advantageous structural and spatial fusion features. Then, adjacent depth information is globally integrated to obtain complementary context features. We also introduce a weighted atrous spatial pyramid pooling (WASPP) module to extract the multiscale local information of depth features. Finally, global and local features are fused in a bottom-up scheme to effectively highlight salient objects. Comprehensive experiments on eight representative datasets demonstrate that the proposed IRFR-Net outperforms 11 state-of-the-art (SOTA) RGB-D approaches in various evaluation indicators.

Quantitative Assessment of the Intracranial Vasculature of Infants and Adults Using iCafe (Intracranial Artery Feature Extraction).

Comprehensive quantification of intracranial artery features may help to assess and understand regional variations of blood supply during early brain development and aging. We analyzed vasculature features of 27 healthy infants during natural sleep, 13 infants at 7-months (7.3 ± 1.0 month), and 14 infants at 12-months (11.7 ± 0.4 month), and 13 older healthy, awake adults (62.8 ± 8.7 years) to investigate age-related vascular differences as a preliminary study of vascular changes associated with brain development. 3D time-of-flight (TOF) magnetic resonance angiography (MRA) acquisitions were processed in iCafe, a technique to quantify arterial features (http://icafe.clatfd.cn), to characterize intracranial vasculature. Overall, adult subjects were found to have increased ACA length, tortuosity, and vasculature density compared to both 7-month-old and 12-month-old infants, as well as MCA length compared to 7-month-old infants. No brain laterality differences were observed for any vascular measures in either infant or adult age groups. Reduced skull and brain sharpness, indicative of increased head motion and brain/vascular pulsation, respectively, were observed in infants but not correlated with length, tortuosity, or vasculature density measures. Quantitative analysis of TOF MRA using iCafe may provide an objective approach for systematic study of infant brain vascular development and for clinical assessment of adult and pediatric brain vascular diseases.

Multi-Target Multi-Camera Tracking of Vehicles Using Metadata-Aided Re-ID and Trajectory-Based Camera Link Model.

In this paper, we propose a novel framework for multi-target multi-camera tracking (MTMCT) of vehicles based on metadata-aided re-identification (MA-ReID) and the trajectory-based camera link model (TCLM). Given a video sequence and the corresponding frame-by-frame vehicle detections, we first address the isolated tracklets issue from single camera tracking (SCT) by the proposed traffic-aware single-camera tracking (TSCT). Then, after automatically constructing the TCLM, we solve MTMCT by the MA-ReID. The TCLM is generated from camera topological configuration to obtain the spatial and temporal information to improve the performance of MTMCT by reducing the candidate search of ReID. We also use the temporal attention model to create more discriminative embeddings of trajectories from each camera to achieve robust distance measures for vehicle ReID. Moreover, we train a metadata classifier for MTMCT to obtain the metadata feature, which is concatenated with the temporal attention based embeddings. Finally, the TCLM and hierarchical clustering are jointly applied for global ID assignment. The proposed method is evaluated on the CityFlow dataset, achieving IDF1 76.77%, which outperforms the state-of-the-art MTMCT methods.

Domain adaptive and fully automated carotid artery atherosclerotic lesion detection using an artificial intelligence approach (LATTE) on 3D MRI.

PURPOSE: To develop and evaluate a domain adaptive and fully automated review workflow (lesion assessment through tracklet evaluation, LATTE) for assessment of atherosclerotic disease in 3D carotid MR vessel wall imaging (MR VWI). METHODS: VWI of 279 subjects with carotid atherosclerosis were used to develop LATTE, mainly convolutional neural network (CNN)-based domain adaptive lesion classification after image quality assessment and artery of interest localization. Heterogeneity in test sets from various sites usually causes inferior CNN performance. With our novel unsupervised domain adaptation (DA), LATTE was designed to accurately classify arteries into normal arteries and early and advanced lesions without additional annotations on new datasets. VWI of 271 subjects from four datasets (eight sites) with slightly different imaging parameters/signal patterns were collected to assess the effectiveness of DA of LATTE using the area under the receiver operating characteristic curve (AUC) on all lesions and advanced lesions before and after DA. RESULTS: LATTE had good performance with advanced/all lesion classification, with the AUC of >0.88/0.83, significant improvements from >0.82/0.80 if without DA. CONCLUSIONS: LATTE can locate target arteries and distinguish carotid atherosclerotic lesions with consistently improved performance with DA on new datasets. It may be useful for carotid atherosclerosis detection and assessment on various clinical sites.

Geometry-Based Camera Calibration Using Closed-Form Solution of Principal Line.

Camera calibration is a crucial prerequisite in many applications of computer vision. In this paper, a new geometry-based camera calibration technique is proposed, which resolves two main issues associated with the widely used Zhang's method: (i) the lack of guidelines to avoid outliers in the computation and (ii) the assumption of fixed camera focal length. The proposed approach is based on the closed-form solution of principal lines with their intersection being the principal point while each principal line can concisely represent relative orientation/position (up to one degree of freedom for both) between a special pair of coordinate systems of image plane and calibration pattern. With such analytically tractable image features, computations associated with the calibration are greatly simplified, while the guidelines in (i) can be established intuitively. Experimental results for synthetic and real data show that the proposed approach does compare favorably with Zhang's method, in terms of correctness, robustness, and flexibility, and addresses issues (i) and (ii) satisfactorily.

A novel algorithm for refining cerebral vascular measurements in infants and adults.

BACKGROUND: Comprehensive quantification of intracranial vascular characteristics by vascular tracing provides an objective clinical assessment of vascular structure. However, weak signal or low contrast in small distal arteries, artifacts due to volitional motion, and vascular pulsation are challenges for accurate vessel tracing from 3D time-of-flight (3D-TOF) magnetic resonance angiography (MRA) images. NEW METHOD: A vascular measurement refinement algorithm is developed and validated for robust quantification of intracranial vasculature from 3D-TOF MRA. After automated vascular tracing, centerline positions, lumen radii and centerline deviations are jointly optimized to restrict traces to within vascular regions in the straightened curved planar reformation (CPR) views. The algorithm is validated on simulated vascular images and on repeat 3D-TOF MRA acquired from infants and adults. RESULTS: The refinement algorithm can reliably estimate vascular radius and correct deviated centerlines. For the simulated vascular image with noise level of 1 and deviation of centerline of 3, the mean radius difference is below 15.3 % for scan-rescan reliability. Vascular features from repeated clinical scans show significantly improved measurement agreement, with intra-class correlation coefficient (ICC) improvement from 0.55 to 0.7 for infants and from 0.59 to 0.92 for adults. COMPARISON WITH EXISTING METHODS: The refinement algorithm is novel because it utilizes straightened CPR views that incorporate information from the entire artery. In addition, the optimization corrects centerline positions, lumen radii and centerline deviations simultaneously. CONCLUSIONS: Intracranial vasculature quantification using a novel refinement algorithm for vascular tracing improves the reliability of vascular feature measurements in both infants and adults.

Fully automated and robust analysis technique for popliteal artery vessel wall evaluation (FRAPPE) using neural network models from standardized knee MRI.

PURPOSE: To develop a fully automated vessel wall (VW) analysis workflow (fully automated and robust analysis technique for popliteal artery evaluation, FRAPPE) on the popliteal artery in standardized knee MR images. METHODS: Popliteal artery locations were detected from each MR slice by a deep neural network model and connected into a 3D artery centerline. Vessel wall regions around the centerline were then segmented using another neural network model for segmentation in polar coordinate system. Contours from vessel wall segmentations were used for vascular feature calculation, such as mean wall thickness and wall area. A transfer learning and active learning framework was applied in training the localization and segmentation neural network models to maintain accuracy while reducing manual annotations. This new popliteal artery analysis technique (FRAPPE) was validated against manual segmentation qualitatively and quantitatively in a series of 225 cases from the Osteoarthritis Initiative (OAI) dataset. RESULTS: FRAPPE demonstrated high accuracy and robustness in locating popliteal arteries, segmenting artery walls, and quantifying arterial features. Qualitative evaluations showed 1.2% of slices had noticeable major errors, including segmenting the wrong target and irregular vessel wall contours. The mean Dice similarity coefficient with manual segmentation was 0.79, which is comparable to inter-rater variations. Repeatability evaluations show most of the vascular features have good to excellent repeatability from repeated scans of same subjects, with intra-class coefficient ranging from 0.80 to 0.98. CONCLUSION: This technique can be used in large population-based studies, such as OAI, to efficiently assess the burden of atherosclerosis from routine MR knee scans.

Adversarial Learning for Joint Optimization of Depth and Ego-Motion.

In recent years, supervised deep learning methods have shown a great promise in dense depth estimation. However, massive high-quality training data are expensive and impractical to acquire. Alternatively, self-supervised learning-based depth estimators can learn the latent transformation from monocular or binocular video sequences by minimizing the photometric warp error between consecutive frames, but they suffer from the scale ambiguity problem or have difficulty in estimating precise pose changes between frames. In this paper, we propose a joint self-supervised deep learning pipeline for depth and ego-motion estimation by employing the advantages of adversarial learning and joint optimization with spatial-temporal geometrical constraints. The stereo reconstruction error provides the spatial geometric constraint to estimate the absolute scale depth. Meanwhile, the depth map with an absolute scale and a pre-trained pose network serves as a good starting point for direct visual odometry (DVO). DVO optimization based on spatial geometric constraints can result in a fine-grained ego-motion estimation with the additional backpropagation signals provided to the depth estimation network. Finally, the spatial and temporal domain-based reconstructed views are concatenated, and the iterative coupling optimization process is implemented in combination with the adversarial learning for accurate depth and precise ego-motion estimation. The experimental results show superior performance compared with state-of-the-art methods for monocular depth and ego-motion estimation on the KITTI dataset and a great generalization ability of the proposed approach.

Automated Artery Localization and Vessel Wall Segmentation using Tracklet Refinement and Polar Conversion.

Quantitative analysis of blood vessel wall structures is important to study atherosclerotic diseases and assess cardiovascular event risks. To achieve this, accurate identification of vessel luminal and outer wall contours is needed. Computer-assisted tools exist, but manual preprocessing steps, such as region of interest identification and/or boundary initialization, are still needed. In addition, prior knowledge of the ring shape of vessel walls has not been fully explored in designing segmentation methods. In this work, a fully automated artery localization and vessel wall segmentation system is proposed. A tracklet refinement algorithm was adapted to robustly identify the artery of interest from a neural network-based artery centerline identification architecture. Image patches were extracted from the centerlines and converted in a polar coordinate system for vessel wall segmentation. The segmentation method used 3D polar information and overcame problems such as contour discontinuity, complex vessel geometry, and interference from neighboring vessels. Verified by a large (>32000 images) carotid artery dataset collected from multiple sites, the proposed system was shown to better automatically segment the vessel wall than traditional vessel wall segmentation methods or standard convolutional neural network approaches. In addition, a segmentation uncertainty score was estimated to effectively identify slices likely to have errors and prompt manual confirmation of the segmentation. This robust vessel wall segmentation system has applications in different vascular beds and will facilitate vessel wall feature extraction and cardiovascular risk assessment.

Quantitative assessment of the intracranial vasculature in an older adult population using iCafe.

Comprehensive quantification of intracranial artery features may help us assess and understand variations of blood supply during brain development and aging. We analyzed vasculature features of 163 participants (age 56-85 years, mean of 71) from a community study to investigate if any of the features varied with age. Three-dimensional time-of-flight magnetic resonance angiography images of these participants were processed in IntraCranial artery feature extraction technique (a recently developed technique to obtain quantitative features of arteries) to divide intracranial vasculatures into anatomical segments and generate 8 morphometry and intensity features for each segment. Overall, increase in age was found negatively associated with number of branches and average order of intracranial arteries while positively associated with tortuosity, which remained after adjusting for cardiovascular risk factors. The associations with number of branches and average order were consistently found between 3 main intracranial artery regions, whereas the association with tortuosity appeared to be present only in middle cerebral artery/distal arteries. The combination of time-of-flight magnetic resonance angiography and IntraCranial artery feature extraction technique may provide an effective way to study vascular conditions and changes in the aging brain.

Uncertainty-Based Active Learning via Sparse Modeling for Image Classification.

Uncertainty sampling-based active learning has been well studied for selecting informative samples to improve the performance of a classifier. In batch-mode active learning, a batch of samples are selected for a query at the same time. The samples with top uncertainty are encouraged to be selected. However, this selection strategy ignores the relations among the samples, because the selected samples may have much redundant information with each other. This paper addresses this problem by proposing a novel method that combines uncertainty, diversity, and density via sparse modeling in the sample selection. We use sparse linear combination to represent the uncertainty of unlabeled pool data with Gaussian kernels, in which the diversity and density are well incorporated. The selective sampling method is proposed before optimization to reduce the representation error. To deal with ${l}_{0}$ norm constraint in the sparse problem, two approximated approaches are adopted for efficient optimization. Four image classification data sets are used for evaluation. Extensive experiments related to batch size, feature space, seed size, significant analysis, data transform, and time efficiency demonstrate the advantages of the proposed method.

Quantification of morphometry and intensity features of intracranial arteries from 3D TOF MRA using the intracranial artery feature extraction (iCafe): A reproducibility study.

BACKGROUND: Accurate and reliable vascular features extracted from 3D time-of-flight (TOF) magnetic resonance angiography (MRA) can help evaluate cerebral vascular diseases and conditions. The goal of this study was to evaluate the reproducibility of an intracranial artery feature extraction (iCafe) algorithm for quantitative analysis of intracranial arteries from TOF MRA. METHODS: Twenty-four patients with known intracranial artery stenosis were recruited and underwent two separate MRA scans within 2 weeks of each other. Each dataset was blinded to associated imaging and clinical data and then processed independently using iCafe. Inter-scan reproducibility analysis was performed on the 24 pairs of scans while intra-/inter-operator reproducibility and stenosis detection were assessed on 8 individual MRA scans. After tracing the vessels visualized on TOF MRA, iCafe was used to automatically extract the locations with stenosis and eight other vascular features. The vascular features included the following six morphometry and two signal intensity features: artery length (total, distal, and proximal), volume, number of branches, average radius of the M1 segment of the middle cerebral artery, and average normalized intensity of all arteries and large vertical arteries. A neuroradiologist independently reviewed the images to identify locations of stenosis for the reference standard. Reproducibility of stenosis detection and vascular features was assessed using Cohen's kappa, the intra-class correlation coefficient (ICC), and within-subject coefficient of variation (CV). RESULTS: The segment-based sensitivity of iCafe for stenosis detection ranged from 83.3-91.7% while specificity was 97.4%. Kappa values for inter-scan and intra-operator reproducibility were 0.73 and 0.77, respectively. All vascular features demonstrated excellent inter-scan and intra-operator reproducibility (ICC = 0.91-1.00, and CV = 1.21-8.78% for all markers), and good to excellent inter-operator reproducibility (ICC = 0.76-0.99, and CV = 3.27-15.79% for all markers). CONCLUSION: Intracranial artery features can be reliably quantified from TOF MRA using iCafe to provide both clinical diagnostic assistance and facilitate future investigative quantitative analyses.

DesnowNet: Context-Aware Deep Network for Snow Removal.

Existing learning-based atmospheric particle-removal approaches such as those used for rainy and hazy images are designed with strong assumptions regarding spatial frequency, trajectory, and translucency. However, the removal of snow particles is more complicated because they possess additional attributes of particle size and shape, and these attributes may vary within a single image. Currently, hand-crafted features are still the mainstream for snow removal, making significant generalization difficult to achieve. In response, we have designed a multistage network named DesnowNet to in turn deal with the removal of translucent and opaque snow particles. We also differentiate snow attributes of translucency and chromatic aberration for accurate estimation. Moreover, our approach individually estimates residual complements of the snow-free images to recover details obscured by opaque snow. Additionally, a multi-scale design is utilized throughout the entire network to model the diversity of snow. As demonstrated in the qualitative and quantitative experiments, our approach outperforms state-of-the-art learning-based atmospheric phenomena removal methods and one semantic segmentation baseline on the proposed Snow100K dataset. The results indicate our network would benefit applications involving computer vision and graphics.

Development of a quantitative intracranial vascular features extraction tool on 3D MRA using semiautomated open-curve active contour vessel tracing.

PURPOSE: To develop a quantitative intracranial artery measurement technique to extract comprehensive artery features from time-of-flight MR angiography (MRA). METHODS: By semiautomatically tracing arteries based on an open-curve active contour model in a graphical user interface, 12 basic morphometric features and 16 basic intensity features for each artery were identified. Arteries were then classified as one of 24 types using prediction from a probability model. Based on the anatomical structures, features were integrated within 34 vascular groups for regional features of vascular trees. Eight 3D MRA acquisitions with intracranial atherosclerosis were assessed to validate this technique. RESULTS: Arterial tracings were validated by an experienced neuroradiologist who checked agreement at bifurcation and stenosis locations. This technique achieved 94% sensitivity and 85% positive predictive values (PPV) for bifurcations, and 85% sensitivity and PPV for stenosis. Up to 1,456 features, such as length, volume, and averaged signal intensity for each artery, as well as vascular group in each of the MRA images, could be extracted to comprehensively reflect characteristics, distribution, and connectivity of arteries. Length for the M1 segment of the middle cerebral artery extracted by this technique was compared with reviewer-measured results, and the intraclass correlation coefficient was 0.97. CONCLUSION: A semiautomated quantitative method to trace, label, and measure intracranial arteries from 3D-MRA was developed and validated. This technique can be used to facilitate quantitative intracranial vascular research, such as studying cerebrovascular adaptation to aging and disease conditions. Magn Reson Med 79:3229-3238, 2018. © 2017 International Society for Magnetic Resonance in Medicine.

Detecting objects in image sequences using rule-based control in an active contour model.

A method is presented for tracking object boundaries in sequential images using an active contour model, based on fuzzy reasoning rule-based control. Evolution of contour segments is controlled by separate processes based on whether the segment is judged to be inside, outside, or near the boundary of the object, leading to robust boundary detection.

Fractal analysis of region-based vascular change in the normal and non-proliferative diabetic retina.

PURPOSE: Evaluation of normal and abnormal vascular pattern in the human retina using a novel method: quantitative region-based fractal analysis. METHODS: Binary (black/white) vascular patterns of the human retina originating at the optic disc were obtained by semi-automatic computer processing of digital images from 60-degree fundus fluorescein angiography of 5 normal eyes and 5 eyes with non-proliferative diabetic retinopathy (NPDR). As determined by image resolution, vascular patterns included vessels with diameters >or=50 microm and excluded small vessels and capillaries. The density of linearized (i.e., skeletonized) vascular patterns in the macular region versus paramacular region (termed "region-based" linearized vascular pattern) was quantified with the fractal dimension (D(f)) and confirmed by grid intersection (rho(v)). RESULTS: By region-based quantification, D(f) and rho( v) were significantly higher in the normal macular region than in the NPDR macular region (p = 0.008 and p = 0.019, respectively). However, differences in D(f) and rho(v) between the normal and NPDR paramacular regions were not strongly significant (p = 0.168 and p = 0.337, respectively). CONCLUSIONS: Results from the retrospective analytical study demonstrate the feasibility of using quantitative region-based fractal analysis of early-stage vascular disease in the human retina. The results are encouraging for a broader study of diverse patient populations.