Advanced Global Prototypical Segmentation Framework for Few-Shot Hyperspectral Image Classification.

With the advancement of deep learning, related networks have shown strong performance for Hyperspectral Image (HSI) classification. However, these methods face two main challenges in HSI classification: (1) the inability to capture global information of HSI due to the restriction of patch input and (2) insufficient utilization of information from limited labeled samples. To overcome these challenges, we propose an Advanced Global Prototypical Segmentation (AGPS) framework. Within the AGPS framework, we design a patch-free feature extractor segmentation network (SegNet) based on a fully convolutional network (FCN), which processes the entire HSI to capture global information. To enrich the global information extracted by SegNet, we propose a Fusion of Lateral Connection (FLC) structure that fuses the low-level detailed features of the encoder output with the high-level features of the decoder output. Additionally, we propose an Atrous Spatial Pyramid Pooling-Position Attention (ASPP-PA) module to capture multi-scale spatial positional information. Finally, to explore more valuable information from limited labeled samples, we propose an advanced global prototypical representation learning strategy. Building upon the dual constraints of the global prototypical representation learning strategy, we introduce supervised contrastive learning (CL), which optimizes our network with three different constraints. The experimental results of three public datasets demonstrate that our method outperforms the existing state-of-the-art methods.

Classification of cow behavior patterns using inertial measurement units and a fully convolutional network model.

In this study, we aimed to classify 7 cow behavior patterns automatically with an inertial measurement unit (IMU) using a fully convolutional network (FCN) algorithm. Behavioral data of 12 cows were collected by attaching an IMU in a waterproof box on the neck behind the head of each cow. Seven behavior patterns were considered: rub scratching (leg), ruminating-lying, lying, feeding, self-licking, rub scratching (neck), and social licking. To simplify the data and compare classification performance with or without magnetometer data, the 9-axis IMU data were reduced using the square root of the sum of squares to develop 2 datasets. Comparing the classification accuracy of the 3 models using a window size of 64 with 6-axis data and a window size of 128 with both 6-axis and 9-axis data, the best overall accuracy (83.75%) was achieved using the FCN model with a window size of 128 (12.8 s) using all IMU data. This model achieved classification accuracies of 83.2, 96.5, 92.8, 98.1, 82.9, 87.2, and 45.2% for ruminating-lying, lying, feeding, rub scratching (leg), self-licking, rub scratching (neck), and social licking, respectively. As a sequence of varied and intensive movement, the classification accuracy of behavior patterns related to skin disease was lower; better classification of these behavior patterns could be achieved with full IMU data and a larger window size. In the future, additional data will take into account different data types, such as audio and video data, to further enhance performance. In addition, an adaptive sliding window size will be used to improve model performance.

Cascaded Segmentation U-Net for Quality Evaluation of Scraping Workpiece.

In the terms of industry, the hand-scraping method is a key technology for achieving high precision in machine tools, and the quality of scraping workpieces directly affects the accuracy and service life of the machine tool. However, most of the quality evaluation of the scraping workpieces is carried out by the scraping worker's subjective judgment, which results in differences in the quality of the scraping workpieces and is time-consuming. Hence, in this research, an edge-cloud computing system was developed to obtain the relevant parameters, which are the percentage of point (POP) and the peak point per square inch (PPI), for evaluating the quality of scraping workpieces. On the cloud computing server-side, a novel network called cascaded segmentation U-Net is proposed to high-quality segment the height of points (HOP) (around 40 µm height) in favor of small datasets training and then carries out a post-processing algorithm that automatically calculates POP and PPI. This research emphasizes the architecture of the network itself instead. The design of the components of our network is based on the basic idea of identity function, which not only solves the problem of the misjudgment of the oil ditch and the residual pigment but also allows the network to be end-to-end trained effectively. At the head of the network, a cascaded multi-stage pixel-wise classification is designed for obtaining more accurate HOP borders. Furthermore, the "Cross-dimension Compression" stage is used to fuse high-dimensional semantic feature maps across the depth of the feature maps into low-dimensional feature maps, producing decipherable content for final pixel-wise classification. Our system can achieve an error rate of 3.7% and 0.9 points for POP and PPI. The novel network achieves an Intersection over Union (IoU) of 90.2%.

Prehensile and Non-Prehensile Robotic Pick-and-Place of Objects in Clutter Using Deep Reinforcement Learning.

In this study, we develop a framework for an intelligent and self-supervised industrial pick-and-place operation for cluttered environments. Our target is to have the agent learn to perform prehensile and non-prehensile robotic manipulations to improve the efficiency and throughput of the pick-and-place task. To achieve this target, we specify the problem as a Markov decision process (MDP) and deploy a deep reinforcement learning (RL) temporal difference model-free algorithm known as the deep Q-network (DQN). We consider three actions in our MDP; one is 'grasping' from the prehensile manipulation category and the other two are 'left-slide' and 'right-slide' from the non-prehensile manipulation category. Our DQN is composed of three fully convolutional networks (FCN) based on the memory-efficient architecture of DenseNet-121 which are trained together without causing any bottleneck situations. Each FCN corresponds to each discrete action and outputs a pixel-wise map of affordances for the relevant action. Rewards are allocated after every forward pass and backpropagation is carried out for weight tuning in the corresponding FCN. In this manner, non-prehensile manipulations are learnt which can, in turn, lead to possible successful prehensile manipulations in the near future and vice versa, thus increasing the efficiency and throughput of the pick-and-place task. The Results section shows performance comparisons of our approach to a baseline deep learning approach and a ResNet architecture-based approach, along with very promising test results at varying clutter densities across a range of complex scenario test cases.

Multi-Color Space Network for Salient Object Detection.

The salient object detection (SOD) technology predicts which object will attract the attention of an observer surveying a particular scene. Most state-of-the-art SOD methods are top-down mechanisms that apply fully convolutional networks (FCNs) of various structures to RGB images, extract features from them, and train a network. However, owing to the variety of factors that affect visual saliency, securing sufficient features from a single color space is difficult. Therefore, in this paper, we propose a multi-color space network (MCSNet) to detect salient objects using various saliency cues. First, the images were converted to HSV and grayscale color spaces to obtain saliency cues other than those provided by RGB color information. Each saliency cue was fed into two parallel VGG backbone networks to extract features. Contextual information was obtained from the extracted features using atrous spatial pyramid pooling (ASPP). The features obtained from both paths were passed through the attention module, and channel and spatial features were highlighted. Finally, the final saliency map was generated using a step-by-step residual refinement module (RRM). Furthermore, the network was trained with a bidirectional loss to supervise saliency detection results. Experiments on five public benchmark datasets showed that our proposed network achieved superior performance in terms of both subjective results and objective metrics.

Segmentation of liver tumors with abdominal computed tomography using fully convolutional networks.

BACKGROUND: Dividing liver organs or lesions depicting on computed tomography (CT) images could be applied to help tumor staging and treatment. However, most existing image segmentation technologies use manual or semi-automatic analysis, making the analysis process costly and time-consuming. OBJECTIVE: This research aims to develop and apply a deep learning network architecture to segment liver tumors automatically after fine tuning parameters. METHODS AND MATERIALS: The medical imaging is obtained from the International Symposium on Biomedical Imaging (ISBI), which includes 3D abdominal CT scans of 131 patients diagnosed with liver tumors. From these CT scans, there are 7,190 2D CT images along with the labeled binary images. The labeled binary images are regarded as gold standard for evaluation of the segmented results by FCN (Fully Convolutional Network). The backbones of FCN are extracted from Xception, InceptionresNetv2, MobileNetv2, ResNet18, ResNet50 in this study. Meanwhile, the parameters including optimizers (SGDM and ADAM), size of epoch, and size of batch are investigated. CT images are randomly divided into training and testing sets using a ratio of 9:1. Several evaluation indices including Global Accuracy, Mean Accuracy, Mean IoU (Intersection over Union), Weighted IoU and Mean BF Score are applied to evaluate tumor segmentation results in the testing images. RESULTS: The Global Accuracy, Mean Accuracy, Mean IoU, Weighted IoU, and Mean BF Scores are 0.999, 0.969, 0.954, 0.998, 0.962 using ResNet50 in FCN with optimizer SGDM, batch size 12, and epoch 9. It is important to fine tuning the parameters in FCN model. Top 20 FNC models enable to achieve higher tumor segmentation accuracy with Mean IoU over 0.900. The occurred frequency of InceptionresNetv2, MobileNetv2, ResNet18, ResNet50, and Xception are 9, 6, 3, 5, and 2 times. Therefore, the InceptionresNetv2 has higher performance than others. CONCLUSIONS: This study develop and test an automated liver tumor segmentation model based on FCN. Study results demonstrate that many deep learning models including InceptionresNetv2, MobileNetv2, ResNet18, ResNet50, and Xception have high potential to segment liver tumors from CT images with accuracy exceeding 90%. However, it is still difficult to accurately segment tiny and small size tumors by FCN models.

A coronary artery CTA segmentation approach based on deep learning.

Presence of plaque and coronary artery stenosis are the main causes of coronary heart disease. Detection of plaque and coronary artery segmentation have become the first choice in detecting coronary artery disease. The purpose of this study is to investigate a new method for plaque detection and automatic segmentation and diagnosis of coronary arteries and to test its feasibility of applying to clinical medical image diagnosis. A multi-model fusion coronary CT angiography (CTA) vessel segmentation method is proposed based on deep learning. The method includes three network layer models namely, an original 3-dimensional full convolutional network (3D FCN) and two networks that embed the attention gating (AG) model in the original 3D FCN. Then, the prediction results of the three networks are merged by using the majority voting algorithm and thus the final prediction result of the networks is obtained. In the post-processing stage, the level set function is used to further iteratively optimize the results of network fusion prediction. The JI (Jaccard index) and DSC (Dice similarity coefficient) scores are calculated to evaluate accuracy of blood vessel segmentations. Applying to a CTA dataset of 20 patients, accuracy of coronary blood vessel segmentation using FCN, FCN-AG1, FCN-AG2 network and the fusion method are tested. The average values of JI and DSC of using the first three networks are (0.7962, 0.8843), (0.8154, 0.8966) and (0.8119, 0.8936), respectively. When using new fusion method, average JI and DSC of segmentation results increase to (0.8214, 0.9005), which are better than the best result of using FCN, FCN-AG1 and FCN-AG2 model independently.

Time Series Classification with InceptionFCN.

Deep neural networks (DNN) have proven to be efficient in computer vision and data classification with an increasing number of successful applications. Time series classification (TSC) has been one of the challenging problems in data mining in the last decade, and significant research has been proposed with various solutions, including algorithm-based approaches as well as machine and deep learning approaches. This paper focuses on combining the two well-known deep learning techniques, namely the Inception module and the Fully Convolutional Network. The proposed method proved to be more efficient than the previous state-of-the-art InceptionTime method. We tested our model on the univariate TSC benchmark (the UCR/UEA archive), which includes 85 time-series datasets, and proved that our network outperforms the InceptionTime in terms of the training time and overall accuracy on the UCR archive.

A Fully Convolutional Network-Based Tube Contour Detection Method Using Multi-Exposure Images.

The tube contours in two-dimensional images are important cues for optical three-dimensional reconstruction. Aiming at the practical problems encountered in the application of tube contour detection under complex background, a fully convolutional network (FCN)-based tube contour detection method is proposed. Multi-exposure (ME) images are captured as the input of FCN in order to get information of tube contours in different dynamic ranges, and the U-Net type architecture is adopted by the FCN to achieve pixel-level dense classification. In addition, we propose a new loss function that can help eliminate the adverse effects caused by the positional deviation and jagged morphology of tube contour labels. Finally, we introduce a new dataset called multi-exposure tube contour dataset (METCD) and a new evaluation metric called dilate inaccuracy at optimal dataset scale (DIA-ODS) to reach an overall evaluation of our proposed method. The experimental results show that the proposed method can effectively improve the integrity and accuracy of tube contour detection in complex scenes.

Deep Learning Enabled Strain Mapping of Single-Atom Defects in Two-Dimensional Transition Metal Dichalcogenides with Sub-Picometer Precision.

Two-dimensional (2D) materials offer an ideal platform to study the strain fields induced by individual atomic defects, yet challenges associated with radiation damage have so far limited electron microscopy methods to probe these atomic-scale strain fields. Here, we demonstrate an approach to probe single-atom defects with sub-picometer precision in a monolayer 2D transition metal dichalcogenide, WSe2-2xTe2x. We utilize deep learning to mine large data sets of aberration-corrected scanning transmission electron microscopy images to locate and classify point defects. By combining hundreds of images of nominally identical defects, we generate high signal-to-noise class averages which allow us to measure 2D atomic spacings with up to 0.2 pm precision. Our methods reveal that Se vacancies introduce complex, oscillating strain fields in the WSe2-2xTe2x lattice that correspond to alternating rings of lattice expansion and contraction. These results indicate the potential impact of computer vision for the development of high-precision electron microscopy methods for beam-sensitive materials.

Alberta Stroke Program Early CT Score Calculation Using the Deep Learning-Based Brain Hemisphere Comparison Algorithm.

OBJECTIVES: The Alberta Stroke Program Early Computed Tomography Score (ASPECTS) is a promising tool for the evaluation of stroke expansion to determine suitability for reperfusion therapy. The aim of this study was to validate deep learning-based ASPECTS calculation software that utilizes a three-dimensional fully convolutional network-based brain hemisphere comparison algorithm (3D-BHCA). MATERIALS AND METHODS: We retrospectively collected head non-contrast computed tomography (CT) data from 71 patients with acute ischemic stroke and 80 non-stroke patients. The results for ASPECTS on CT assessed by 5 stroke neurologists and by the 3D-BHCA model were compared with the ground truth by means of region-based and score-based analyses. RESULTS: In total, 151 patients and 3020 (151 × 20) ASPECTS regions were investigated. Median time from onset to CT was 195 min in the stroke patients. In region-based analysis, the sensitivity (0.80), specificity (0.97), and accuracy (0.96) of the 3D-BHCA model were superior to those of stroke neurologists. The sensitivity (0.98), specificity (0.92), and accuracy (0.97) of dichotomized ASPECTS > 5 analysis and the intraclass correlation coefficient (0.90) in total score-based analysis of the 3D-BHCA model were superior to those of stroke neurologists overall. When patients with stroke were stratified by onset-to-CT time, the 3D-BHCA model exhibited the highest performance to calculate ASPECTS, even in the earliest time period. CONCLUSIONS: The automated ASPECTS calculation software we developed using a deep learning-based algorithm was superior or equal to stroke neurologists in performing ASPECTS calculation in patients with acute stroke and non-stroke patients.

Automated computer-assisted detection system for cerebral aneurysms in time-of-flight magnetic resonance angiography using fully convolutional network.

BACKGROUND: As the rupture of cerebral aneurysm may lead to fatal results, early detection of unruptured aneurysms may save lives. At present, the contrast-unenhanced time-of-flight magnetic resonance angiography is one of the most commonly used methods for screening aneurysms. The computer-assisted detection system for cerebral aneurysms can help clinicians improve the accuracy of aneurysm diagnosis. As fully convolutional network could classify the image pixel-wise, its three-dimensional implementation is highly suitable for the classification of the vascular structure. However, because the volume of blood vessels in the image is relatively small, 3D convolutional neural network does not work well for blood vessels. RESULTS: The presented study developed a computer-assisted detection system for cerebral aneurysms in the contrast-unenhanced time-of-flight magnetic resonance angiography image. The system first extracts the volume of interest with a fully automatic vessel segmentation algorithm, then uses 3D-UNet-based fully convolutional network to detect the aneurysm areas. A total of 131 magnetic resonance angiography image data are used in this study, among which 76 are training sets, 20 are internal test sets and 35 are external test sets. The presented system obtained 94.4% sensitivity in the fivefold cross-validation of the internal test sets and obtained 82.9% sensitivity with 0.86 false positive/case in the detection of the external test sets. CONCLUSIONS: The proposed computer-assisted detection system can automatically detect the suspected aneurysm areas in contrast-unenhanced time-of-flight magnetic resonance angiography images. It can be used for aneurysm screening in the daily physical examination.

Deep Learning for Pulmonary Image Analysis: Classification, Detection, and Segmentation.

Image-based computer-aided diagnosis (CAD) algorithms by the use of convolutional neural network (CNN) which do not require the image-feature extractor are powerful compared with conventional feature-based CAD algorithms which require the image-feature extractor for classification of lung abnormalities. Moreover, computer-aided detection and segmentation algorithms by the use of CNN are useful for analysis of lung abnormalities. Deep learning will improve the performance of CAD systems dramatically. Therefore, they will change the roles of radiologists in the near future. In this article, we introduce development and evaluation of such image-based CAD algorithms for various kinds of lung abnormalities such as lung nodules and diffuse lung diseases.

Mapping and Discriminating Rural Settlements Using Gaofen-2 Images and a Fully Convolutional Network.

New ongoing rural construction has resulted in an extensive mixture of new settlements with old ones in the rural areas of China. Understanding the spatial characteristic of these rural settlements is of crucial importance as it provides essential information for land management and decision-making. Despite a great advance in High Spatial Resolution (HSR) satellite images and deep learning techniques, it remains a challenging task for mapping rural settlements accurately because of their irregular morphology and distribution pattern. In this study, we proposed a novel framework to map rural settlements by leveraging the merits of Gaofen-2 HSR images and representation learning of deep learning. We combined a dilated residual convolutional network (Dilated-ResNet) and a multi-scale context subnetwork into an end-to-end architecture in order to learn high resolution feature representations from HSR images and to aggregate and refine the multi-scale features extracted by the aforementioned network. Our experiment in Tongxiang city showed that the proposed framework effectively mapped and discriminated rural settlements with an overall accuracy of 98% and Kappa coefficient of 85%, achieving comparable and improved performance compared to other existing methods. Our results bring tangible benefits to support other convolutional neural network (CNN)-based methods in accurate and timely rural settlement mapping, particularly when up-to-date ground truth is absent. The proposed method does not only offer an effective way to extract rural settlement from HSR images but open a new opportunity to obtain spatial-explicit understanding of rural settlements.

A Deep-Learning Method for Radar Micro-Doppler Spectrogram Restoration.

Radio frequency interference, which makes it difficult to produce high-quality radar spectrograms, is a major issue for micro-Doppler-based human activity recognition (HAR). In this paper, we propose a deep-learning-based method to detect and cut out the interference in spectrograms. Then, we restore the spectrograms in the cut-out region. First, a fully convolutional neural network (FCN) is employed to detect and remove the interference. Then, a coarse-to-fine generative adversarial network (GAN) is proposed to restore the part of the spectrogram that is affected by the interferences. The simulated motion capture (MOCAP) spectrograms and the measured radar spectrograms with interference are used to verify the proposed method. Experimental results from both qualitative and quantitative perspectives show that the proposed method can mitigate the interference and restore high-quality radar spectrograms. Furthermore, the comparison experiments also demonstrate the efficiency of the proposed approach.

Novel Method of Semantic Segmentation Applicable to Augmented Reality.

This paper proposes a novel method of semantic segmentation, consisting of modified dilated residual network, atrous pyramid pooling module, and backpropagation, that is applicable to augmented reality (AR). In the proposed method, the modified dilated residual network extracts a feature map from the original images and maintains spatial information. The atrous pyramid pooling module places convolutions in parallel and layers feature maps in a pyramid shape to extract objects occupying small areas in the image; these are converted into one channel using a 1 × 1 convolution. Backpropagation compares the semantic segmentation obtained through convolution from the final feature map with the ground truth provided by a database. Losses can be reduced by applying backpropagation to the modified dilated residual network to change the weighting. The proposed method was compared with other methods on the Cityscapes and PASCAL VOC 2012 databases. The proposed method achieved accuracies of 82.8 and 89.8 mean intersection over union (mIOU) and frame rates of 61 and 64.3 frames per second (fps) for the Cityscapes and PASCAL VOC 2012 databases, respectively. These results prove the applicability of the proposed method for implementing natural AR applications at actual speeds because the frame rate is greater than 60 fps.

SoC FPGA Accelerated Sub-Optimized Binary Fully Convolutional Neural Network for Robotic Floor Region Segmentation.

In this article, a new Binary Fully Convolutional Neural Network (B-FCN) based on Taguchi method sub-optimization for the segmentation of robotic floor regions, which can precisely distinguish floor regions in complex indoor environments is proposed. This methodology is quite suitable for robot vision in an embedded platform and the segmentation accuracy is up to 84.80% on average. A total of 6000 training datasets were used to improve the accuracy and reach convergence. On the other hand, to reach real-time computation, a PYNQ FPGA platform with heterogeneous computing acceleration was used to accelerate the proposed B-FCN architecture. Overall, robots would benefit from better navigation and route planning in our approach. The FPGA synthesis of our binarization method indicates an efficient reduction in the BRAM size to 0.5-1% and also GOPS/W is sufficiently high. Notably, the proposed faster architecture is ideal for low power embedded devices that need to solve the shortest path problem, path searching, and motion planning.

Monitoring of Assembly Process Using Deep Learning Technology.

Monitoring the assembly process is a challenge in the manual assembly of mass customization production, in which the operator needs to change the assembly process according to different products. If an assembly error is not immediately detected during the assembly process of a product, it may lead to errors and loss of time and money in the subsequent assembly process, and will affect product quality. To monitor assembly process, this paper explored two methods: recognizing assembly action and recognizing parts from complicated assembled products. In assembly action recognition, an improved three-dimensional convolutional neural network (3D CNN) model with batch normalization is proposed to detect a missing assembly action. In parts recognition, a fully convolutional network (FCN) is employed to segment, recognize different parts from complicated assembled products to check the assembly sequence for missing or misaligned parts. An assembly actions data set and an assembly segmentation data set are created. The experimental results of assembly action recognition show that the 3D CNN model with batch normalization reduces computational complexity, improves training speed and speeds up the convergence of the model, while maintaining accuracy. Experimental results of FCN show that FCN-2S provides a higher pixel recognition accuracy than other FCNs.

A Metal Artifact Reduction Method Using a Fully Convolutional Network in the Sinogram and Image Domains for Dental Computed Tomography.

The reconstruction quality of dental computed tomography (DCT) is vulnerable to metal implants because the presence of dense metallic objects causes beam hardening and streak artifacts in the reconstructed images. These metal artifacts degrade the images and decrease the clinical usefulness of DCT. Although interpolation-based metal artifact reduction (MAR) methods have been introduced, they may not be efficient in DCT because teeth as well as metallic objects have high X-ray attenuation. In this study, we investigated an effective MAR method based on a fully convolutional network (FCN) in both sinogram and image domains. The method consisted of three main steps: (1) segmentation of the metal trace, (2) FCN-based restoration in the sinogram domain, and (3) FCN-based restoration in image domain followed by metal insertion. We performed a computational simulation and an experiment to investigate the image quality and evaluated the effectiveness of the proposed method. The results of the proposed method were compared with those obtained by the normalized MAR method and the deep learning-based MAR algorithm in the sinogram domain with respect to the root-mean-square error and the structural similarity. Our results indicate that the proposed MAR method significantly reduced the presence of metal artifacts in DCT images and demonstrated better image performance than those of the other algorithms in reducing the streak artifacts without introducing any contrast anomaly.

Segmentation of Masses on Mammograms Using Data Augmentation and Deep Learning.

The diagnosis of breast cancer in early stage is essential for successful treatment. Detection can be performed in several ways, the most common being through mammograms. The projections acquired by this type of examination are directly affected by the composition of the breast, which density can be similar to the suspicious masses, being a challenge the identification of malignant lesions. In this article, we propose a computer-aided detection (CAD) system to aid in the diagnosis of masses in digitized mammograms using a model based in the U-Net, allowing specialists to monitor the lesion over time. Unlike most of the studies, we propose the use of an entire base of digitized mammograms using normal, benign, and malignant cases. Our research is divided into four stages: (1) pre-processing, with the removal of irrelevant information, enhancement of the contrast of 7989 images of the Digital Database for Screening Mammography (DDSM), and obtaining regions of interest. (2) Data augmentation, with horizontal mirroring, zooming, and resizing of images; (3) training, with tests of six-based U-Net models, with different characteristics; (4) testing, evaluating four metrics, accuracy, sensitivity, specificity, and Dice Index. The tested models obtained different results regarding the assessed parameters. The best model achieved a sensitivity of 92.32%, specificity of 80.47%, accuracy of 85.95% Dice Index of 79.39%, and AUC of 86.40%. Even using a full base without case selection bias, the results obtained demonstrate that the use of a complete database can provide knowledge to the CAD expert.