A bidirectional registration neural network for cardiac motion tracking using cine MRI images.

Using cine magnetic resonance imaging (cine MRI) images to track cardiac motion helps users to analyze the myocardial strain, and is of great importance in clinical applications. At present, most of the automatic deep learning-based motion tracking methods compare two images without considering temporal information between MRI frames, which easily leads to the lack of consistency of the generated motion fields. Even though a small number of works take into account the temporal factor, they are usually computationally intensive or have limitations on image length. To solve this problem, we propose a bidirectional convolution neural network for motion tracking of cardiac cine MRI images. This network leverages convolutional blocks to extract spatial features from three-dimensional (3D) image registration pairs, and models the temporal relations through a bidirectional recurrent neural network to obtain the Lagrange motion field between the reference image and other images. Compared with previous pairwise registration methods, the proposed method can automatically learn spatiotemporal information from multiple images with fewer parameters. We evaluated our model on three public cardiac cine MRI datasets. The experimental results demonstrated that the proposed method can significantly improve the motion tracking accuracy. The average Dice coefficient between estimated segmentation and manual segmentation has reached almost 0.85 on the widely used Automatic Cardiac Diagnostic Challenge (ACDC) dataset.

A discontinuity-preserving regularization for deep learning-based cardiac image registration.

Objective. Sliding motion may occur between organs in anatomical regions due to respiratory motion and heart beating. This issue is often neglected in previous studies, resulting in poor image registration performance. A new approach is proposed to handle discontinuity at the boundary and improve registration accuracy.Approach. The proposed discontinuity-preserving regularization (DPR) term can maintain local discontinuities. It leverages the segmentation mask to find organ boundaries and then relaxes the displacement field constraints in these boundary regions. A weakly supervised method using mask dissimilarity loss (MDL) is also proposed. It employs a simple formula to calculate the similarity between the fixed image mask and the deformed moving image mask. These two strategies are added to the loss function during network training to guide the model better to update parameters. Furthermore, during inference time, no segmentation mask information is needed.Main results. Adding the proposed DPR term increases the Dice coefficients by 0.005, 0.009, and 0.081 for three existing registration neural networks CRNet, VoxelMorph, and ViT-V-Net, respectively. It also shows significant improvements in other metrics, including Hausdorff Distance and Average Surface Distance. All quantitative indicator results with MDL have been slightly improved within 1%. After applying these two regularization terms, the generated displacement field is more reasonable at the boundary, and the deformed moving image is closer to the fixed image.Significance. This study demonstrates that the proposed regularization terms can effectively handle discontinuities at the boundaries of organs and improve the accuracy of deep learning-based cardiac image registration methods. Besides, they are generic to be extended to other networks.

Prostate lesion segmentation based on a 3D end-to-end convolution neural network with deep multi-scale attention.

Prostate cancer is one of the deadest cancers among human beings. To better diagnose the prostate cancer, prostate lesion segmentation becomes a very important work, but its progress is very slow due to the prostate lesions small in size, irregular in shape, and blurred in contour. Therefore, automatic prostate lesion segmentation from mp-MRI is a great significant work and a challenging task. However, the most existing multi-step segmentation methods based on voxel-level classification are time-consuming, may introduce errors in different steps and lead to error accumulation. To decrease the computation time, harness richer 3D spatial features, and fuse the multi-level contextual information of mp-MRI, we present an automatic segmentation method in which all steps are optimized conjointly as one step to form our end-to-end convolutional neural network. The proposed end-to-end network DMSA-V-Net consists of two parts: (1) a 3D V-Net is used as the backbone network, it is the first attempt in employing 3D convolutional neural network for CS prostate lesion segmentation, (2) a deep multi-scale attention mechanism is introduced into the 3D V-Net which can highly focus on the ROI while suppressing the redundant background. As a merit, the attention can adaptively re-align the context information between the feature maps at different scales and the saliency maps in high-levels. We performed experiments based on five cross-fold validation with data including 97 patients. The results show that the Dice and sensitivity are 0.7014 and 0.8652 respectively, which demonstrates that our segmentation approach is more significant and accurate compared to other methods.

Automatic 3D CT liver segmentation based on fast global minimization of probabilistic active contour.

PURPOSE: Automatic liver segmentation from computed tomography (CT) images is an essential preprocessing step for computer-aided diagnosis of liver diseases. However, due to the large differences in liver shapes, low-contrast to adjacent tissues, and existence of tumors or other abnormalities, liver segmentation has been very challenging. This study presents an accurate and fast liver segmentation method based on a novel probabilistic active contour (PAC) model and its fast global minimization scheme (3D-FGMPAC), which is explainable as compared with deep learning methods. METHODS: The proposed method first constructs a slice-indexed-histogram to localize the volume of interest (VOI) and estimate the probability that a voxel belongs to the liver according its intensity. The probabilistic image would be used to initialize the 3D PAC model. Secondly, a new contour indicator function, which is a component of the model, is produced by combining the gradient-based edge detection and Hessian-matrix-based surface detection. Then, a fast numerical scheme derived for the 3D PAC model is performed to evolve the initial probabilistic image into the global minimizer of the model, which is a smoothed probabilistic image showing a distinctly highlighted liver. Next, a simple region-growing strategy is applied to extract the whole liver mask from the smoothed probabilistic image. Finally, a B-spline surface is constructed to fit the patch of the rib cage to prevent possible leakage into adjacent intercostal tissues. RESULTS: The proposed method is evaluated on two public datasets. The average Dice score, volume overlap error, volume difference, symmetric surface distance and volume processing time are 0.96, 7.35%, 0.02%, 1.17 mm and 19.8 s for the Sliver07 dataset, and 0.95, 8.89%, - 0.02 % $-0.02\%$ , 1.45 mm and 23.08 s for the 3Dircadb dataset, respectively. CONCLUSIONS: The proposed fully-automatic approach can effectively segment the liver from low-contrast and complex backgrounds. The quantitative and qualitative results demonstrate that the proposed segmentation method outperforms state-of-the-art traditional automatic liver segmentation algorithms and achieves very competitive performance compared with recent deep leaning-based methods.

Automatic CT liver Couinaud segmentation based on key bifurcation detection with attentive residual hourglass-based cascaded network.

This paper presents an automatic Couinaud segmentation method based on deep learning of key point detection. Assuming that the liver mask has been extracted, the proposed method can automatically divide the liver into eight anatomical segments according to Couinaud's definition. Firstly, an attentive residual hourglass-based cascaded network (ARH-CNet) is proposed to identify six key bifurcation points of the hepatic vascular system. Subsequently, the detected points are used to derive the planes that divide the liver into different functional units, and the caudate lobe is segmented slice-by-slice based on the circles defined by the detected points. We comprehensively evaluate our method on a public dataset from MICCAI 2018. Experiments firstly demonstrate the effectiveness of our landmark detection network ARH-CNet, which is superior to that of two baseline methods, also robust to noisy data. The average error distance of all predicted key points is 4.68 ± 3.17 mm, and the average accuracy of all points is 90% with the detection error distance of 7 mm. We also verify that summation of the corresponding heat-maps can improve the accuracy of point localization. Furthermore, the overlap-based accuracy and the Dice score of our landmark-derived Couinaud segmentation are respectively 91% and 84%, which are better than the performance of the direct segmentation approach and the traditional plane-based method, thus our method can be regarded as a good alternative for automatic Couinaud segmentation.

Optical Pulling Forces Enabled by Hyperbolic Metamaterials.

We propose a novel approach to generating optical pulling forces on a gold nanowire, which are placed inside or above a hyperbolic metamaterial and subjected to plane wave illumination. Two mechanisms are found to induce the optical pulling force, including the concave isofrequency contour of the hyperbolic metamaterial and the excitation of directional surface plasmon polaritons. We systematically study the optical forces under various conditions, including the wavelength, the angle of incidence of light, and the nanowire radius. It is shown that the optical pulling force enabled by hyperbolic metamaterials is broadband and insensitive to the angle of incidence. The mechanisms and results reported here open a new avenue to manipulating nanoscale objects.

Lung-CRNet: A convolutional recurrent neural network for lung 4DCT image registration.

PURPOSE: Deformable image registration (DIR) of lung four-dimensional computed tomography (4DCT) plays a vital role in a wide range of clinical applications. Most of the existing deep learning-based lung 4DCT DIR methods focus on pairwise registration which aims to register two images with large deformation. However, the temporal continuities of deformation fields between phases are ignored. This paper proposes a fast and accurate deep learning-based lung 4DCT DIR approach that leverages the temporal component of 4DCT images. METHODS: We present Lung-CRNet, an end-to-end convolutional recurrent registration neural network for lung 4DCT images and reformulate 4DCT DIR as a spatiotemporal sequence predicting problem in which the input is a sequence of three-dimensional computed tomography images from the inspiratory phase to the expiratory phase in a respiratory cycle. The first phase in the sequence is selected as the only reference image and the rest as moving images. Multiple convolutional gated recurrent units (ConvGRUs) are stacked to capture the temporal clues between images. The proposed network is trained in an unsupervised way using a spatial transformer layer. During inference, Lung-CRNet is able to yield the respective displacement field for each reference-moving image pair in the input sequence. RESULTS: We have trained the proposed network using a publicly available lung 4DCT dataset and evaluated performance on the widely used the DIR-Lab dataset. The mean and standard deviation of target registration error are 1.56 ± 1.05 mm on the DIR-Lab dataset. The computation time for each forward prediction is less than 1 s on average. CONCLUSIONS: The proposed Lung-CRNet is comparable to the existing state-of-the-art deep learning-based 4DCT DIR methods in both accuracy and speed. Additionally, the architecture of Lung-CRNet can be generalized to suit other groupwise registration tasks which align multiple images simultaneously.

Learning U-Net Based Multi-Scale Features in Encoding-Decoding for MR Image Brain Tissue Segmentation.

Accurate brain tissue segmentation of MRI is vital to diagnosis aiding, treatment planning, and neurologic condition monitoring. As an excellent convolutional neural network (CNN), U-Net is widely used in MR image segmentation as it usually generates high-precision features. However, the performance of U-Net is considerably restricted due to the variable shapes of the segmented targets in MRI and the information loss of down-sampling and up-sampling operations. Therefore, we propose a novel network by introducing spatial and channel dimensions-based multi-scale feature information extractors into its encoding-decoding framework, which is helpful in extracting rich multi-scale features while highlighting the details of higher-level features in the encoding part, and recovering the corresponding localization to a higher resolution layer in the decoding part. Concretely, we propose two information extractors, multi-branch pooling, called MP, in the encoding part, and multi-branch dense prediction, called MDP, in the decoding part, to extract multi-scale features. Additionally, we designed a new multi-branch output structure with MDP in the decoding part to form more accurate edge-preserving predicting maps by integrating the dense adjacent prediction features at different scales. Finally, the proposed method is tested on datasets MRbrainS13, IBSR18, and ISeg2017. We find that the proposed network performs higher accuracy in segmenting MRI brain tissues and it is better than the leading method of 2018 at the segmentation of GM and CSF. Therefore, it can be a useful tool for diagnostic applications, such as brain MRI segmentation and diagnosing.

Grayscale image for broadband linear polarization measurement by an ultracompact metasurface.

The polarization of light, the vector nature of electromagnetic waves, is one of the fundamental parameters. Finding a direct and efficient method to measure the state of polarized light is extremely urgent for nano-optical applications. Based on Malus's law, we design an ultracompact metasurface composed of silver nanorods, which is demonstrated to directly measure the state of linear polarization by a grayscale image. Using an ultrathin metasurface, we generate grayscale images with gradient grayscale levels which are linked directly to the polarization state of the incident light. The direction of the linear polarization of incident light can be conveniently and efficiently obtained through extracting the angle of the brightest area of the grayscale image. The ultrathin metasurface operates in the broadband 750-1100 nm spectral range. It is a novel and significant method to analyze the linear polarization state of light, which provides opportunities for various applications, such as polarimetric multispectral imaging and miniaturized polarimeter.

An Explainable System for Diagnosis and Prognosis of COVID-19.

The outbreak of Coronavirus Disease-2019 (COVID-19) has posed a threat to world health. With the increasing number of people infected, healthcare systems, especially those in developing countries, are bearing tremendous pressure. There is an urgent need for the diagnosis of COVID-19 and the prognosis of inpatients. To alleviate these problems, a data-driven medical assistance system is put forward in this article. Based on two real-world data sets in Wuhan, China, the proposed system integrates data from different sources with tools of machine learning (ML) to predict COVID-19 infected probability of suspected patients in their first visit, and then predict mortality of confirmed cases. Rather than choosing an interpretable algorithm, this system separates the explanations from ML models. It can do help to patient triaging and provide some useful advice for doctors.

A new data augmentation method based on local image warping for medical image segmentation.

PURPOSE: The segmentation accuracy of medical images was improved by increasing the number of training samples using a local image warping technique. The performance of the proposed method was evaluated in the segmentation of breast masses, prostate and brain tumors, and lung nodules. METHODS: We propose a simple data augmentation method which is called stochastic evolution (SE). Specifically, the idea of SE stems from our thinking about the deterioration of the diseased tissue and the healing process. In order to simulate this natural process, we implement it according to the local distortion algorithm in image warping. In other words, the irregular deterioration and healing processes of the diseased tissue is simulated according to the direction of the local distortion, thereby producing a natural sample that is indistinguishable by humans. RESULTS: The proposed method is evaluated on four segmentation tasks of breast masses, prostate, brain tumors, and lung nodules. Comparing the experimental results of four segmentation methods based on the UNet segmentation architecture without adding any expanded data during training, the accuracy and the Hausdorff distance obtained in our approach remain almost the same as other methods. However, the dice similarity coefficient (DSC) and sensitivity (SEN) have both improved to some extent. Among them, DSC is increased by 5.2%, 2.8%, 1.0%, and 3.2%, respectively; SEN is increased by 6.9%, 4.3%, 1.2%, and 4.5%, respectively. CONCLUSIONS: Experimental results show that the proposed SE data augmentation method could improve the segmentation accuracy of breast masses, prostate, brain tumors, and lung nodules. The method also shows the robustness with different image datasets and imaging modalities.

Spin-dependent dual-wavelength multiplexing metalens.

The Pancharatnam-Berry (PB) phase is generally utilized to realize a single wavelength spin-dependent function or dual-wavelength functions but operating only in one spin state. A dual-wavelength multifunctional metasurface relying on both spins has been rarely designed due to the rather complicated degrees of freedom to be considered. In this Letter, both dynamic and PB phases are adopted, instead of a pure PB phase, to propose a multiplexing metasurface that can independently and simultaneously manipulate left- and right-handed circularly polarized incidences at dual wavelengths. It is demonstrated experimentally as well as numerically that such spin-dependent dual-wavelength metalenses can make circularly polarized incidences of different wavelengths split into and focus at multi-dimensional positions. Our work demonstrates a new avenue in designing spin-dependent dual-wavelength multifunctional optical devices.

Multi-model Ensemble Learning Architecture Based on 3D CNN for Lung Nodule Malignancy Suspiciousness Classification.

Classification of benign and malignant in lung nodules using chest CT images is a key step in the diagnosis of early-stage lung cancer, as well as an effective way to improve the patients' survival rate. However, due to the diversity of lung nodules and the visual similarity of lung nodules to their surrounding tissues, it is difficult to construct a robust classification model with conventional deep learning-based diagnostic methods. To address this problem, we propose a multi-model ensemble learning architecture based on 3D convolutional neural network (MMEL-3DCNN). This approach incorporates three key ideas: (1) Constructed multi-model network architecture can be well adapted to the heterogeneity of lung nodules. (2) The input that concatenated of the intensity image corresponding to the nodule mask, the original image, and the enhanced image corresponding to which can help training model to extract advanced feature with more discriminative capacity. (3) Select the corresponding model to different nodule size dynamically for prediction, which can improve the generalization ability of the model effectively. In addition, ensemble learning is applied in this paper to further improve the robustness of the nodule classification model. The proposed method has been experimentally verified on the public dataset, LIDC-IDRI. The experimental results show that the proposed MMEL-3DCNN architecture can obtain satisfactory classification results.

Automated labeling of the airway tree in terms of lobes based on deep learning of bifurcation point detection.

This paper presents an automatic lobe-based labeling of airway tree method, which can detect the bifurcation points for reconstructing and labeling the airway tree from a computed tomography image. A deep learning-based network structure is designed to identify the four key bifurcation points. Then, based on the detected bifurcation points, the entire airway tree is reconstructed by a new region-growing method. Finally, with the basic airway tree anatomy and topology knowledge, individual branches of the airway tree are classified into different categories in terms of pulmonary lobes. There are several advantages in our method such as the detection of the bifurcation points does not depend on the segmentation of airway tree and only four bifurcation points need to be manually labeled for each sample to prepare the training dataset. The segmentation of airway tree is guided by the detected points, which overcomes the difficulty of manual seed selection of conventional region-growing algorithm. In addition, the bifurcation points can help analyze the tree structure, which provides a basis for effective airway tree labeling. Experimental results show that our method is fast, stable, and the accuracy of our method is 97.85%, which is higher than that of the traditional skeleton-based method. Graphical Abstract The pipeline of our proposed lobe-based airway tree labeling method. Given a raw CT volume, a neural network structure is designed to predict major bifurcation points of airway tree. Based on the detected points, airway tree is reconstructed and labeled in terms of lobes.

A Two-Stage Convolutional Neural Networks for Lung Nodule Detection.

Early detection of lung cancer is an effective way to improve the survival rate of patients. It is a critical step to have accurate detection of lung nodules in computed tomography (CT) images for the diagnosis of lung cancer. However, due to the heterogeneity of the lung nodules and the complexity of the surrounding environment, it is a challenge to develop a robust nodule detection method. In this study, we propose a two-stage convolutional neural networks (TSCNN) for lung nodule detection. The first stage based on the improved U-Net segmentation network is to establish an initial detection of lung nodules. During this stage, in order to obtain a high recall rate without introducing excessive false positive nodules, we propose a new sampling strategy for training. Simultaneously, a two-phase prediction method is also proposed in this stage. The second stage in the TSCNN architecture based on the proposed dual pooling structure is built into three 3D-CNN classification networks for false positive reduction. Since the network training requires a significant amount of training data, we designed a random mask as the data augmentation method in this study. Furthermore, we have improved the generalization ability of the false positive reduction model by means of ensemble learning. We verified the proposed architecture on the LUNA dataset in our experiments, which showed that the proposed TSCNN architecture did obtain competitive detection performance.

A cascaded dual-pathway residual network for lung nodule segmentation in CT images.

It is difficult to obtain an accurate segmentation due to the variety of lung nodules in computed tomography (CT) images. In this study, we propose a data-driven model, called the Cascaded Dual-Pathway Residual Network (CDP-ResNet) to improve the segmentation of lung nodules in the CT images. Our approach incorporates the multi-view and multi-scale features of different nodules from CT images. The proposed residual block based dual-path network extracts local features and rich contextual information of lung nodules. In addition, we designed an improved weighted sampling strategy to select training samples based on the edge. The proposed method was extensively evaluated on an LIDC dataset, which contains 986 nodules. Experimental results show that the CDP-ResNet achieves superior segmentation performance with an average DICE score (standard deviation) of 81.58% (11.05) on the LIDC dataset. Moreover, we compared our results with those of four radiologists on the same dataset. The comparison shows that the CDP-ResNet is slightly better than human experts in terms of segmentation accuracy. Meanwhile, the proposed segmentation method outperforms existing methods.

Automatic Labeling of MR Brain Images Through the Hashing Retrieval Based Atlas Forest.

The multi-atlas method is one of the efficient and common automatic labeling method, which uses the prior information provided by expert-labeled images to guide the labeling of the target. However, most multi-atlas-based methods depend on the registration that may not give the correct information during the label propagation. To address the issue, we designed a new automatic labeling method through the hashing retrieval based atlas forest. The proposed method propagates labels without registration to reduce the errors, and constructs a target-oriented learning model to integrate information among the atlases. This method innovates a coarse classification strategy to preprocess the dataset, which retains the integrity of dataset and reduces computing time. Furthermore, the method considers each voxel in the atlas as a sample and encodes these samples with hashing for the fast sample retrieval. In the stage of labeling, the method selects suitable samples through hashing learning and trains atlas forests by integrating the information from the dataset. Then, the trained model is used to predict the labels of the target. Experimental results on two datasets illustrated that the proposed method is promising in the automatic labeling of MR brain images.

Dual-mode subwavelength trapping by plasmonic tweezers based on V-type nanoantennas.

We propose novel plasmonic tweezers based on silver V-type nanoantennas placed on a conducting ground layer, which can effectively mitigate the plasmonic heating effect and thus enable subwavelength plasmonic trapping in the near-infrared region. Using the centroid algorithm to analyze the motion of trapped spheres, we can experimentally extract the value of optical trapping potential. The result confirms that the plasmonic tweezers have a dual-mode subwavelength trapping capability when the incident laser beam is linearly polarized along two orthogonal directions. We have also performed full-wave simulations, which agree with the experimental data very well in terms of spectral response and trapping potential. It is expected that the dual-mode subwavelength trapping can be used in non-contact manipulations of a single nanoscale object, such as a biomolecule or quantum dot, and find important applications in biology, life science, and applied physics.

Enhanced asymmetric transmissions attributed to the cavity coupling hybrid resonance in a continuous omega-shaped metamaterial.

In this paper, the infinite-length metallic bar is folded to a continuous omega-shaped resonator and then arranged as a bi-layer metamaterial, which presents a hybrid resonance and a Fabry-Perot-like cavity mode. The asymmetric transmission (AT) for linearly polarized light is powerfully enhanced at a near-infrared regime by strongly coupling the hybrid resonance to the cavity, with the maximum value of the high-efficiency AT effect reaching 0.8 at around 1364 nm. At this near-infrared band, such a high-efficiency AT effect has never been realized previously by a bi-layer metamaterial. More importantly, we demonstrate that our design is robust to the misalignments, which greatly decreases the difficulties in sample fabrications. Accordingly, the proposed omega-shaped metamaterial provides potential applications in designing polarization filters, polarization switches, and other nano-devices.

Experimental verification of asymmetric transmission in continuous omega-shaped metamaterials.

A bi-layer continuous omega-shaped metamaterial was proposed and fabricated to measure the asymmetric transmission (AT) effect of a linearly polarized light at near-infrared region. The metamaterial was fabricated by the electron-beam lithography method, and the AT effect was demonstrated by the difference between total transmittances in the two opposite propagation directions for x-/y-polarized incident light. The experimental results were confirmed by the full-wave simulated results. Importantly, we also experimentally demonstrated that the AT effect is robust against the misalignments between the first and the second omega-shaped layers. Accordingly, the successfully prepared sample and its characterization provide a bright future for applications in light-controlled switchers and optical diodes in on-chip optical systems and information communication systems.