Synthetic-to-real: instance segmentation of clinical cluster cells with unlabeled synthetic training.

MOTIVATION: The presence of tumor cell clusters in pleural effusion may be a signal of cancer metastasis. The instance segmentation of single cell from cell clusters plays a pivotal role in cluster cell analysis. However, current cell segmentation methods perform poorly for cluster cells due to the overlapping/touching characters of clusters, multiple instance properties of cells, and the poor generalization ability of the models. RESULTS: In this article, we propose a contour constraint instance segmentation framework (CC framework) for cluster cells based on a cluster cell combination enhancement module. The framework can accurately locate each instance from cluster cells and realize high-precision contour segmentation under a few samples. Specifically, we propose the contour attention constraint module to alleviate over- and under-segmentation among individual cell-instance boundaries. In addition, to evaluate the framework, we construct a pleural effusion cluster cell dataset including 197 high-quality samples. The quantitative results show that the numeric result of APmask is > 90%, a more than 10% increase compared with state-of-the-art semantic segmentation algorithms. From the qualitative results, we can observe that our method rarely has segmentation errors.

Deep learning image transmission through a multimode fiber based on a small training dataset.

An improved deep neural network incorporating attention mechanism and DSSIM loss function (AM_U_Net) is used to recover input images with speckles transmitted through a multimode fiber (MMF). The network is trained on a relatively small dataset and demonstrates an optimal reconstruction ability and generalization ability. Furthermore, a bimodal fusion method is developed based on S polarization and P polarization speckles, greatly improving the recognition accuracy. These findings prove that AM_U_Net has remarkable capabilities for information recovery and transfer learning and good tolerance and robustness under different MMF transmission conditions, indicating its significant application potential in medical imaging and secure communication.

Resonance prediction and inverse design of multi-core selective couplers based on neural networks.

Resonance analysis and structural optimization of multi-channel selective fiber couplers currently rely on numerical simulation and manual trial and error, which is very repetitive and time consuming. To realize fast and accurate resonance analysis and calculation, we start with dual-core structures and establish forward classification and regression neural networks to classify and predict different resonance properties, including resonance types, operating wavelength, coupling coefficient, coupling length, 3 dB bandwidth, and conversion efficiency. The pre-trained forward neural networks for dual-core fibers can also realize accurate and fast prediction for multi-core fibers if the mode energy exchange occurs only between one surrounding core and the central core. For the inverse design, a tandem neural network has been constructed by cascading the pre-trained forward neural network and the inverse network to solve the non-uniqueness problem and provide an approach to search for appropriate and desired multi-core structures. The proposed forward and inverse neural networks are efficient and accurate, which provides great convenience for resonance analysis and structural optimization of multi-channel fiber structures and devices.

2D shape reconstruction of irregular particles with deep learning based on interferometric particle imaging.

Interferometric particle imaging (IPI) technology is widely used in the measurement of various particles. Obtaining particle shape information directly by IPI is challenging because of the complex relationship between the speckle distribution of interference-defocused speckle patterns and the shape of the corresponding irregular particles. Considering this challenge, we implement a deep learning method based on the convolutional neural network (CNN) to reconstruct defocused images of sand particles with sparse features. We also introduce the negative Pearson correlation coefficient as the loss function. To verify the feasibility of our method, we implemented it to reconstruct defocused images obtained from IPI experiments. Finally, compared with another common CNN-based structure, we confirmed that our network structure has good performance in the shape reconstruction of irregular particles.

Comparison of aspect ratios of ellipsoidal particles through interferometric out-of-focus images.

Based on the interferometric particle imaging (IPI) technology, we present a method for comparison of aspect ratios of ellipsoidal particles. By simulating the interference in-focus and out-of-focus images of transparent ellipsoidal particles with different aspect ratios, we find that, under the same orientation angle, the larger the particle aspect ratio is, the higher the spatial frequency of the out-of-focus image. The IPI system is established to experimentally acquire the out-of-focus images of the transparent ellipsoidal particles. Because the experimental results agree with the simulation, we propose a method to compare the aspect ratios of ellipsoidal particles using out-of-focus images. The method features potential applications in particle measurements.

Mechanism and characteristics of a tunable dispersion-compensating dual-ring microstructure fiber for different orbital angular momentum modes.

A tunable dual-ring microstructure fiber that can support stable transmission for different orbital angular momentum (OAM) states and possess ultrahigh dispersion coefficients and low confinement losses is proposed and theoretically investigated. The proposed fiber is composed of two high-refractive-index rings and a double-cladding structure. Owing to the central air core and outer cladding, the dual-ring structure can support stable transmission for the OAM states. The mode fields of different OAM states in the inner ring can spread to the outer ring under certain conditions, which leads to high absolute values of dispersion around the coupling wavelengths. By tuning the refractive indices of the dual rings, the proposed fiber can achieve dispersion control for different OAM modes. Moreover, the specially designed two-layer air holes in the inner cladding can affect the mode-coupling coefficients, which are characterized by the effective mode areas and the overlap integral of the electric fields between the resonant ring modes. Therefore, the dispersion curves and operating wavelengths of the OAM modes can be modulated by regulating the physical parameters (the radius of the two-layer air holes or the infiltrated functional materials) of the inner cladding. We built a theoretical model and analyzed the modulation method and mechanism of the dispersion curves based on the coupled mode theory. The theoretical results indicate that the proposed fiber is flexible and has potential dispersion-compensating applications in fiber OAM systems.

Orbital angular momentum generation in a dual-ring fiber based on the phase-shifted coupling mechanism and the interference of supermodes.

Based on the phased-shifted interference between supermodes, a novel method that can directly convert LP01 mode to orbital angular momentum (OAM) mode in a dual-ring microstructure optical fiber is proposed. In this fiber, the resonance between even and odd HE11 modes in inner ring and higher order mode in outer ring will form two pairs of supermodes, and the intensities and phases of the complete superposition mode fields for the involved supermodes created by the resonance at different wavelengths and propagating lengths are investigated and exhibited in this paper. We demonstrate that OAM mode can be generated from π/2-phase-shifted linear combinations of supermodes, and the phase difference of the even and odd higher order eigenmodes can accumulate to π/2 during the coupling process, which is defined as "phase-shifted" conversion. We build a complete theoretical model and systematically analyze the phase-shifted coupling mechanism, and the design principle and optimization method of this fiber are also illustrated in detail. The proposed microstructure fiber is compact, and the OAM mode conversion method is simple and flexible, which could provide a new approach to generate OAM states.

Accuracy improvement of time-of-flight depth measurement by combination of a high-resolution color camera.

Time-of-flight (ToF) cameras can acquire the distance between the sensor and objects with high frame rates, offering bright prospects for ToF cameras in many applications. Low-resolution and depth errors limit the accuracy of ToF cameras, however. In this paper, we present a flexible accuracy improvement method for depth compensation and feature points position correction of ToF cameras. First, a distance-error model of each pixel in the depth image is established to model sinusoidal waves of ToF cameras and compensate for the measured depth data. Second, a more accurate feature point position is estimated with the aid of a high-resolution camera. Experiments evaluate the proposed method, and the result shows the root mean square error is reduced from 4.38 mm to 3.57 mm.

Real-time neural network scheduling of emergency medical mask production during COVID-19.

During the outbreak of the novel coronavirus pneumonia (COVID-19), there is a huge demand for medical masks. A mask manufacturer often receives a large amount of orders that must be processed within a short response time. It is of critical importance for the manufacturer to schedule and reschedule mask production tasks as efficiently as possible. However, when the number of tasks is large, most existing scheduling algorithms require very long computational time and, therefore, cannot meet the needs of emergency response. In this paper, we propose an end-to-end neural network, which takes a sequence of production tasks as inputs and produces a schedule of tasks in a real-time manner. The network is trained by reinforcement learning using the negative total tardiness as the reward signal. We applied the proposed approach to schedule emergency production tasks for a medical mask manufacturer during the peak of COVID-19 in China. Computational results show that the neural network scheduler can solve problem instances with hundreds of tasks within seconds. The objective function value obtained by the neural network scheduler is significantly better than those of existing constructive heuristics, and is close to those of the state-of-the-art metaheuristics whose computational time is unaffordable in practice.

Determination and adjustment of optimal defocus level for fringe projection systems.

The sinusoidal fringe pattern is widely used in fringe projection profilometry. Too much or too little defocusing will affect the quality of sinusoidal fringe patterns and consequently jeopardize the accuracy of measurement results. This paper proposes a method to quantify and ascertain the defocus level by simulations and experiments. By simulating the defocus pattern with a Gaussian low-pass filter, the optimum defocus level of the fringe pattern is determined so that the projected fringe pattern is closer to the sinusoidal function. Then, a method is proposed to adjust the projector to make the projected pattern in the optimal defocus degree. Experiments show the feasibility and the validity of the proposed method, and the accuracy is improved up to 9.9%, compared with the focus-projected pattern.

Distributed leader-following finite-time consensus control for linear multiagent systems under switching topology.

This paper investigates the finite-time consensus problem of leader-following multiagent systems. The dynamical models for all following agents and the leader are assumed the same general form of linear system, and the interconnection topology among the agents is assumed to be switching and undirected. We mostly consider the continuous-time case. By assuming that the states of neighbouring agents are known to each agent, a sufficient condition is established for finite-time consensus via a neighbor-based state feedback protocol. While the states of neighbouring agents cannot be available and only the outputs of neighbouring agents can be accessed, the distributed observer-based consensus protocol is proposed for each following agent. A sufficient condition is provided in terms of linear matrix inequalities to design the observer-based consensus protocol, which makes the multiagent systems achieve finite-time consensus under switching topologies. Then, we discuss the counterparts for discrete-time case. Finally, we provide an illustrative example to show the effectiveness of the design approach.

Masked Generative Light Field Prompting for Pixel-Level Structure Segmentations.

Pixel-level structure segmentations have attracted considerable attention, playing a crucial role in autonomous driving within the metaverse and enhancing comprehension in light field-based machine vision. However, current light field modeling methods fail to integrate appearance and geometric structural information into a coherent semantic space, thereby limiting the capability of light field transmission for visual knowledge. In this paper, we propose a general light field modeling method for pixel-level structure segmentation, comprising a generative light field prompting encoder (LF-GPE) and a prompt-based masked light field pretraining (LF-PMP) network. Our LF-GPE, serving as a light field backbone, can extract both appearance and geometric structural cues simultaneously. It aligns these features into a unified visual space, facilitating semantic interaction. Meanwhile, our LF-PMP, during the pretraining phase, integrates a mixed light field and a multi-view light field reconstruction. It prioritizes considering the geometric structural properties of the light field, enabling the light field backbone to accumulate a wealth of prior knowledge. We evaluate our pretrained LF-GPE on two downstream tasks: light field salient object detection and semantic segmentation. Experimental results demonstrate that LF-GPE can effectively learn high-quality light field features and achieve highly competitive performance in pixel-level segmentation tasks.

Identity-Guided Collaborative Learning for Cloth-Changing Person Reidentification.

Cloth-changing person reidentification (ReID) is a newly emerging research topic aimed at addressing the issues of large feature variations due to cloth-changing and pedestrian view/pose changes. Although significant progress has been achieved by introducing extra information (e.g., human contour sketching information, human body keypoints, and 3D human information), cloth-changing person ReID remains challenging because pedestrian appearance representations can change at any time. Moreover, human semantic information and pedestrian identity information are not fully explored. To solve these issues, we propose a novel identity-guided collaborative learning scheme (IGCL) for cloth-changing person ReID, where the human semantic is effectively utilized and the identity is unchangeable to guide collaborative learning. First, we design a novel clothing attention degradation stream to reasonably reduce the interference caused by clothing information where clothing attention and mid-level collaborative learning are employed. Second, we propose a human semantic attention and body jigsaw stream to highlight the human semantic information and simulate different poses of the same identity. In this way, the extraction features not only focus on human semantic information that is unrelated to the background but are also suitable for pedestrian pose variations. Moreover, a pedestrian identity enhancement stream is proposed to enhance the identity importance and extract more favorable identity robust features. Most importantly, all these streams are jointly explored in an end-to-end unified framework, and the identity is utilized to guide the optimization. Extensive experiments on six public clothing person ReID datasets (LaST, LTCC, PRCC, NKUP, Celeb-reID-light, and VC-Clothes) demonstrate the superiority of the IGCL method. It outperforms existing methods on multiple datasets, and the extracted features have stronger representation and discrimination ability and are weakly correlated with clothing.

Prediction of LncRNA-Protein Interactions Based on Kernel Combinations and Graph Convolutional Networks.

The complexes of long non-coding RNAs bound to proteins can be involved in regulating life activities at various stages of organisms. However, in the face of the growing number of lncRNAs and proteins, verifying LncRNA-Protein Interactions (LPI) based on traditional biological experiments is time-consuming and laborious. Therefore, with the improvement of computing power, predicting LPI has met new development opportunity. In virtue of the state-of-the-art works, a framework called LncRNA-Protein Interactions based on Kernel Combinations and Graph Convolutional Networks (LPI-KCGCN) has been proposed in this article. We first construct kernel matrices by taking advantage of extracting both the lncRNAs and protein concerning the sequence features, sequence similarity features, expression features, and gene ontology. Then reconstruct the existent kernel matrices as the input of the next step. Combined with known LPI interactions, the reconstructed similarity matrices, which can be used as features of the topology map of the LPI network, are exploited in extracting potential representations in the lncRNA and protein space using a two-layer Graph Convolutional Network. The predicted matrix can be finally obtained by training the network to produce scoring matrices w.r.t. lncRNAs and proteins. Different LPI-KCGCN variants are ensemble to derive the final prediction results and testify on balanced and unbalanced datasets. The 5-fold cross-validation shows that the optimal feature information combination on a dataset with 15.5% positive samples has an AUC value of 0.9714 and an AUPR value of 0.9216. On another highly unbalanced dataset with only 5% positive samples, LPI-KCGCN also has outperformed the state-of-the-art works, which achieved an AUC value of 0.9907 and an AUPR value of 0.9267.

Gaze Estimation by Attention-Induced Hierarchical Variational Auto-Encoder.

Appearance-based gaze estimation has been widely studied recently with promising performance. The majority of appearance-based gaze estimation methods are developed under the deterministic frameworks. However, the deterministic gaze estimation methods suffer from large performance drop upon challenging eye images in low-resolution, darkness, partial occlusions, etc. To alleviate this problem, in this article, we alternatively reformulate the appearance-based gaze estimation problem under a generative framework. Specifically, we propose a variational inference model, that is, variational gaze estimation network (VGE-Net), to generate multiple gaze maps as complimentary candidates simultaneously supervised by the ground-truth gaze map. To achieve robust estimation, we adaptively fuse the gaze directions predicted on these candidate gaze maps by a regression network through a simple attention mechanism. Experiments on three benchmarks, that is, MPIIGaze, EYEDIAP, and Columbia, demonstrate that our VGE-Net outperforms state-of-the-art gaze estimation methods, especially on challenging cases. Comprehensive ablation studies also validate the effectiveness of our contributions. The code will be publicly released.

Identification of autism spectrum disorder based on electroencephalography: A systematic review.

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by difficulties in social communication and repetitive and stereotyped behaviors. According to the World Health Organization, about 1 in 100 children worldwide has autism. With the global prevalence of ASD, timely and accurate diagnosis has been essential in enhancing the intervention effectiveness for ASD children. Traditional ASD diagnostic methods rely on clinical observations and behavioral assessment, with the disadvantages of time-consuming and lack of objective biological indicators. Therefore, automated diagnostic methods based on machine learning and deep learning technologies have emerged and become significant since they can achieve more objective, efficient, and accurate ASD diagnosis. Electroencephalography (EEG) is an electrophysiological monitoring method that records changes in brain spontaneous potential activity, which is of great significance for identifying ASD children. By analyzing EEG data, it is possible to detect abnormal synchronous neuronal activity of ASD children. This paper gives a comprehensive review of the EEG-based ASD identification using traditional machine learning methods and deep learning approaches, including their merits and potential pitfalls. Additionally, it highlights the challenges and the opportunities ahead in search of more effective and efficient methods to automatically diagnose autism based on EEG signals, which aims to facilitate automated ASD identification.

Smoke veil prior regularized surgical field desmoking without paired in-vivo data.

Though deep learning-based surgical smoke removal methods have shown significant improvements in effectiveness and efficiency, the lack of paired smoke and smoke-free images in real surgical scenarios limits the performance of these methods. Therefore, methods that can achieve good generalization performance without paired in-vivo data are in high demand. In this work, we propose a smoke veil prior regularized two-stage smoke removal framework based on the physical model of smoke image formation. More precisely, in the first stage, we leverage a reconstruction loss, a consistency loss and a smoke veil prior-based regularization term to perform fully supervised training on a synthetic paired image dataset. Then a self-supervised training stage is deployed on the real smoke images, where only the consistency loss and the smoke veil prior-based loss are minimized. Experiments show that the proposed method outperforms the state-of-the-art ones on synthetic dataset. The average PSNR, SSIM and RMSE values are 21.99±2.34, 0.9001±0.0252 and 0.2151±0.0643, respectively. The qualitative visual inspection on real dataset further demonstrates the effectiveness of the proposed method.

A Semantic-Aware Attention and Visual Shielding Network for Cloth-Changing Person Re-Identification.

Cloth-changing person re-identification (ReID) is a newly emerging research topic that aims to retrieve pedestrians whose clothes are changed. Since the human appearance with different clothes exhibits large variations, it is very difficult for existing approaches to extract discriminative and robust feature representations. Current works mainly focus on body shape or contour sketches, but the human semantic information and the potential consistency of pedestrian features before and after changing clothes are not fully explored or are ignored. To solve these issues, in this work, a novel semantic-aware attention and visual shielding network for cloth-changing person ReID (abbreviated as SAVS) is proposed where the key idea is to shield clues related to the appearance of clothes and only focus on visual semantic information that is not sensitive to view/posture changes. Specifically, a visual semantic encoder is first employed to locate the human body and clothing regions based on human semantic segmentation information. Then, a human semantic attention (HSA) module is proposed to highlight the human semantic information and reweight the visual feature map. In addition, a visual clothes shielding (VCS) module is also designed to extract a more robust feature representation for the cloth-changing task by covering the clothing regions and focusing the model on the visual semantic information unrelated to the clothes. Most importantly, these two modules are jointly explored in an end-to-end unified framework. Extensive experiments demonstrate that the proposed method can significantly outperform state-of-the-art methods, and more robust features can be extracted for cloth-changing persons. Compared with multibiometric unified network (MBUNet) (published in TIP2023), this method can achieve improvements of 17.5% (30.9%) and 8.5% (10.4%) on the LTCC and Celeb-reID datasets in terms of mean average precision (mAP) (rank-1), respectively. When compared with the Swin Transformer (Swin-T), the improvements can reach 28.6% (17.3%), 22.5% (10.0%), 19.5% (10.2%), and 8.6% (10.1%) on the PRCC, LTCC, Celeb, and NKUP datasets in terms of rank-1 (mAP), respectively.

Predicting Demands of COVID-19 Prevention and Control Materials via Co-Evolutionary Transfer Learning.

The novel coronavirus pneumonia (COVID-19) has created great demands for medical resources. Determining these demands timely and accurately is critically important for the prevention and control of the pandemic. However, even if the infection rate has been estimated, the demands of many medical materials are still difficult to estimate due to their complex relationships with the infection rate and insufficient historical data. To alleviate the difficulties, we propose a co-evolutionary transfer learning (CETL) method for predicting the demands of a set of medical materials, which is important in COVID-19 prevention and control. CETL reuses material demand knowledge not only from other epidemics, such as severe acute respiratory syndrome (SARS) and bird flu but also from natural and manmade disasters. The knowledge or data of these related tasks can also be relatively few and imbalanced. In CETL, each prediction task is implemented by a fuzzy deep contractive autoencoder (CAE), and all prediction networks are cooperatively evolved, simultaneously using intrapopulation evolution to learn task-specific knowledge in each domain and using interpopulation evolution to learn common knowledge shared across the domains. Experimental results show that CETL achieves high prediction accuracies compared to selected state-of-the-art transfer learning and multitask learning models on datasets during two stages of COVID-19 spreading in China.

Learning Self-Corrective Network via Adaptive Self-Labeling and Dynamic NMS for High-Performance Long-Term Tracking.

This article presents a self-corrective network-based long-term tracker (SCLT) including a self-modulated tracking reliability evaluator (STRE) and a self-adjusting proposal postprocessor (SPPP). The targets in the long-term sequences often suffer from severe appearance variations. Existing long-term trackers often online update their models to adapt the variations, but the inaccurate tracking results introduce cumulative error into the updated model that may cause severe drift issue. To this end, a robust long-term tracker should have the self-corrective capability that can judge whether the tracking result is reliable or not, and then it is able to recapture the target when severe drift happens caused by serious challenges (e.g., full occlusion and out-of-view). To address the first issue, the STRE designs an effective tracking reliability classifier that is built on a modulation subnetwork. The classifier is trained using the samples with pseudo labels generated by an adaptive self-labeling strategy. The adaptive self-labeling can automatically label the hard negative samples that are often neglected in existing trackers according to the statistical characteristics of target state, and the network modulation mechanism can guide the backbone network to learn more discriminative features without extra training data. To address the second issue, after the STRE has been triggered, the SPPP follows it with a dynamic NMS to recapture the target in time and accurately. In addition, the STRE and the SPPP demonstrate good transportability ability, and their performance is improved when combined with multiple baselines. Compared to the commonly used greedy NMS, the proposed dynamic NMS leverages an adaptive strategy to effectively handle the different conditions of in view and out of view, thereby being able to select the most probable object box that is essential to accurately online update the basic tracker. Extensive evaluations on four large-scale and challenging benchmark datasets including VOT2021LT, OxUvALT, TLP, and LaSOT demonstrate superiority of the proposed SCLT to a variety of state-of-the-art long-term trackers in terms of all measures. Source codes and demos can be found at https://github.com/TJUT-CV/SCLT.