Multi-Task Foreground-Aware Network with Depth Completion for Enhanced RGB-D Fusion Object Detection Based on Transformer.

Fusing multiple sensor perceptions, specifically LiDAR and camera, is a prevalent method for target recognition in autonomous driving systems. Traditional object detection algorithms are limited by the sparse nature of LiDAR point clouds, resulting in poor fusion performance, especially for detecting small and distant targets. In this paper, a multi-task parallel neural network based on the Transformer is constructed to simultaneously perform depth completion and object detection. The loss functions are redesigned to reduce environmental noise in depth completion, and a new fusion module is designed to enhance the network's perception of the foreground and background. The network leverages the correlation between RGB pixels for depth completion, completing the LiDAR point cloud and addressing the mismatch between sparse LiDAR features and dense pixel features. Subsequently, we extract depth map features and effectively fuse them with RGB features, fully utilizing the depth feature differences between foreground and background to enhance object detection performance, especially for challenging targets. Compared to the baseline network, improvements of 4.78%, 8.93%, and 15.54% are achieved in the difficult indicators for cars, pedestrians, and cyclists, respectively. Experimental results also demonstrate that the network achieves a speed of 38 fps, validating the efficiency and feasibility of the proposed method.

MSFF-MA-DDI: Multi-Source Feature Fusion with Multiple Attention blocks for predicting Drug-Drug Interaction events.

The interaction of multiple drugs could lead to severe events, which cause medical injuries and expenses. Accurate prediction of drug-drug interaction (DDI) events can help clinicians make effective decisions and establish appropriate therapy programs. However, there exist two issues worthy of further consideration. (i) The global features of drug molecules should be paid attention to, rather than just their local characteristics. (ii) The fusion of multi-source features should also be studied to capture the comprehensive features of the drug. This study designs a Multi-Source Feature Fusion framework with Multiple Attention blocks named MSFF-MA-DDI that utilizes multimodal data for DDI event prediction. MSFF-MA-DDI can (i) encode global correlations between long-distance atoms in drug molecular sequences by a self-attention layer based on a position embedding block and (ii) fuse drug sequence features and heterogeneous features (chemical substructure, target, and enzyme) through a multi-head attention block to better represent the features of drugs. Experiments on real-world datasets show that MSFF-MA-DDI can achieve performance that is close to or even better than state-of-the-art models. Especially in cold start scenarios, the model can achieve the best performance. The effectiveness of the model is also supported by the case study on nervous system drugs. The source codes and data are available at https://github.com/BioCenter-SHU/MSFF-MA-DDI.

Self-Attention Mechanism-Based Head Pose Estimation Network with Fusion of Point Cloud and Image Features.

Head pose estimation serves various applications, such as gaze estimation, fatigue-driven detection, and virtual reality. Nonetheless, achieving precise and efficient predictions remains challenging owing to the reliance on singular data sources. Therefore, this study introduces a technique involving multimodal feature fusion to elevate head pose estimation accuracy. The proposed method amalgamates data derived from diverse sources, including RGB and depth images, to construct a comprehensive three-dimensional representation of the head, commonly referred to as a point cloud. The noteworthy innovations of this method encompass a residual multilayer perceptron structure within PointNet, designed to tackle gradient-related challenges, along with spatial self-attention mechanisms aimed at noise reduction. The enhanced PointNet and ResNet networks are utilized to extract features from both point clouds and images. These extracted features undergo fusion. Furthermore, the incorporation of a scoring module strengthens robustness, particularly in scenarios involving facial occlusion. This is achieved by preserving features from the highest-scoring point cloud. Additionally, a prediction module is employed, combining classification and regression methodologies to accurately estimate head poses. The proposed method improves the accuracy and robustness of head pose estimation, especially in cases involving facial obstructions. These advancements are substantiated by experiments conducted using the BIWI dataset, demonstrating the superiority of this method over existing techniques.

Prediction of Pedestrian Crossing Behavior Based on Surveillance Video.

Prediction of pedestrian crossing behavior is an important issue faced by the realization of autonomous driving. The current research on pedestrian crossing behavior prediction is mainly based on vehicle camera. However, the sight line of vehicle camera may be blocked by other vehicles or the road environment, making it difficult to obtain key information in the scene. Pedestrian crossing behavior prediction based on surveillance video can be used in key road sections or accident-prone areas to provide supplementary information for vehicle decision-making, thereby reducing the risk of accidents. To this end, we propose a pedestrian crossing behavior prediction network for surveillance video. The network integrates pedestrian posture, local context and global context features through a new cross-stacked gated recurrence unit (GRU) structure to achieve accurate prediction of pedestrian crossing behavior. Applied onto the surveillance video dataset from the University of California, Berkeley to predict the pedestrian crossing behavior, our model achieves the best results regarding accuracy, F1 parameter, etc. In addition, we conducted experiments to study the effects of time to prediction and pedestrian speed on the prediction accuracy. This paper proves the feasibility of pedestrian crossing behavior prediction based on surveillance video. It provides a reference for the application of edge computing in the safety guarantee of automatic driving.

MDF-SA-DDI: predicting drug-drug interaction events based on multi-source drug fusion, multi-source feature fusion and transformer self-attention mechanism.

One of the main problems with the joint use of multiple drugs is that it may cause adverse drug interactions and side effects that damage the body. Therefore, it is important to predict potential drug interactions. However, most of the available prediction methods can only predict whether two drugs interact or not, whereas few methods can predict interaction events between two drugs. Accurately predicting interaction events of two drugs is more useful for researchers to study the mechanism of the interaction of two drugs. In the present study, we propose a novel method, MDF-SA-DDI, which predicts drug-drug interaction (DDI) events based on multi-source drug fusion, multi-source feature fusion and transformer self-attention mechanism. MDF-SA-DDI is mainly composed of two parts: multi-source drug fusion and multi-source feature fusion. First, we combine two drugs in four different ways and input the combined drug feature representation into four different drug fusion networks (Siamese network, convolutional neural network and two auto-encoders) to obtain the latent feature vectors of the drug pairs, in which the two auto-encoders have the same structure, and their main difference is the number of neurons in the input layer of the two auto-encoders. Then, we use transformer blocks that include self-attention mechanism to perform latent feature fusion. We conducted experiments on three different tasks with two datasets. On the small dataset, the area under the precision-recall-curve (AUPR) and F1 scores of our method on task 1 reached 0.9737 and 0.8878, respectively, which were better than the state-of-the-art method. On the large dataset, the AUPR and F1 scores of our method on task 1 reached 0.9773 and 0.9117, respectively. In task 2 and task 3 of two datasets, our method also achieved the same or better performance as the state-of-the-art method. More importantly, the case studies on five DDI events are conducted and achieved satisfactory performance. The source codes and data are available at https://github.com/ShenggengLin/MDF-SA-DDI.

Semantic Segmentation Leveraging Simultaneous Depth Estimation.

Semantic segmentation is one of the most widely studied problems in computer vision communities, which makes a great contribution to a variety of applications. A lot of learning-based approaches, such as Convolutional Neural Network (CNN), have made a vast contribution to this problem. While rich context information of the input images can be learned from multi-scale receptive fields by convolutions with deep layers, traditional CNNs have great difficulty in learning the geometrical relationship and distribution of objects in the RGB image due to the lack of depth information, which may lead to an inferior segmentation quality. To solve this problem, we propose a method that improves segmentation quality with depth estimation on RGB images. Specifically, we estimate depth information on RGB images via a depth estimation network, and then feed the depth map into the CNN which is able to guide the semantic segmentation. Furthermore, in order to parse the depth map and RGB images simultaneously, we construct a multi-branch encoder-decoder network and fuse the RGB and depth features step by step. Extensive experimental evaluation on four baseline networks demonstrates that our proposed method can enhance the segmentation quality considerably and obtain better performance compared to other segmentation networks.

Heterogeneous Iris One-to-One Certification with Universal Sensors based On Quality Fuzzy Inference and Multi-Feature Fusion Lightweight Neural Network.

Due to the unsteady morphology of heterogeneous irises generated by a variety of different devices and environments, the traditional processing methods of statistical learning or cognitive learning for a single iris source are not effective. Traditional iris recognition divides the whole process into several statistically guided steps, which cannot solve the problem of correlation between various links. The existing iris data set size and situational classification constraints make it difficult to meet the requirements of learning methods under a single deep learning framework. Therefore, aiming at a one-to-one iris certification scenario, this paper proposes a heterogeneous iris one-to-one certification method with universal sensors based on quality fuzzy inference and a multi-feature entropy fusion lightweight neural network. The method is divided into an evaluation module and a certification module. The evaluation module can be used by different devices to design a quality fuzzy concept inference system and an iris quality knowledge concept construction mechanism, transform human logical cognition concepts into digital concepts, and select appropriate concepts to determine iris quality according to different iris quality requirements and get a recognizable iris. The certification module is a lightweight neural network based on statistical learning ideas and a multi-source feature fusion mechanism. The information entropy of the iris feature label was used to set the iris entropy feature category label and design certification module functions according to the category label to obtain the certification module result. As the requirements for the number and quality of irises changes, the category labels in the certification module function were dynamically adjusted using a feedback learning mechanism. This paper uses iris data collected from three different sensors in the JLU(Jilin University) iris library. The experimental results prove that for the lightweight multi-state irises, the abovementioned problems are ameliorated to a certain extent by this method.