Haze optical-model-based nighttime image dehazing by modifying attenuation and atmospheric light.

Scene imaging is often affected by artificial light sources within a hazy environment at night, causing degraded images with low brightness, color distortion, and glow. These problems render the traditional atmospheric scattering optical model obsolete and incompatible. To address this issue, we established an optical imaging model suitable for nighttime dehazing, and an illumination component is incorporated into the attenuation term. We also introduced the near-light source coefficient to redefine the glow. Based on this model, we propose a new nighttime dehazing method. First, the rough atmospheric light is estimated using its low-frequency characteristics. Then, the glow is calculated by the near-light source coefficient. Finally, we remove the haze and illumination to get a clear image. Extensive experiments prove that our method exhibits a better color recovery effect, which effectively improves the visibility and detail. Furthermore, we believe our method outperforms other methods, both qualitatively and quantitatively.

Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning.

Dexterous manipulation of the robot is an important part of realizing intelligence, but manipulators can only perform simple tasks such as sorting and packing in a structured environment. In view of the existing problem, this paper presents a state-of-the-art survey on an intelligent robot with the capability of autonomous deciding and learning. The paper first reviews the main achievements and research of the robot, which were mainly based on the breakthrough of automatic control and hardware in mechanics. With the evolution of artificial intelligence, many pieces of research have made further progresses in adaptive and robust control. The survey reveals that the latest research in deep learning and reinforcement learning has paved the way for highly complex tasks to be performed by robots. Furthermore, deep reinforcement learning, imitation learning, and transfer learning in robot control are discussed in detail. Finally, major achievements based on these methods are summarized and analyzed thoroughly, and future research challenges are proposed.

An Interactive Image Segmentation Method in Hand Gesture Recognition.

In order to improve the recognition rate of hand gestures a new interactive image segmentation method for hand gesture recognition is presented, and popular methods, e.g., Graph cut, Random walker, Interactive image segmentation using geodesic star convexity, are studied in this article. The Gaussian Mixture Model was employed for image modelling and the iteration of Expectation Maximum algorithm learns the parameters of Gaussian Mixture Model. We apply a Gibbs random field to the image segmentation and minimize the Gibbs Energy using Min-cut theorem to find the optimal segmentation. The segmentation result of our method is tested on an image dataset and compared with other methods by estimating the region accuracy and boundary accuracy. Finally five kinds of hand gestures in different backgrounds are tested on our experimental platform, and the sparse representation algorithm is used, proving that the segmentation of hand gesture images helps to improve the recognition accuracy.

Simultaneous Calibration: A Joint Optimization Approach for Multiple Kinect and External Cameras.

Camera calibration is a crucial problem in many applications, such as 3D reconstruction, structure from motion, object tracking and face alignment. Numerous methods have been proposed to solve the above problem with good performance in the last few decades. However, few methods are targeted at joint calibration of multi-sensors (more than four devices), which normally is a practical issue in the real-time systems. In this paper, we propose a novel method and a corresponding workflow framework to simultaneously calibrate relative poses of a Kinect and three external cameras. By optimizing the final cost function and adding corresponding weights to the external cameras in different locations, an effective joint calibration of multiple devices is constructed. Furthermore, the method is tested in a practical platform, and experiment results show that the proposed joint calibration method can achieve a satisfactory performance in a project real-time system and its accuracy is higher than the manufacturer's calibration.

Distributed Resilient Tracking of Multiagent Systems Under Actuator and Sensor Faults.

The distributed resilient tracking problem for multiagent systems (MASs) is investigated in the presence of actuator/sensor faults over directed topology. Both actuator fault and sensor fault are taken into account. Meanwhile, using the local information, the fault compensators are introduced. Then, based on the fuzzy-logic systems (FLSs) and modification technique of adaptive law, a novel distributed adaptive resilient control protocol is developed, which can compensate the effect of faults on the actuator and sensor. It turns out that all signals of MASs are bounded, while the tracking errors enter an adjustable bounded region around the origin. Toward the end, two simulations are provided to validate the effectiveness of the theoretical results.

Dual-Hand Motion Capture by Using Biological Inspiration for Bionic Bimanual Robot Teleoperation.

Bionic bimanual robot teleoperation can transfer the grasping and manipulation skills of human dual hands to the bionic bimanual robots to realize natural and flexible manipulation. The motion capture of dual hands plays an important role in the teleoperation. The motion information of dual hands can be captured through the hand detection, localization, and pose estimation and mapped to the bionic bimanual robots to realize the teleoperation. However, although the motion capture technology has achieved great achievements in recent years, visual dual-hand motion capture is still a great challenge. So, this work proposed a dual-hand detection method and a 3-dimensional (3D) hand pose estimation method based on body and hand biological inspiration to achieve convenient and accurate monocular dual-hand motion capture and bionic bimanual robot teleoperation. First, a dual-hand detection method based on body structure constraints is proposed, which uses a parallel structure to combine hand and body relationship features. Second, a 3D hand pose estimation method with bone-constraint loss from single RGB images is proposed. Then, a bionic bimanual robot teleoperation method is designed by using the proposed hand detection and pose estimation methods. Experiment results on public hand datasets show that the performances of the proposed hand detection and 3D hand pose estimation outperform state-of-the-art methods. Experiment results on a bionic bimanual robot teleoperation platform shows the effectiveness of the proposed teleoperation method.

Propagation Structure Fusion for Rumor Detection Based on Node-Level Contrastive Learning.

With the rise of social media, the rapid spread of rumors online has resulted in numerous negative effects on society and the economy. The methods for rumor detection have attracted great interest from both academia and industry. Given the widespread effectiveness of contrastive learning, many graph contrastive learning models for rumor detection have been proposed by using the event propagation structure as graph data. However, the existing contrastive models usually treat the propagation structure of other events similar to the anchor events as negative samples. While this design choice allows for discriminative learning, on the other hand, it also inevitably pushes apart semantically similar samples and, thus, degrades model performance. In this article, we propose a novel propagation fusion model called propagation structure fusion model based on node-level contrastive learning (PFNC) for rumor detection based on node-level contrastive learning. PFNC first obtains three augmented propagation structures by masking the text of each node in the propagation structure randomly and perturbing some edges in the propagation structure based on the importance of edges. Then, PFNC applies the node-level contrastive learning method between every two augmented propagation structures to prevent the samples with similar propagation structure from far away. Finally, a convolutional neural network (CNN)-based model is proposed to capture the relevant information that is consistent and supplementary among three augmented propagation structures by regarding the propagation structure of the event as a color picture, three augmented propagation structures as color channels, and each node as a pixel. The experimental results on real datasets show that the PFNC significantly outperforms the state-of-the-art models for rumor detection.

SDE-YOLO: A Novel Method for Blood Cell Detection.

This paper proposes an improved target detection algorithm, SDE-YOLO, based on the YOLOv5s framework, to address the low detection accuracy, misdetection, and leakage in blood cell detection caused by existing single-stage and two-stage detection algorithms. Initially, the Swin Transformer is integrated into the back-end of the backbone to extract the features in a better way. Then, the 32 × 32 network layer in the path-aggregation network (PANet) is removed to decrease the number of parameters in the network while increasing its accuracy in detecting small targets. Moreover, PANet substitutes traditional convolution with depth-separable convolution to accurately recognize small targets while maintaining a fast speed. Finally, replacing the complete intersection over union (CIOU) loss function with the Euclidean intersection over union (EIOU) loss function can help address the imbalance of positive and negative samples and speed up the convergence rate. The SDE-YOLO algorithm achieves a mAP of 99.5%, 95.3%, and 93.3% on the BCCD blood cell dataset for white blood cells, red blood cells, and platelets, respectively, which is an improvement over other single-stage and two-stage algorithms such as SSD, YOLOv4, and YOLOv5s. The experiment yields excellent results, and the algorithm detects blood cells very well. The SDE-YOLO algorithm also has advantages in accuracy and real-time blood cell detection performance compared to the YOLOv7 and YOLOv8 technologies.

Versatile Graph Neural Networks Toward Intuitive Human Activity Understanding.

Benefiting from the advanced human visual system, humans naturally classify activities and predict motions in a short time. However, most existing computer vision studies consider those two tasks separately, resulting in an insufficient understanding of human actions. Moreover, the effects of view variations remain challenging for most existing skeleton-based methods, and the existing graph operators cannot fully explore multiscale relationship. In this article, a versatile graph-based model (Vers-GNN) is proposed to deal with those two tasks simultaneously. First, a skeleton representation self-regulated scheme is proposed. It is among the first trials that successfully integrate the idea of view adaptation into a graph-based human activity analysis system. Next, several novel graph operators are proposed to model the positional relationships and learn the abstract dynamics between different human joints and parts. Finally, a practical multitask learning framework and a multiobjective self-supervised learning scheme are proposed to promote both the tasks. The comparative experimental results show that Vers-GNN outperforms the recent state-of-the-art methods for both the tasks, with the to date highest recognition accuracies on the datasets of NTU RGB + D (CV: 97.2%), UWA3D (88.7%), and CMU (1000 ms: 1.13).

Active Disturbance Rejection Control of Euler-Lagrange Systems Exploiting Internal Damping.

Active disturbance rejection control (ADRC) is an efficient control technique to accommodate both internal uncertainties and external disturbances. In the typical ADRC framework, however, the design philosophy is to "force" the system dynamics into a double-integral form by an extended state observer (ESO) and then the controller is designed. Especially, the systems' physical structure has been neglected in such a design paradigm. In this article, a new ADRC framework is proposed by incorporating at a fundamental level the physical structure of the Euler-Lagrange (EL) systems. In particular, the differential feedback gain can be selected considerably small or even 0, due to the effective exploitation of the system's internal damping. The design principle stems from an analysis of the energy balance of EL systems, yielding a physically interpretable design. Moreover, the exploitation of the system's internal damping is thoroughly discussed, which is of practical significance for applications of the proposed design. Besides, a sliding-mode ESO is designed to improve the estimation performance over traditional linear ESO. Finally, the proposed control framework is illustrated through tracking control of an omnidirectional mobile robot. Extensive experimental tests are conducted to verify the proposed design as well as the discussions.

A Physics-Guided Coordinated Distributed MPC Method for Shape Control of an Antenna Reflector.

Active shape control for an antenna reflector is a significant procedure used to compensate for the impacts of a complicated space environment. In this article, a physics-guided distributed model predictive control (DMPC) framework for reflector shape control with input saturation is proposed. First, guided by the actual physical characteristics, an overall structural system is decomposed into multilevel subsystems with the help of a so-called substructuring technique. For each subsystem, a prediction model with information interaction is discretized by an explicit Newmark- ß method. Then, to improve the system-wide control performance, a coordinator among all the subsystems is designed in an iterative fashion. The input saturation constraints are addressed by transforming the original problem into a linear complementarity problem (LCP). Finally, by solving the LCP, the input trajectory can be obtained. The performance of the proposed DMPC algorithm is validated through an experiment on the shape control of an antenna reflector structure.

Deep Temporal Model-Based Identity-Aware Hand Detection for Space Human-Robot Interaction.

Hand detection is a crucial technology for space human-robot interaction (SHRI), and the awareness of hand identities is particularly critical. However, most advanced works have three limitations: 1) the low detection accuracy of small-size objects; 2) insufficient temporal feature modeling between frames in videos; and 3) the inability of real-time detection. In the article, a temporal detector (called TA-RSSD) is proposed based on the SSD and spatiotemporal long short-term memory (ST-LSTM) for real-time detection in SHRI applications. Next, based on the online tubelet analysis, a real-time identity-awareness module is designed for multiple hand object identification. Several notable properties are described as follows: 1) the hybrid structure of the Resnet-101 and the SSD improves the detection accuracy of small objects; 2) three-level feature pyramidal structure retains rich semantic information without losing detailed information; 3) a group of the redesigned temporal attentional LSTM (TA-LSTM) is utilized for three-level feature map modeling, which effectively achieves background suppression and scale suppression; 4) low-level attention maps are used to eliminate in-class similarity between hand objects, which improves the accuracy of identity awareness; and 5) a novel association training scheme enhances the temporal coherence between frames. The proposed model is evaluated on the SHRI-VID dataset (collected according to the task requirements), the AU-AIR dataset, and the ImageNet-VID benchmark. Extensive ablation studies and comparisons on detection and identity-awareness capacities show the superiority of the proposed model. Finally, a set of actual testing is conducted on a space robot, and the results show that the proposed model achieves a real-time speed and high accuracy.

Physical Human-Robot Collaboration: Robotic Systems, Learning Methods, Collaborative Strategies, Sensors, and Actuators.

This article presents a state-of-the-art survey on the robotic systems, sensors, actuators, and collaborative strategies for physical human-robot collaboration (pHRC). This article starts with an overview of some robotic systems with cutting-edge technologies (sensors and actuators) suitable for pHRC operations and the intelligent assist devices employed in pHRC. Sensors being among the essential components to establish communication between a human and a robotic system are surveyed. The sensor supplies the signal needed to drive the robotic actuators. The survey reveals that the design of new generation collaborative robots and other intelligent robotic systems has paved the way for sophisticated learning techniques and control algorithms to be deployed in pHRC. Furthermore, it revealed the relevant components needed to be considered for effective pHRC to be accomplished. Finally, a discussion of the major advances is made, some research directions, and future challenges are presented.

Attribute-Driven Granular Model for EMG-Based Pinch and Fingertip Force Grand Recognition.

Fine multifunctional prosthetic hand manipulation requires precise control on the pinch-type and the corresponding force, and it is a challenge to decode both aspects from myoelectric signals. This paper proposes an attribute-driven granular model (AGrM) under a machine-learning scheme to solve this problem. The model utilizes the additionally captured attribute as the latent variable for a supervised granulation procedure. It was fulfilled for EMG-based pinch-type classification and the fingertip force grand prediction. In the experiments, 16 channels of surface electromyographic signals (i.e., main attribute) and continuous fingertip force (i.e., subattribute) were simultaneously collected while subjects performing eight types of hand pinches. The use of AGrM improved the pinch-type recognition accuracy to around 97.2% by 1.8% when constructing eight granules for each grasping type and received more than 90% force grand prediction accuracy at any granular level greater than six. Further, sensitivity analysis verified its robustness with respect to different channel combination and interferences. In comparison with other clustering-based granulation methods, AGrM achieved comparable pinch recognition accuracy but was of lowest computational cost and highest force grand prediction accuracy.

Fuzzy rule-based model for outlier detection in a Topical Negative Pressure Wound Therapy Device.

This paper proposes a novel method for offline outlier detection in nonlinear dynamical systems using an input-output dataset of a Topical Negative Pressure Wound Therapy Device, NPWT. The fundamental characteristics of an NPWT describe a chaotic system whose states vary over time and may result in unpredictable and possibly anomalous divergent behavior in the presence of perturbations and other unmodeled system dynamics, despite a quasi-stable controller. Bacterial Memetic Algorithm, BMA, is used to generate fuzzy rule-based models of the input-output dataset. The error definition in the fuzzy rule extraction features a novel application of the Canberra Distance. The optimal number of rules for identifying the outliers, validated against both artificial and real system datasets, is calculated from the sample of inferred fuzzy models. The optimal number of rules is two in both cases based on the maximum average-error-drop. Using three or more rules results in better error performance; however, the algorithm learns the nuances of the outlier patterns instead. Novel methods for creating the outlier list and determining the optimal number of rules for the outlier detection problem are proposed.

Robot Motor Skill Transfer With Alternate Learning in Two Spaces.

Recent research achievements in learning from demonstration (LfD) illustrate that the reinforcement learning is effective for the robots to improve their movement skills. The current challenge mainly remains in how to generate new robot motions automatically to perform new tasks, which have a similar preassigned performance indicator but are different from the demonstration tasks. To deal with the abovementioned issue, this article proposes a framework to represent the policy and conduct imitation learning and optimization for robot intelligent trajectory planning, based on the improved locally weighted regression (iLWR) and policy improvement with path integral by dual perturbation (PI2-DP). Besides, the reward-guided weight searching and basis function's adaptive evolving are performed alternately in two spaces, i.e., the basis function space and the weight space, to deal with the abovementioned problem. The alternate learning process constructs a sequence of two-tuples that join the demonstration task and new one together for motor skill transfer, so that the robot gradually acquires motor skill, from the task similar to demonstration to dissimilar tasks with different performance metrics. Classical via-points trajectory planning experiments are performed with the SCARA manipulator, a 10-degree of freedom (DOF) planar, and the UR robot. These results show that the proposed method is not only feasible but also effective.

Effect of Muscle Fatigue on Surface Electromyography-Based Hand Grasp Force Estimation.

Surface electromyography- (sEMG-) based hand grasp force estimation plays an important role with a promising accuracy in a laboratory environment, yet hardly clinically applicable because of physiological changes and other factors. One of the critical factors is the muscle fatigue concomitant with daily activities which degrades the accuracy and reliability of force estimation from sEMG signals. Conventional qualitative measurements of muscle fatigue contribute to an improved force estimation model with limited progress. This paper proposes an easy-to-implement method to evaluate the muscle fatigue quantitatively and demonstrates that the proposed metrics can have a substantial impact on improving the performance of hand grasp force estimation. Specifically, the reduction in the maximal capacity to generate force is used as the metric of muscle fatigue in combination with a back-propagation neural network (BPNN) is adopted to build a sEMG-hand grasp force estimation model. Experiments are conducted in the three cases: (1) pooling training data from all muscle fatigue states with time-domain feature only, (2) employing frequency domain feature for expression of muscle fatigue information based on case 1, and 3) incorporating the quantitative metric of muscle fatigue value as an additional input for estimation model based on case 1. The results show that the degree of muscle fatigue and task intensity can be easily distinguished, and the additional input of muscle fatigue in BPNN greatly improves the performance of hand grasp force estimation, which is reflected by the 6.3797% increase in R2 (coefficient of determination) value.

Design of Muscle Reflex Control for Upright Standing Push-Recovery Based on a Series Elastic Robot Ankle Joint.

In physical human-robot interaction environment, ankle joint muscle reflex control remains significant and promising in human bipedal stance. The reflex control mechanism contains rich information of human joint dynamic behavior, which is valuable in the application of real-time decoding motion intention. Thus, investigating the human muscle reflex mechanism is not only meaningful in human physiology study but also useful for the robotic system design in the field of human-robot physical interaction. In this paper, a specialized ankle joint muscle reflex control algorithm for human upright standing push-recovery is proposed. The proposed control algorithm is composed of a proportional-derivative (PD)-like controller and a positive force controller, which are employed to mimic the human muscle stretch reflex and muscle tendon force reflex, respectively. Reflex gains are regulated by muscle activation levels of contralateral ankle muscles. The proposed method was implemented on a self-designed series elastic robot ankle joint (SERAJ), where the series elastic actuator (SEA) has the potential to mimic human muscle-tendon unit (MTU). During the push-recovery experimental study, the surface electromyography (sEMG), ankle torque, body sway angle, and velocity of each subject were recorded in the case where the SERAJ was unilaterally kneed on each subject. The experimental results indicate that the proposed muscle reflex control method can easily realize upright standing push-recovery behavior, which is analogous to the original human behavior.

An enhanced teaching interface for a robot using DMP and GMR.

This paper develops an enhanced teaching interface tested on both a Baxter robot and a KUKA iiwa robot. Movements are collected from a human demonstrator by using a Kinect v2 sensor, and then the data is sent to a remote PC for the teleoperation with Baxter. Meanwhile, data is saved locally for the playback process of the Baxter. The dynamic movement primitive (DMP) is used to model and generalize the movements. In order to learn from multiple demonstrations accurately, dynamic time warping (DTW), is used to pretreat the data recorded by the robot platform and Gaussian mixture model (GMM), aiming to generate multiple patterns after the teaching process, are employed for the calculation of the DMP. Then the Gaussian mixture regression (GMR) algorithm is applied to generate a synthesized trajectory with smaller position errors in 3D space. This proposed approach is tested by performing two tasks on a KUKA iiwa and a Baxter robot.