An Online 3D Modeling Method for Pose Measurement under Uncertain Dynamic Occlusion Based on Binocular Camera.

3D modeling plays a significant role in many industrial applications that require geometry information for pose measurements, such as grasping, spraying, etc. Due to random pose changes in the workpieces on the production line, demand for online 3D modeling has increased and many researchers have focused on it. However, online 3D modeling has not been entirely determined due to the occlusion of uncertain dynamic objects that disturb the modeling process. In this study, we propose an online 3D modeling method under uncertain dynamic occlusion based on a binocular camera. Firstly, focusing on uncertain dynamic objects, a novel dynamic object segmentation method based on motion consistency constraints is proposed, which achieves segmentation by random sampling and poses hypotheses clustering without any prior knowledge about objects. Then, in order to better register the incomplete point cloud of each frame, an optimization method based on local constraints of overlapping view regions and a global loop closure is introduced. It establishes constraints in covisibility regions between adjacent frames to optimize the registration of each frame, and it also establishes them between the global closed-loop frames to jointly optimize the entire 3D model. Finally, a confirmatory experimental workspace is designed and built to verify and evaluate our method. Our method achieves online 3D modeling under uncertain dynamic occlusion and acquires an entire 3D model. The pose measurement results further reflect the effectiveness.

A Liquid Metal Temperature Detection System Based on Multi-Node Sapphire Fiber Sensor.

In order to accurately detect the temperature of molten aluminum and overcome the adverse influence of high temperature and corrosiveness on the sensing results, a temperature detection system based on a multi-node sapphire fiber sensor was proposed and developed. Through the structural parameter design of the fiber sensor, the scheme of utilizing the 0.7 mm diameter fiber and 0.5 mm groove was formulated. Simulation and analysis were carried out to determine the ultrasonic response distribution of the signal passing through the whole fiber sensor. The results indicate that the system is capable of distinguishing test signals from various positions and temperatures. Following the completion of the static calibration, the temperature of the molten aluminum was observed in real-time, and the data of the temperature measurements conducted at the two groove locations were compared. According to the obtained results, the test accuracy was greater than 1 degree Celsius and the temperature test stability was good, laying a solid foundation for the potential development of temperature measurement devices.

Learning and Reusing Quadruped Robot Movement Skills from Biological Dogs for Higher-Level Tasks.

In the field of quadruped robots, the most classic motion control algorithm is based on model prediction control (MPC). However, this method poses challenges as it necessitates the precise construction of the robot's dynamics model, making it difficult to achieve agile movements similar to those of a biological dog. Due to these limitations, researchers are increasingly turning to model-free learning methods, which significantly reduce the difficulty of modeling and engineering debugging and simultaneously reduce real-time optimization computational burden. Inspired by the growth process of humans and animals, from learning to walk to fluent movements, this article proposes a hierarchical reinforcement learning framework for the motion controller to learn some higher-level tasks. First, some basic motion skills can be learned from motion data captured from a biological dog. Then, with these learned basic motion skills as a foundation, the quadruped robot can focus on learning higher-level tasks without starting from low-level kinematics, which saves redundant training time. By utilizing domain randomization techniques during the training process, the trained policy function can be directly transferred to a physical robot without modification, and the resulting controller can perform more biomimetic movements. By implementing the method proposed in this article, the agility and adaptability of the quadruped robot can be maximally utilized to achieve efficient operations in complex terrains.

Image Semantic Segmentation of Underwater Garbage with Modified U-Net Architecture Model.

Autonomous underwater garbage grasping and collection pose a great challenge to underwater robots. To assist underwater robots in locating and recognizing underwater garbage objects efficiently, a modified U-Net-based architecture consisting of a deeper contracting path and an expansive path is proposed to accomplish end-to-end image semantic segmentation. In addition, a dataset for underwater garbage semantic segmentation is established. The proposed architecture is further verified in the underwater garbage dataset and the effects of different hyperparameters, loss functions, and optimizers on the performance of refining the predicted segmented mask are examined. It is confirmed that the focal loss function will lead to a boost in solving the target-background unbalance problem. Eventually, the obtained results offer a solid foundation for fast and precise underwater target recognition and operations.

A Hierarchical Stabilization Control Method for a Three-Axis Gimbal Based on Sea-Sky-Line Detection.

Obtaining a stable video sequence for cameras on surface vehicles is always a challenging problem due to the severe disturbances in heavy sea environments. Aiming at this problem, this paper proposes a novel hierarchical stabilization method based on real-time sea-sky-line detection. More specifically, a hierarchical image stabilization control method that combines mechanical image stabilization with electronic image stabilization is adopted. With respect to the mechanical image stabilization method, a gimbal with three degrees of freedom (DOFs) and with a robust controller is utilized for the primary motion compensation. In addition, the electronic image stabilization method based on sea-sky-line detection in video sequences accomplishes motion estimation and compensation. The Canny algorithm and Hough transform are utilized to detect the sea-sky line. Noticeably, an image-clipping strategy based on prior information is implemented to ensure real-time performance, which can effectively improve the processing speed and reduce the equipment performance requirements. The experimental results indicate that the proposed method for mechanical and electronic stabilization can reduce the vibration by 74.2% and 42.1%, respectively.

Find Outliers of Image Edge Consistency by Weighted Local Linear Regression with Equality Constraints.

In this paper, we propose a general method to detect outliers from contaminated estimates of various image estimation applications. The method does not require any prior knowledge about the purpose, theory or hardware of the application but simply relies on the law of edge consistency between sources and estimates. The method is termed as ALRe (anchored linear residual) because it is based on the residual of weighted local linear regression with an equality constraint exerted on the measured pixel. Given a pair of source and contaminated estimate, ALRe offers per-pixel outlier likelihoods, which can be used to compose the data weights of post-refinement algorithms, improving the quality of refined estimate. ALRe has the features of asymmetry, no false positive and linear complexity. Its effectiveness is verified on four applications, four post-refinement algorithms and three datasets. It demonstrates that, with the help of ALRe, refined estimates are better in the aspects of both quality and edge consistency. The results are even comparable to model-based and hardware-based methods. Accuracy comparison on synthetic images shows that ALRe could detect outliers reliably. It is as effective as the mainstream weighted median filter at spike detection and is significantly better at bad region detection.

A Novel Visual Sensor Stabilization Platform for Robotic Sharks Based on Improved LADRC and Digital Image Algorithm.

Autonomous underwater missions require the construction of a stable visual sensing system. However, acquiring continuous steady image sequences is a very challenging task for bionic robotic fish due to their tight internal space and the inherent periodic disturbance caused by the tail beating. To solve this problem, this paper proposes a modified stabilization strategy that combines mechanical devices and digital image techniques to enhance the visual sensor stability and resist periodic disturbance. More specifically, an improved window function-based linear active disturbance rejection control (LADRC) was utilized for mechanical stabilization. Furthermore, a rapid algorithm with inertial measurement units (IMUs) was implemented for digital stabilization. The experiments regarding mechanical stabilization, digital stabilization, and target recognition on the experimental platform for simulating fishlike oscillations demonstrated the effectiveness of the proposed methods. The success of these experiments provides valuable insight into the construction of underwater visual sensing systems and also establishes a solid foundation for the visual applications for robotic fish in dynamic aquatic environments.

Design and control of an embedded vision guided robotic fish with multiple control surfaces.

This paper focuses on the development and control issues of a self-propelled robotic fish with multiple artificial control surfaces and an embedded vision system. By virtue of the hybrid propulsion capability in the body plus the caudal fin and the complementary maneuverability in accessory fins, a synthesized propulsion scheme including a caudal fin, a pair of pectoral fins, and a pelvic fin is proposed. To achieve flexible yet stable motions in aquatic environments, a central pattern generator- (CPG-) based control method is employed. Meanwhile, a monocular underwater vision serves as sensory feedback that modifies the control parameters. The integration of the CPG-based motion control and the visual processing in an embedded microcontroller allows the robotic fish to navigate online. Aquatic tests demonstrate the efficacy of the proposed mechatronic design and swimming control methods. Particularly, a pelvic fin actuated sideward swimming gait was first implemented. It is also found that the speeds and maneuverability of the robotic fish with coordinated control surfaces were largely superior to that of the swimming robot propelled by a single control surface.

Target-Following Control of a Biomimetic Autonomous System Based on Predictive Reinforcement Learning.

Biological fish often swim in a schooling manner, the mechanism of which comes from the fact that these schooling movements can improve the fishes' hydrodynamic efficiency. Inspired by this phenomenon, a target-following control framework for a biomimetic autonomous system is proposed in this paper. Firstly, a following motion model is established based on the mechanism of fish schooling swimming, in which the follower robotic fish keeps a certain distance and orientation from the leader robotic fish. Second, by incorporating a predictive concept into reinforcement learning, a predictive deep deterministic policy gradient-following controller is provided with the normalized state space, action space, reward, and prediction design. It can avoid overshoot to a certain extent. A nonlinear model predictive controller is designed and can be selected for the follower robotic fish, together with the predictive reinforcement learning. Finally, extensive simulations are conducted, including the fix point and dynamic target following for single robotic fish, as well as cooperative following with the leader robotic fish. The obtained results indicate the effectiveness of the proposed methods, providing a valuable sight for the cooperative control of underwater robots to explore the ocean.

CylinderTag: An Accurate and Flexible Marker for Cylinder-Shape Objects Pose Estimation Based on Projective Invariants.

High-precision pose estimation based on visual markers has been a thriving research topic in the field of computer vision. However, the suitability of traditional flat markers on curved objects is limited due to the diverse shapes of curved surfaces, which hinders the development of high-precision pose estimation for curved objects. Therefore, this paper proposes a novel visual marker called CylinderTag, which is designed for developable curved surfaces such as cylindrical surfaces. CylinderTag is a cyclic marker that can be firmly attached to objects with a cylindrical shape. Leveraging the manifold assumption, the cross-ratio in projective invariance is utilized for encoding in the direction of zero curvature on the surface. Additionally, to facilitate the usage of CylinderTag, we propose a heuristic search-based marker generator and a high-performance recognizer as well. Moreover, an all-encompassing evaluation of CylinderTag properties is conducted by means of extensive experimentation, covering detection rate, detection speed, dictionary size, localization jitter, and pose estimation accuracy. CylinderTag showcases superior detection performance from varying view angles in comparison to traditional visual markers, accompanied by higher localization accuracy. Furthermore, CylinderTag boasts real-time detection capability and an extensive marker dictionary, offering enhanced versatility and practicality in a wide range of applications. Experimental results demonstrate that the CylinderTag is a highly promising visual marker for use on cylindrical-like surfaces, thus offering important guidance for future research on high-precision visual localization of cylinder-shaped objects. The code is available at: https://github.com/wsakobe/CylinderTag.

Adaptive Gait Training of a Lower Limb Rehabilitation Robot Based on Human-Robot Interaction Force Measurement.

The existing fixed gait lower limb rehabilitation robots perform a predetermined walking trajectory for patients, ignoring their residual muscle strength. To enhance patient participation and safety in training, this paper aims to develop a lower limb rehabilitation robot with adaptive gait training capability relying on human-robot interaction force measurement. Firstly, a novel lower limb rehabilitation robot system with several active and passive driven joints is developed, and 2 face-to-face mounted cantilever beam force sensors are employed to measure the human-robot interaction forces. Secondly, a dynamic model of the rehabilitation training robot is constructed to estimate the driven forces of the human lower leg in a completely passive state. Thereafter, based on the theoretical moment from the dynamics and the actual joint interaction force collected by the sensors, an adaptive gait adjustment method is proposed to achieve the goal of adapting to the wearer's movement intention. Finally, interactive experiments are carried out to validate the effectiveness of the developed rehabilitation training robot system. The proposed rehabilitation training robot system with adaptive gaits offers great potential for future high-quality rehabilitation training, e.g., improving participation and safety.

Toward the Intelligent, Safe Exploration of a Biomimetic Underwater Robot: Modeling, Planning, and Control.

Safe, underwater exploration in the ocean is a challenging task due to the complex environment, which often contains areas with dense coral reefs, uneven terrain, or many obstacles. To address this issue, an intelligent underwater exploration framework of a biomimetic robot is proposed in this paper, including an obstacle avoidance model, motion planner, and yaw controller. Firstly, with the aid of the onboard distance sensors in robotic fish, the obstacle detection model is established. On this basis, two types of obstacles, i.e., rectangular and circular, are considered, followed by the obstacle collision model's construction. Secondly, a deep reinforcement learning method is adopted to plan the plane motion, and the performances of different training setups are investigated. Thirdly, a backstepping method is applied to derive the yaw control law, in which a sigmoid function-based transition method is employed to smooth the planning output. Finally, a series of simulations are carried out to verify the effectiveness of the proposed method. The obtained results indicate that the biomimetic robot can not only achieve intelligent motion planning but also accomplish yaw control with obstacle avoidance, offering a valuable solution for underwater operation in the ocean.

A Fish-like Binocular Vision System for Underwater Perception of Robotic Fish.

Biological fish exhibit a remarkably broad-spectrum visual perception capability. Inspired by the eye arrangement of biological fish, we design a fish-like binocular vision system, thereby endowing underwater bionic robots with an exceptionally broad visual perception capacity. Firstly, based on the design principles of binocular visual field overlap and tangency to streamlined shapes, a fish-like vision system is developed for underwater robots, enabling wide-field underwater perception without a waterproof cover. Secondly, addressing the significant distortion and parallax of the vision system, a visual field stitching algorithm is proposed to merge the binocular fields of view and obtain a complete perception image. Thirdly, an orientation alignment method is proposed that draws scales for yaw and pitch angles in the stitched images to provide a reference for the orientation of objects of interest within the field of view. Finally, underwater experiments evaluate the perception capabilities of the fish-like vision system, confirming the effectiveness of the visual field stitching algorithm and the orientation alignment method. The results show that the constructed vision system, when used underwater, achieves a horizontal field of view of 306.56°. The conducted work advances the visual perception capabilities of underwater robots and presents a novel approach to and insight for fish-inspired visual systems.

Design and Analysis of a Novel Bionic Tensegrity Robotic Fish with a Continuum Body.

Biological fish exhibit remarkable adaptability and exceptional swimming performance through their powerful and flexible bodies. Therefore, designing a continuum flexible body is significantly important for the development of a robotic fish. However, it is still challenging to replicate these functions of a biological body due to the limitations of actuation and material. In this paper, based on a tensegrity structure, we propose a bionic design scheme for a continuum robotic fish body with a property of stiffness variation. Its detailed structures and actuation principles are also presented. A mathematical model was established to analyze the bending characteristics of the tensegrity structure, which demonstrates the feasibility of mimicking the fish-like oscillation propulsion. Additionally, the stiffness variation mechanism is also exhibited experimentally to validate the effectiveness of the designed tensegrity fish body. Finally, a novel bionic robotic fish design scheme is proposed, integrating an electronic module-equipped fish head, a tensegrity body, and a flexible tail with a caudal fin. Subsequently, a prototype was developed. Extensive experiments were conducted to explore how control parameters and stiffness variation influence swimming velocity and turning performance. The obtained results reveal that the oscillation amplitude, frequency, and stiffness variation of the tensegrity robotic fish play crucial roles in swimming motions. With the stiffness variation, the developed tensegrity robotic fish achieves a maximum swimming velocity of 295 mm/s (0.84 body length per second, BL/s). Moreover, the bionic tensegrity robotic fish also performs a steering motion with a minimum turning radius of 230 mm (0.68 BL) and an angular velocity of 46.6°/s. The conducted studies will shed light on the novel design of a continuum robotic fish equipped with stiffness variation mechanisms.

Decoupled Metric Network for Single-Stage Few-Shot Object Detection.

Within the last few years, great efforts have been made to study few-shot learning. Although general object detection is advancing at a rapid pace, few-shot detection remains a very challenging problem. In this work, we propose a novel decoupled metric network (DMNet) for single-stage few-shot object detection. We design a decoupled representation transformation (DRT) and an image-level distance metric learning (IDML) to solve the few-shot detection problem. The DRT can eliminate the adverse effect of handcrafted prior knowledge by predicting objectness and anchor shape. Meanwhile, to alleviate the problem of representation disagreement between classification and location (i.e., translational invariance versus translational variance), the DRT adopts a decoupled manner to generate adaptive representations so that the model is easier to learn from only a few training data. As for a few-shot classification in the detection task, we design an IDML tailored to enhance the generalization ability. This module can perform metric learning for the whole visual feature, so it can be more efficient than traditional DML due to the merit of parallel inference for multiobjects. Based on the DRT and IDML, our DMNet efficiently realizes a novel paradigm for few-shot detection, called single-stage metric detection. Experiments are conducted on the PASCAL VOC dataset and the MS COCO dataset. As a result, our method achieves state-of-the-art performance in few-shot object detection. The codes are available at https://github.com/yrqs/DMNet.

A Torque Control Strategy for a Robotic Dolphin Platform Based on Angle of Attack Feedback.

Biological fish can always sense the state of water flow and regulate the angle of attack in time, so as to maintain the highest movement efficiency during periodic flapping. The biological adjustment of the caudal fin's angle of attack (AoA) depends on the contraction/relaxation of the tail muscles, accompanying the variation in tail stiffness. During an interaction with external fluid, it helps to maintain the optimal angle of attack during movement, to improve the propulsion performance. Inspired by this, this paper proposes a tail joint motion control scheme based on AoA feedback for the high-speed swimming of bionic dolphins. Firstly, the kinematic characteristics of the designed robot dolphin are analyzed, and the hardware basis is clarified. Second, aiming at the deficiency of the tail motor, which cannot effectively cooperate with the waist joint motor during high-frequency movement, a compensation model for the friction force and latex skin-restoring force is designed, and a joint angle control algorithm based on fuzzy inference is proposed to realize the tracking of the desired joint angle for the tail joint in torque mode. In addition, a tail joint closed-loop control scheme based on angle of attack feedback is proposed to improve the motion performance. Finally, experiments verify the effectiveness of the proposed motion control scheme.

Quantifying the Leaping Motion Using a Self-Propelled Bionic Robotic Dolphin Platform.

Kinematic analysis of leaping motions can provide meaningful insights into unraveling the efficient and agile propulsive mechanisms in dolphin swimming. However, undisturbed kinematic examination of live dolphins has been very scarce due to the restriction of close-up biological observation with a motion capture system. The main objective of this study is to quantify the leaping motion of a self-propelled bionic robotic dolphin using a combined numerical and experimental method. More specifically, a dynamic model was established for the hydrodynamic analysis of a changeable submerged portion, and experimental data were then employed to identify hydrodynamic parameters and validate the effectiveness. The effects of wave-making resistance were explored, indicating that there is a varying nonlinear relationship between power and speed at different depths. In addition, the wave-making resistance can be reduced significantly when swimming at a certain depth, which leads to a higher speed and less consumed power. Quantitative estimation of leaping motion is carried out, and the results suggest that with increase of the exiting velocity and angle, the maximum height of the center of mass (CM) increases as well; furthermore, a small exiting angle usually requires a much larger exiting velocity to achieve a complete exiting motion. These findings provide implications for optimizing motion performance, which is an integral part of underwater operations in complex aquatic environments.

Proactivity of fish and leadership of self-propelled robotic fish during interaction.

Fish interacting with biomimetic robotic fish is beneficial for animal behavior research, particularly in the study of collective behavior. Compared with passive-dragging robotic fish, self-propelled robotic fish floats in water, and its movement matches the flow field formed by the caudal fin oscillation, leading to more realistic interaction with animals. In this paper, we propose a self-propelled koi-mimicking robotic fish entity, develop a system for robotic fish and koi fish interaction, and conduct extensive experiments on quantity variation and parameter variation. The results showed that fish exhibited significantly lower proactivity when alone, and the most proactive case is one robotic fish interacting with two real fish. The experiments on parameter variation indicated that fish may respond more proactivity to robotic fish that swim with high frequency and low amplitude, but may also move together with high-frequency and high-amplitude swimming robotic fish. These findings could provide insights into fish collective behavior, guide the design of further fish-robot interaction experiments, and suggest directions for future improvements in goal-oriented robotic fish platforms.

A Circular Formation Method for Biomimetic Robotic Fish Inspired by Fish Milling.

Circular motion phenomena, akin to fish milling, are prevalent within the animal kingdom. This paper delineates two fundamental mechanisms underlying such occurrences: forward following and circular topological communication. Leveraging these pivotal concepts, we present a multi-agent formation circular model based on a second-order integrator. This model engenders the attainment of homogeneous intelligence convergence along the circumferential trajectory. The convergence characteristics are intricately linked to the number of agents and the model parameters. Consequently, we propose positive and negative solutions for ascertaining the convergent circle property and model parameters. Furthermore, by integrating our proposed formation control methodology with a robotic fish dynamics model, we have successfully implemented simulations and experiments, demonstrating the circular formation of multiple biomimetic robotic fish. This study provides a mathematical explication for the circular motion observed in animal groups and introduces a novel approach to achieving circular formation in multiple robots inspired by biological phenomena.

Barrier-Based Adaptive Line-of-Sight 3-D Path-Following System for a Multijoint Robotic Fish With Sideslip Compensation.

This article proposes a novel barrier-based adaptive line-of-sight (ALOS) three-dimensional (3-D) path-following system for an underactuated multijoint robotic fish. The framework of the developed path-following system is established based on a detailed dynamic model, including a barrier-based ALOS guidance strategy, three integrated inner-loop controllers, and a nonlinear disturbance observer (NDOB)-based sideslip angle compensation, which is employed to preserve a reliable tracking under a frequently varying sideslip angle of the robotic fish. First, a barrier-based convergence strategy is proposed to deal with probable along-track error disruption and suppress the error within a manageable range. Meanwhile, an improved adaptive guidance scheme is adopted with an appropriate look-ahead distance. Afterward, a novel NDOB-based sideslip angle compensation is put forward to identify the varying sideslip angle independent of speed estimation. Subsequently, inner-loop controllers are intended for regulation about the controlled references, including a super-twisting sliding-mode control (STSMC)-based speed controller, a global fast terminal sliding-mode control (GFTSMC)-based heading controller, and a GFTSMC-based depth controller. Finally, simulations and experiments with quantitative comparison in 3-D linear and helical path following are presented to verify the effectiveness and robustness of the proposed system. This path-following system provides a solid foundation for future marine autonomous cruising of the underwater multijoint robot.