Accurate Robot Arm Attitude Estimation Based on Multi-View Images and Super-Resolution Keypoint Detection Networks.

Robot arm monitoring is often required in intelligent industrial scenarios. A two-stage method for robot arm attitude estimation based on multi-view images is proposed. In the first stage, a super-resolution keypoint detection network (SRKDNet) is proposed. The SRKDNet incorporates a subpixel convolution module in the backbone neural network, which can output high-resolution heatmaps for keypoint detection without significantly increasing the computational resource consumption. Efficient virtual and real sampling and SRKDNet training methods are put forward. The SRKDNet is trained with generated virtual data and fine-tuned with real sample data. This method decreases the time and manpower consumed in collecting data in real scenarios and achieves a better generalization effect on real data. A coarse-to-fine dual-SRKDNet detection mechanism is proposed and verified. Full-view and close-up dual SRKDNets are executed to first detect the keypoints and then refine the results. The keypoint detection accuracy, PCK@0.15, for the real robot arm reaches up to 96.07%. In the second stage, an equation system, involving the camera imaging model, the robot arm kinematic model and keypoints with different confidence values, is established to solve the unknown rotation angles of the joints. The proposed confidence-based keypoint screening scheme makes full use of the information redundancy of multi-view images to ensure attitude estimation accuracy. Experiments on a real UR10 robot arm under three views demonstrate that the average estimation error of the joint angles is 0.53 degrees, which is superior to that achieved with the comparison methods.

Vacuum-powered soft actuator with oblique air chambers for easy detachment of artificial dry adhesive by coupled contraction and twisting.

A gecko foot-inspired, mushroom-shaped artificial dry adhesive exploiting intermolecular forces between microstructure and surface has drawn research attention for its strong adhesive force. However, the high pull-off strength corresponding to the adhesive force matters when detaching fragile substrates. In this study, we report a vacuum-powered soft actuator having oblique air chambers and a dry adhesive. The soft actuator performs coupled contraction and twisting by applying negative pneumatic pressure inward and exhibits not only high pull-off strength but also easy detachment. This effective detachment can be achieved thanks to the twisting motion of the soft actuator. The detachment performances of the actuator models are assessed using a 6-degrees-of-freedom robot arm. Results show that the soft actuators exhibit remarkable pull-off strength decrement from ~20 N cm-2 to ~2 N cm-2 due to the twisting. Finally, to verify a feasible application of this study, we utilize the inherent compliance of the actuators and introduce a glass transfer system for which a glass substrate on a slope is gripped by the flexibility of the soft actuators and delivered to the destination without any fracture.

The role of petal transpiration in floral humidity generation.

MAIN CONCLUSION: Using petrolatum gel as an antitranspirant on the flowers of California poppy and giant bindweed, we show that transpiration provides a large contribution to floral humidity generation. Floral humidity, an area of elevated humidity in the headspace of flowers, is believed to be produced predominantly through a combination of evaporation of liquid nectar and transpirational water loss from the flower. However, the role of transpiration in floral humidity generation has not been directly tested and is largely inferred by continued humidity production when nectar is removed from flowers. We test whether transpiration contributes to the floral humidity generation of two species previously identified to produce elevated floral humidity, Calystegia silvatica and Eschscholzia californica. Floral humidity production of flowers that underwent an antitranspirant treatment, petrolatum gel which blocks transpiration from treated tissues, is compared to flowers that did not receive such treatments. Gel treatments reduced floral humidity production to approximately a third of that produced by untreated flowers in C. silvatica, and half of that in E. californica. This confirms the previously untested inferences that transpiration has a large contribution to floral humidity generation and that this contribution may vary between species.

A Novel Collision-Free Homotopy Path Planning for Planar Robotic Arms.

Achieving the smart motion of any autonomous or semi-autonomous robot requires an efficient algorithm to determine a feasible collision-free path. In this paper, a novel collision-free path homotopy-based path-planning algorithm applied to planar robotic arms is presented. The algorithm utilizes homotopy continuation methods (HCMs) to solve the non-linear algebraic equations system (NAES) that models the robot's workspace. The method was validated with three case studies with robotic arms in different configurations. For the first case, a robot arm with three links must enter a narrow corridor with two obstacles. For the second case, a six-link robot arm with a gripper is required to take an object inside a narrow corridor with two obstacles. For the third case, a twenty-link arm must take an object inside a maze-like environment. These case studies validated, by simulation, the versatility and capacity of the proposed path-planning algorithm. The results show that the CPU time is dozens of milliseconds with a memory consumption less than 4.5 kB for the first two cases. For the third case, the CPU time is around 2.7 s and the memory consumption around 18 kB. Finally, the method's performance was further validated using the industrial robot arm CRS CataLyst-5 by Thermo Electron.

Computer Vision-Based Path Planning for Robot Arms in Three-Dimensional Workspaces Using Q-Learning and Neural Networks.

Computer vision-based path planning can play a crucial role in numerous technologically driven smart applications. Although various path planning methods have been proposed, limitations, such as unreliable three-dimensional (3D) localization of objects in a workspace, time-consuming computational processes, and limited two-dimensional workspaces, remain. Studies to address these problems have achieved some success, but many of these problems persist. Therefore, in this study, which is an extension of our previous paper, a novel path planning approach that combined computer vision, Q-learning, and neural networks was developed to overcome these limitations. The proposed computer vision-neural network algorithm was fed by two images from two views to obtain accurate spatial coordinates of objects in real time. Next, Q-learning was used to determine a sequence of simple actions: up, down, left, right, backward, and forward, from the start point to the target point in a 3D workspace. Finally, a trained neural network was used to determine a sequence of joint angles according to the identified actions. Simulation and experimental test results revealed that the proposed combination of 3D object detection, an agent-environment interaction in the Q-learning phase, and simple joint angle computation by trained neural networks considerably alleviated the limitations of previous studies.

Performance Evaluation of a Robot-Mounted Interferometer for an Industrial Environment.

High value manufacturing requires production-integrated, fast, multi-sensor and multi-scale inspection. To meet this need, the robotic deployment of sensors within the factory environment is becoming increasingly popular. For microscale measurement applications, robot-mountable versions of high-resolution instruments, that are traditionally deployed in a laboratory environment, are now becoming available. However, standard methodologies for the evaluation of these instruments, particularly when mounted to a robot, have yet to be fully defined, and therefore, there is limited independent evaluation data to describe the potential performance of these systems. In this paper, a detailed evaluation approach is presented for light-weight robot mountable scanning interferometric sensors. Traditional evaluation approaches are considered and extended to account for robotic sensor deployment within industrial environments. The applicability and value of proposed evaluation is demonstrated through the comprehensive characterization of a Heliotis H6 interferometric sensors. The results indicate the performance of the sensor, in comparison to a traditional laboratory-based system, and demonstrate the limits of the sensor capability. Based-on the evaluation an effective strategy for robotic deployment of the sensor is demonstrated.

Inline Inspection with an Industrial Robot (IIIR) for Mass-Customization Production Line.

Robots are essential for the rapid development of Industry 4.0. In order to truly achieve autonomous robot control in customizable production lines, robots need to be accurate enough and capable of recognizing the geometry and orientation of an arbitrarily shaped object. This paper presents a method of inline inspection with an industrial robot (IIIR) for mass-customization production lines. A 3D scanner was used to capture the geometry and orientation of the object to be inspected. As the object entered the working range of the robot, the end effector moved along with the object and the camera installed at the end effector performed the requested optical inspections. The detailed information about the developed methodology was introduced in this paper. The experiments showed there was a relative movement between the moving object and the following camera and the speed was around 0.34 mm per second (worst case was around 0.94 mm per second). For a camera of 60 frames per second, the relative moving speed between the object and the camera was around 6 micron (around 16 micron for the worst case), which was stable enough for most industrial production inspections.

A New Adaptive Synergetic Control Design for Single Link Robot Arm Actuated by Pneumatic Muscles.

This paper suggests a new control design based on the concept of Synergetic Control theory for controlling a one-link robot arm actuated by Pneumatic artificial muscles (PAMs) in opposing bicep/tricep positions. The synergetic control design is first established based on known system parameters. However, in real PAM-actuated systems, the uncertainties are inherited features in their parameters and hence an adaptive synergetic control algorithm is proposed and synthesized for a PAM-actuated robot arm subjected to perturbation in its parameters. The adaptive synergetic laws are developed to estimate the uncertainties and to guarantee the asymptotic stability of the adaptive synergetic controlled PAM-actuated system. The work has also presented an improvement in the performance of proposed synergetic controllers (classical and adaptive) by applying a modern optimization technique based on Particle Swarm Optimization (PSO) to tune their design parameters towards optimal dynamic performance. The effectiveness of the proposed classical and adaptive synergetic controllers has been verified via computer simulation and it has been shown that the adaptive controller could cope with uncertainties and keep the controlled system stable. The proposed optimal Adaptive Synergetic Controller (ASC) has been validated with a previous adaptive controller with the same robot structure and actuation, and it has been shown that the optimal ASC outperforms its opponent in terms of tracking speed and error.

Application of chaos in a recurrent neural network to control in ill-posed problems: a novel autonomous robot arm.

Inspired by a viewpoint that complex/chaotic dynamics would play an important role in biological systems including the brain, chaotic dynamics introduced in a recurrent neural network was applied to robot control in ill-posed situations. By computer experiments we show that a model robot arm without an advanced visual processing function can catch a target object and bring it to a set position under ill-posed situations (e.g., in the presence of unknown obstacles). The key idea in these works is adaptive switching of a system parameter (connectivity) between a chaos regime and attractor regime in a neural network model, which generates, depending on environmental circumstances, either chaotic motions or definite motions corresponding to embedded attractors. The adaptive switching results in useful functional motions of the robot arm. These successful experiments indicate that chaotic dynamics is potentially useful for practical engineering control applications. In addition, this novel autonomous arm system is implemented in a hardware robot arm that can avoid obstacles and reach for a target in a situation where the robot can get only rough target information, including uncertainty, by means of a few sensors, as indicated in the appendix, A1 and A2.

Collision Detection for Underwater ROV Manipulator Systems.

Work-class ROVs equipped with robotic manipulators are extensively used for subsea intervention operations. Manipulators are teleoperated by human pilots relying on visual feedback from the worksite. Operating in a remote environment, with limited pilot perception and poor visibility, manipulator collisions which may cause significant damage are likely to happen. This paper presents a real-time collision detection algorithm for marine robotic manipulation. The proposed collision detection mechanism is developed, integrated into a commercial ROV manipulator control system, and successfully evaluated in simulations and experimental setup using a real industry standard underwater manipulator. The presented collision sensing solution has a potential to be a useful pilot assisting tool that can reduce the task load, operational time, and costs of subsea inspection, repair, and maintenance operations.

Performance of a Multifunctional Robot for Natural Orifice Transluminal Endoscopic Surgery.

Natural orifice transluminal endoscopic surgery (NOTES) has gained attention as a revolutionary technique with its potential advantages in eliminating skin incisions, shortening recovery time, and decreasing postoperative complications; however, its practical application is still constrained by the complexity of navigation through the surgical field and paucity of available instruments. Current progress on NOTES focuses on designing flexible articulated robots or fully inserted bimanual robots to address the limitations. However, the lack of multitasking tools, trade-offs between size and power, and lack of sufficient surgical force are too often neglected. The authors designed a bimanual robot with a multifunctional manipulator, which can realize on-site instrument-change according to surgeon needs. An articulated drive mechanism with 2 independent curvature sections was designed to deliver the robot to the surgical site. A corresponding reconfiguration operation sequence was formulated to ease insertion and thereby decrease the design trade-off between size and power. This article presents 3 benchtop and animal tests to evaluate the robotic surgery approach and demonstrate the effectiveness of the robot.

A Robotic Platform for Corn Seedling Morphological Traits Characterization.

Crop breeding plays an important role in modern agriculture, improving plant performance, and increasing yield. Identifying the genes that are responsible for beneficial traits greatly facilitates plant breeding efforts for increasing crop production. However, associating genes and their functions with agronomic traits requires researchers to observe, measure, record, and analyze phenotypes of large numbers of plants, a repetitive and error-prone job if performed manually. An automated seedling phenotyping system aimed at replacing manual measurement, reducing sampling time, and increasing the allowable work time is thus highly valuable. Toward this goal, we developed an automated corn seedling phenotyping platform based on a time-of-flight of light (ToF) camera and an industrial robot arm. A ToF camera is mounted on the end effector of the robot arm. The arm positions the ToF camera at different viewpoints for acquiring 3D point cloud data. A camera-to-arm transformation matrix was calculated using a hand-eye calibration procedure and applied to transfer different viewpoints into an arm-based coordinate frame. Point cloud data filters were developed to remove the noise in the background and in the merged seedling point clouds. A 3D-to-2D projection and an x-axis pixel density distribution method were used to segment the stem and leaves. Finally, separated leaves were fitted with 3D curves for morphological traits characterization. This platform was tested on a sample of 60 corn plants at their early growth stages with between two to five leaves. The error ratios of the stem height and leave length measurements are 13.7% and 13.1%, respectively, demonstrating the feasibility of this robotic system for automated corn seedling phenotyping.

Detecting contamination events during robotic total joint arthroplasty.

BACKGROUND: Robot-assisted total joint arthroplasty (robotic-TJA) has become more widespread over the last 20 years due to higher patient satisfaction and reduced complications. However, robotic TJA may have longer operative times and increased operating room traffic, which are known risk factors for contamination events. Contamination of surgical instruments may be contact- or airborne-related with documented scalpel blade contamination rates up to 9%. The robot arm is a novel instrument that comes in and out of the surgical field, so our objective was to assess whether the robot arm is a source of contamination when used in robotic TJA compared to other surgical instruments. METHODS: This was a prospective, single-institution, single-surgeon pilot study involving 103 robotic TJAs. The robot arm was swabbed prior to incision and after closure. Pre- and postoperative control swabs were also collected from the suction tip and scalpel blade. Swabs were incubated for 24 hours on tryptic soy agar followed by inspection for growth of any contaminating bacteria. RESULTS: A contamination event was detected in 10 cases (10%). The scalpel blade was the most common site of contamination (8%) followed by the robot arm (2%) and suction tip (0%). DISCUSSION: Robotic TJA is contaminated with bacteria at a rate around 10%. Although the robot arm is an additional source of potential contamination, the robot arm accrues bacterial contamination infrequently compared to the scalpel blade. CONCLUSION: Contamination of the robot arm during robotic TJA is minimal when compared to contamination of the scalpel blade.

A 3D-printed phantom for stereotactic body radiation therapy simulation.

In modern radiation therapy for lung cancer, examining the uncertainty between tumor motion and beam delivery is vitally important. To lower the radiation dose delivery to the patient's normal tissue, narrowing the irradiation field margin to hit the tumor accurately is critical. Thus we proposed a phantom that simulates the thorax and lung tumor's motions by employing a 3D printing technique. The lung tumor is controlled by a linear miniature Delta robot arm, with a maximum displacement of 20 mm in each direction. When we simulated the thoracic breathing movements at 12 mm in A-P (Anterior-Posterior), the control errors were within 10%. The average tracking errors of the prosthetic tumor were within 1.1 mm. Therefore, the 3D-printed phantom with a robot arm can provide a reliable simulation for training and dosimetry measurement before lung radiotherapy, especially SBRT.

Robot-based 6D bioprinting for soft tissue biomedical applications.

Within this interdisciplinary study, we demonstrate the applicability of a 6D printer for soft tissue engineering models. For this purpose, a special plant was constructed, combining the technical requirements for 6D printing with the biological necessities, especially for soft tissue. Therefore, a commercial 6D robot arm was combined with a sterilizable housing (including a high-efficiency particulate air (HEPA) filter and ultraviolet radiation (UVC) lamps) and a custom-made printhead and printbed. Both components allow cooling and heating, which is desirable for working with viable cells. In addition, a spraying unit was installed that allows the distribution of fine droplets of a liquid. Advanced geometries on uneven or angled surfaces can be created with the use of all six axes. Based on often used bioinks in the field of soft tissue engineering (gellan gum, collagen, and gelatin methacryloyl) with very different material properties, we could demonstrate the flexibility of the printing system. Furthermore, cell-containing constructs using primary human adipose-derived stem cells (ASCs) could be produced in an automated manner. In addition to cell survival, the ability to differentiate along the adipogenic lineage could also be demonstrated as a representative of soft tissue engineering.

A Highly Compact Zip Chain Arm with Origami-Inspired Folding Chain Structures.

A deployable robotic arm can be a useful tool for mobile systems to widen accessible areas without removing mobility. For practical use, the deployable robotic arm needs to satisfy two requirements: a high extension-compression ratio and robust structural stiffness against the environment. To this end, this paper suggests, for the first time, an origami-inspired zipper chain to achieve a highly compact, one-degree-of-freedom zipper chain arm. The key component is the foldable chain, which innovatively increases the space-saving capability in the stowed state. The foldable chain is fully flattened in the stowed state, allowing for storage of many more chains in the same space. Moreover, a transmission system was designed to transform a 2D flat pattern into a 3D chain shape in order to control the length of the origami zipper. Additionally, an empirical parametric study was performed to choose design parameters to maximize the bending stiffness. For the feasibility test, a prototype was built and performance tests were executed in relation to extension length, speed, and structural robustness.

Object Grasp Control of a 3D Robot Arm by Combining EOG Gaze Estimation and Camera-Based Object Recognition.

The purpose of this paper is to quickly and stably achieve grasping objects with a 3D robot arm controlled by electrooculography (EOG) signals. A EOG signal is a biological signal generated when the eyeballs move, leading to gaze estimation. In conventional research, gaze estimation has been used to control a 3D robot arm for welfare purposes. However, it is known that the EOG signal loses some of the eye movement information when it travels through the skin, resulting in errors in EOG gaze estimation. Thus, EOG gaze estimation is difficult to point out the object accurately, and the object may not be appropriately grasped. Therefore, developing a methodology to compensate, for the lost information and increase spatial accuracy is important. This paper aims to realize highly accurate object grasping with a robot arm by combining EMG gaze estimation and the object recognition of camera image processing. The system consists of a robot arm, top and side cameras, a display showing the camera images, and an EOG measurement analyzer. The user manipulates the robot arm through the camera images, which can be switched, and the EOG gaze estimation can specify the object. In the beginning, the user gazes at the screen's center position and then moves their eyes to gaze at the object to be grasped. After that, the proposed system recognizes the object in the camera image via image processing and grasps it using the object centroid. The object selection is based on the object centroid closest to the estimated gaze position within a certain distance (threshold), thus enabling highly accurate object grasping. The observed size of the object on the screen can differ depending on the camera installation and the screen display state. Therefore, it is crucial to set the distance threshold from the object centroid for object selection. The first experiment is conducted to clarify the distance error of the EOG gaze estimation in the proposed system configuration. As a result, it is confirmed that the range of the distance error is 1.8-3.0 cm. The second experiment is conducted to evaluate the performance of the object grasping by setting two thresholds from the first experimental results: the medium distance error value of 2 cm and the maximum distance error value of 3 cm. As a result, it is found that the grasping speed of the 3 cm threshold is 27% faster than that of the 2 cm threshold due to more stable object selection.

Low-Cost Angle Sensor for Robotics Applications Using Plastic Optical Fiber Based on Optical Loss Mechanism.

Robotic systems and the human body consist of numerous joint structures, all of which require precise angle adjustments. At present, encoder, strain gauge, and electrical resistance-based sensors are commonly used for angle measurement. However, these sensors have limitations when used in underwater or in environments with strong electromagnetic waves. Therefore, we have developed an angle sensor based on step-index profile plastic optical fiber (SI-POF), which is cost-effective and highly durable, in this study in order to overcome the limitations of existing angle measurement sensors. To this end, the amount of light loss according to the gab and angle changes that occur when the POF angle sensor is applied to the robot arm was experimentally measured, and based on the results, a simulation of the amount of light loss when the two losses occurred at the same time was conducted. In addition, the performance of the POF angle sensor was evaluated by measuring sensitivity and resolution, and comparative verification with a commonly used encoder was conducted to verify the reliability of sensors in extreme environments, such as those with electromagnetic fields and those that are underwater. Through this, the reliability and practicality of the POF angle sensor were confirmed. The results obtained in this study suggest that POF-based angle sensors can contribute to the development of the biomimetic robot industry as well as ordinary robots, especially in environments where existing sensors are difficult to apply, such as areas with underwater or electromagnetic interference (EMI).

Primary Experiences with Robot-assisted Navigation-based Frameless Stereo-electroencephalography: Higher Accuracy than Neuronavigation-guided Manual Adjustment.

The use of robot-assisted frameless stereotactic electroencephalography (SEEG) is becoming more common. Among available robotic arms, Stealth Autoguide (SA) (Medtronic, Minneapolis, MN, USA) functions as an optional instrument of the neuronavigation system. The aims of this study were to present our primary experiences with SEEG using SA and to compare the accuracy of implantation between SA and navigation-guided manual adjustment (MA). Seventeen electrodes from two patients who underwent SEEG with SA and 18 electrodes from four patients with MA were retrospectively reviewed. We measured the distance between the planned location and the actual location at entry (De) and the target (Dt) in each electrode. The length of the trajectory did not show a strong correlation with Dt in SA (Pearson's correlation coefficient [r] = 0.099, p = 0.706) or MA (r = 0.233, p = 0.351). De and Dt in SA were shorter than those in MA (1.99 ± 0.90 vs 4.29 ± 1.92 mm, p = 0.0002; 3.59 ± 2.22 vs 5.12 ± 1.40 mm, p = 0.0065, respectively). SA offered higher accuracy than MA both at entry and target. Surgical times per electrode were 38.9 and 32 min in the two patients with SA and ranged from 51.6 to 88.5 min in the four patients with MA. During the implantation period of 10.3 ± 3.6 days, no patients experienced any complications.

Establishment of low-cost laboratory automation processes using AutoIt and 4-axis robots.

In most small laboratories, many processes are not yet automated because existing laboratory automation solutions are usually expensive and inflexible to use. Examples of this are autosamplers that are only compatible with one specific laboratory instrument or larger liquid handling stations that are expensive and usually self-contained. A flexible and inexpensive way to automate laboratory processes would be to automate existing laboratory equipment with the help of suitable robotic arms. In this study, we investigate the feasibility of such a strategy based on a low-cost 4-axis robot and freely available software. We used the scripting language AutoIt that automates any Windows-based instrument control software. Using these tools, we automated three fundamentally different laboratory processes: a pipetting process, a use as an autosampler for an atomic absorption spectroscopy instrument, and a more complex process involving the inoculation of bacterial cultures. We also integrated a conventional webcam for 2D barcode recognition. Compared to a trained professional who performed all experiments manually, all setups showed no significant differences in accuracy and precision. In summary, the tested system consisting of a 4-axis robot and freely available software is suitable for flexible automation and has potential for even more complex laboratory processes. Limitations such as a lack of collaboration and speed will be addressed in follow-up studies. The system thus represents a well-suited flexible laboratory automation system for both research and teaching purposes.