3D object reconstruction: A comprehensive view-dependent dataset.

The dataset contains RGB, depth, segmentation images of the scenes and information about the camera poses that can be used to create a full 3D model of the scene and develop methods that reconstruct objects from a single RGB-D camera view. Data were collected in the custom simulator that loads random graspable objects and random tables from the ShapeNet dataset. The graspable object is placed above the table in a random position. Then, the scene is simulated using the PhysX engine to make sure that the scene is physically plausible. The simulator captures images of the scene from a random pose and then takes the second image from the camera pose that is on the opposite side of the scene. The second subset was created using Kinect Azure and a set of real objects located on the ArUco board that was used to estimate the camera pose.

Efficient leather spreading operations by dual-arm robotic systems.

To achieve precise grasping and spreading of irregular sheet-like soft objects (such as leather) by robots, this study addresses several challenges, including the irregularity of leather edges and the ambiguity of feature recognition points. To tackle these issues, this paper proposes an innovative method that involves alternately grasping the lowest point twice and using planar techniques to effectively spread the leather. We improved the YOLOV8 algorithm by incorporating the BIFPN network structure and the WIOU loss function, and trained a dedicated dataset for the lowest grasping points and planar grasping points, thereby achieving high-precision recognition. Additionally, we determined the optimal posture for grasping the lowest point and constructed an experimental platform, successfully conducting multiple rounds of leather grasping and spreading experiments with a success rate of 72%. Through an in-depth analysis of the failed experiments, this study reveals the limitations of the current methods and provides valuable guidance for future research.

Dexterous manipulation: differential sensitivity of manipulation and grasp forces to task requirements.

How humans coordinate digit forces to perform dexterous manipulation is not well understood. This gap is due to the use of tasks devoid of dexterity requirements and/or the use of analytical techniques that cannot isolate the roles that digit forces play in preventing object slip and controlling object position and orientation (pose). In our recent work, we used a dexterous manipulation task and decomposed digit forces into FG, the internal force that prevents object slip, and FM, the force responsible for object pose control. Unlike FG, FM was modulated from object lift onset to hold, suggesting their different sensitivity to sensory feedback acquired during object lift. However, the extent to which FG and FM can be controlled independently remains to be determined. Importantly, how FG and FM change as a function of object property is mathematically indeterminate and therefore requires active modulation. To address this gap, we systematically changed either object mass or external torque. The FM normal component responsible for object orientation control was modulated to changes in object torque but not mass. In contrast, FG was distinctly modulated to changes in object mass and torque. These findings point to a differential sensitivity of FG and FM to task requirements and provide novel insights into the neural control of dexterous manipulation. Importantly, our results indicate that the proposed digit force decomposition has the potential to capture important differences in how sensory inputs are processed and integrated to simultaneously ensure grasp stability and dexterous object pose control.NEW & NOTEWORTHY Successful dexterous object manipulation requires simultaneous prevention of object slip and object pose control. How these two task goals are attained can be investigated by decomposing digit forces into grasp and manipulation forces, respectively. We found that these forces were characterized by differential sensitivity to changes in object properties (mass and torque). This finding suggests the involvement of distinct sensorimotor mechanisms that, combined, simultaneously ensure grasp stability and dexterous control of object pose.

Visual sensitivity at the service of action control in posterior parietal cortex.

The posterior parietal cortex (PPC) serves as a crucial hub for the integration of sensory with motor cues related to voluntary actions. Visual input is used in different ways along the dorsomedial and the dorsolateral visual pathways. Here we focus on the dorsomedial pathway and recognize a visual representation at the service of action control. Employing different experimental paradigms applied to behaving monkeys while single neural activity is recorded from the medial PPC (area V6A), we show how plastic visual representation can be, matching the different contexts in which the same object is proposed. We also present data on the exchange between vision and arm actions and highlight how this rich interplay can be used to weight different sensory inputs in order to monitor and correct arm actions online. Indeed, neural activity during reaching or reach-to-grasp actions can be excited or inhibited by visual information, suggesting that the visual perception of action, rather than object recognition, is the most effective factor for area V6A. Also, three-dimensional object shape is encoded dynamically by the neural population, according to the behavioral context of the monkey. Along this line, mirror neuron discharges in V6A indicate the plasticity of visual representation of the graspable objects, that changes according to the context and peaks when the object is the target of one's own action. In other words, object encoding in V6A is a visual encoding for action.

Effects of left-hand contraction on tennis serve performance.

Introduction: The tennis serve is commonly executed in high-pressure scenarios, often leading to performance decline; a condition commonly referred to as choking under pressure. One suggested effective method to avert choking involves contracting the left hand. We examined the effects of left-hand contraction on tennis serve performance using a wearable grasping material (polyurethane foam) which can be incorporated into sportswear. Materials and methods: We assigned 40 right-handed skilled tennis players to either the contraction group (n = 20) or the no-contraction group (n = 20). They were instructed to perform a second-serve task during the pre-test and pressure test. The participants in the contraction group squeezed the grasping material for 20 s before executing the task in the pressure test. We measured performance, including total scores, the number of maximum score achievements, landing positions, and kinematic indices (i.e., ball speed, racket speed, and impact height). Results: Although neither group demonstrated deteriorated performance on the pressure test, the contraction group experienced an increased number of maximum score achievements under the pressure situation compared with the pre-test (p = 0.021). Discussion: Our results suggest that when under pressure, left-hand contraction may improve performance during tennis serves.

Secondary somatosensory and posterior insular cortices: a somatomotor hub for object prehension and manipulation movements.

The secondary somatosensory cortex (SII) and posterior insular cortex (pIC) are recognized for processing touch and movement information during hand manipulation in humans and non-human primates. However, their involvement in three-dimensional (3D) object manipulation remains unclear. To investigate neural activity related to hand manipulation in the SII/pIC, we trained two macaque monkeys to grasp three objects (a cone, a plate, and a ring) and engage in visual fixation on the object. Our results revealed that 19.4% (n = 50/257) of the task-related neurons in SII/pIC were active during hand manipulations, but did not respond to passive somatosensory stimuli. Among these neurons, 44% fired before hand-object contact (reaching to grasping neurons), 30% maintained tonic activity after contact (holding neurons), and 26% showed continuous discharge before and after contact (non-selective neurons). Object grasping-selectivity varied and was weak among these neurons, with only 24% responding to fixation of a 3D object (visuo-motor neurons). Even neurons unresponsive to passive visual stimuli showed responses to set-related activity before the onset of movement (42%, n = 21/50). Our findings suggest that somatomotor integration within SII/pIC is probably integral to all prehension sequences, including reaching, grasping, and object manipulation movements. Moreover, the existence of a set-related activity within SII/pIC may play a role in directing somatomotor attention during object prehension-manipulation in the absence of vision. Overall, SII/pIC may play a role as a somatomotor hub within the lateral grasping network that supports the generation of intentional hand actions based on haptic information.

Helping Blind People Grasp: Evaluating a Tactile Bracelet for Remotely Guiding Grasping Movements.

The problem of supporting visually impaired and blind people in meaningful interactions with objects is often neglected. To address this issue, we adapted a tactile belt for enhanced spatial navigation into a bracelet worn on the wrist that allows visually impaired people to grasp target objects. Participants' performance in locating and grasping target items when guided using the bracelet, which provides direction commands via vibrotactile signals, was compared to their performance when receiving auditory instructions. While participants were faster with the auditory commands, they also performed well with the bracelet, encouraging future development of this system and similar systems.

Computation on Demand: Action-Specific Representations of Visual Task Features Arise during Distinct Movement Phases.

The intention to act influences the computations of various task-relevant features. However, little is known about the time course of these computations. Furthermore, it is commonly held that these computations are governed by conjunctive neural representations of the features. But, support for this view comes from paradigms arbitrarily combining task features and affordances, thus requiring representations in working memory. Therefore, the present study used electroencephalography and a well-rehearsed task with features that afford minimal working memory representations to investigate the temporal evolution of feature representations and their potential integration in the brain. Female and male human participants grasped objects or touched them with a knuckle. Objects had different shapes and were made of heavy or light materials with shape and weight being relevant for grasping, not for "knuckling." Using multivariate analysis showed that representations of object shape were similar for grasping and knuckling. However, only for grasping did early shape representations reactivate at later phases of grasp planning, suggesting that sensorimotor control signals feed back to the early visual cortex. Grasp-specific representations of material/weight only arose during grasp execution after object contact during the load phase. A trend for integrated representations of shape and material also became grasp-specific but only briefly during the movement onset. These results suggest that the brain generates action-specific representations of relevant features as required for the different subcomponents of its action computations. Our results argue against the view that goal-directed actions inevitably join all features of a task into a sustained and unified neural representation.

CEPB dataset: a photorealistic dataset to foster the research on bin picking in cluttered environments.

Several datasets have been proposed in the literature, focusing on object detection and pose estimation. The majority of them are interested in recognizing isolated objects or the pose of objects in well-organized scenarios. This work introduces a novel dataset that aims to stress vision algorithms in the difficult task of object detection and pose estimation in highly cluttered scenes concerning the specific case of bin picking for the Cluttered Environment Picking Benchmark (CEPB). The dataset provides about 1.5M virtually generated photo-realistic images (RGB + depth + normals + segmentation) of 50K annotated cluttered scenes mixing rigid, soft, and deformable objects of varying sizes used in existing robotic picking benchmarks together with their 3D models (40 objects). Such images include three different camera positions, three light conditions, and multiple High Dynamic Range Imaging (HDRI) maps for domain randomization purposes. The annotations contain the 2D and 3D bounding boxes of the involved objects, the centroids' poses (translation + quaternion), and the visibility percentage of the objects' surfaces. Nearly 10K separated object images are presented to perform simple tests and compare them with more complex cluttered scenarios tests. A baseline performed with the DOPE neural network is reported to highlight the challenges introduced by the novel dataset.

Design and testing of fabric-based portable soft exoskeleton glove for hand grasping assistance in daily activity.

Hand exoskeleton robots have been developed as rehabilitation robots and assistive devices. Based on the material used, they can be soft or hard exoskeletons. Soft materials such as fabric can be used as a component of the wearable robot to increase comfortability. In this paper, we proposed an affordable soft hand exoskeleton based on fabric and motor-tendon actuation for hand flexion/extension motion assistance in daily activities. On-off control and PI compensator were implemented to regulate finger flexion and extension of the soft exoskeleton. The controllers were embedded into a microcontroller using Simulink software. The input signal command comes from the potentiometer and electromyography (EMG) sensor to drive the flexion/extension movement. Based on the experiments, the proposed controller successfully controlled the exoskeleton hand to facilitate a user in grasping various objects. The proposed soft hand exoskeleton is lightweight, comfortable, portable, and affordable, making it easily manufactured using available hardware and open-source code. The developed soft exoskeleton is a potential assistive device for a person who lost the ability to grasp objects.

Fully 3D-Printed Miniature Soft Hydraulic Actuators with Shape Memory Effect for Morphing and Manipulation.

Miniature shape-morphing soft actuators driven by external stimuli and fluidic pressure hold great promise in morphing matter and small-scale soft robotics. However, it remains challenging to achieve both rich shape morphing and shape locking in a fast and controlled way due to the limitations of actuation reversibility and fabrication. Here, fully 3D-printed, sub-millimeter thin-plate-like miniature soft hydraulic actuators with shape memory effect (SME) for programable fast shape morphing and shape locking, are reported. It combines commercial high-resolution multi-material 3D printing of stiff shape memory polymers (SMPs) and soft elastomers and direct printing of microfluidic channels and 2D/3D channel networks embedded in elastomers in a single print run. Leveraging spatial patterning of hybrid compositions and expansion heterogeneity of microfluidic channel networks for versatile hydraulically actuated shape morphing, including circular, wavy, helical, saddle, and warping shapes with various curvatures, are demonstrated. The morphed shapes can be temporarily locked and recover to their original planar forms repeatedly by activating SME of the SMPs. Utilizing the fast shape morphing and locking in the miniature actuators, their potential applications in non-invasive manipulation of small-scale objects and fragile living organisms, multimodal entanglement grasping, and energy-saving manipulators, are demonstrated.

A Soft Robot Tactile Finger Using Oxidation-Reduction Graphene-Polyurethane Conductive Sponge.

Currently, intelligent robotics is supplanting traditional industrial applications. It extends to business, service and care industries, and other fields. Stable robot grasping is a necessary prerequisite for all kinds of complex application scenarios. Herein, we propose a method for preparing an elastic porous material with adjustable conductivity, hardness, and elastic modulus. Based on this, we design a soft robot tactile fingertip that is gentle, highly sensitive, and has an adjustable range. It has excellent sensitivity (~1.089 kpa-1), fast response time (~35 ms), and measures minimum pressures up to 0.02 N and stability over 500 cycles. The baseline capacitance of a sensor of the same size can be increased by a factor of 5-6, and graphene adheres better to polyurethane sponge and has good shock absorption. In addition, we demonstrated the application of the tactile fingertip to a two-finger manipulator to achieve stable grasping. In this paper, we demonstrate the great potential of the soft robot tactile finger in the field of adaptive grasping for intelligent robots.

Towards Multi-Objective Object Push-Grasp Policy Based on Maximum Entropy Deep Reinforcement Learning under Sparse Rewards.

In unstructured environments, robots need to deal with a wide variety of objects with diverse shapes, and often, the instances of these objects are unknown. Traditional methods rely on training with large-scale labeled data, but in environments with continuous and high-dimensional state spaces, the data become sparse, leading to weak generalization ability of the trained models when transferred to real-world applications. To address this challenge, we present an innovative maximum entropy Deep Q-Network (ME-DQN), which leverages an attention mechanism. The framework solves complex and sparse reward tasks through probabilistic reasoning while eliminating the trouble of adjusting hyper-parameters. This approach aims to merge the robust feature extraction capabilities of Fully Convolutional Networks (FCNs) with the efficient feature selection of the attention mechanism across diverse task scenarios. By integrating an advantage function with the reasoning and decision-making of deep reinforcement learning, ME-DQN propels the frontier of robotic grasping and expands the boundaries of intelligent perception and grasping decision-making in unstructured environments. Our simulations demonstrate a remarkable grasping success rate of 91.6%, while maintaining excellent generalization performance in the real world.

Grasping affordance judgments depend on the object emotional value.

Introduction: The concept of affordance refers to the opportunities for action provided by the environment, often conveyed through visual information. It has been applied to explain visuomotor processing and movement planning. As emotion modulates both visual perception and the motor system, it is reasonable to ask whether emotion can influence affordance judgments. If present, this relationship can have important ontological implications for affordances. Thus, we investigated whether the emotional value of manipulable objects affected the judgment of the appropriate grasping that could be used to interact with them (i.e., their affordance). Methods: Volunteers were instructed to use a numerical scale to report their judgment on how an observed object should be grasped. We compared these judgments across emotional categories of objects (pleasant, unpleasant and neutral), while also considering the expected effect of object size. Results: We found that unpleasant objects were rated as more appropriately graspable by a precision grip than pleasant and neutral objects. Simultaneously, smaller object size also favored this judgment. This effect was seen in all emotional categories examined in equal magnitude. Discussion: Our findings suggest that the emotional value of objects modulates affordance judgments in a way that favors careful manipulation and minimal physical contact with aversive stimuli. Finally, we discuss how this affective aspect of our experience of objects overlaps with what affordances are conceptualized to be, calling for further reexamination of the relationship between affordances and emotions.

Soft Polymer-Actuated Compliant Microgripper with Adaptive Vibration-Controlled Grasp and Release.

Microgrippers that incorporate soft actuators are appropriate for micromanipulation or microsurgery owing to their ability to grasp objects without causing damage. However, developing a microgripper with a large gripping range that can produce a large force with high speed remains challenging in soft actuation mechanisms. Herein, we introduce a compliant microgripper driven by a soft dielectric elastomer actuator (DEA) called a spiral flexure cone DEA (SFCDEA). The submillimeter-scale SFCDEA exhibited a controllable linear displacement over a high bandwidth and the capability of lifting 100.9 g, which was 670 times higher than its mass. Subsequently, we developed a compliant microgripper based on the SFCDEA using smart composite microstructure technology to fabricate three-dimensional gripper linkages. We demonstrated that the microgripper was able to grasp various millimeter-scale objects with different shapes, sizes, and weights without a complex feedback control owing to its compliance. We proved the versatility of our gripper in robotic manipulation by demonstrating adaptive grasping and releasing of small objects using vibrations owing to its high bandwidth.

Multi-Degree-of-Freedom Force Sensor Incorporated into Soft Robotic Gripper for Improved Grasping Stability.

In recent years, soft robotic grippers have emerged as a promising solution for versatile and safe manipulation of objects in various fields. However, precise force control is critical, especially when handling delicate or fragile objects, to avoid excessive grip force application or to prevent object slippage. Herein, we propose a novel three-degree-of-freedom force sensor incorporated within a soft robotic gripper to realize stable grasping with force feedback. The proposed optical sensor employs lightweight and compact optical fibers, thereby allowing for cost-effective fabrication, and a robust sensing system that is immune to electromagnetic fields. By innervating the soft gripper with optical fibers, a durable system is achieved with the fibers functioning as a strengthening layer, thereby eliminating the need for embedding an external stiffening structure for efficient bending actuation. The innovative contact-based light loss sensing mechanism allows for a robust and stable sensing mechanism with low drift (<0.1% over 9000 cycles) that can be applied to soft pneumatic bending grippers. We used the developed sensor-incorporated soft gripper to grasp various objects, including magnetic materials, and achieved slip detection along with grip force feedback without any signal interference. Overall, this study proposes a robust measuring multi-degree-of-freedom force sensor that can be incorporated into grippers for improved grasping stability.

Endoskeleton Soft Multi-Fingered Hand with Variable Stiffness.

The use of a soft multi-fingered hand in handling fragile objects has been widely acknowledged. Nevertheless, high flexibility often results in decreased load capacity, necessitating the need for variable stiffness. This article introduces a new soft multi-fingered hand featuring variable stiffness. The finger of the hand has three chambers and an endoskeleton mechanism. Two chambers facilitate bending and swinging motions, whereas the third adjusts stiffness. An endoskeleton mechanism is embedded in the third chamber, and the friction between its moving parts increases as negative air pressure rises, causing the finger's stiffness to increase. This mechanism can alter its stiffness in any configuration, which is particularly useful in manipulating irregular-shaped fragile objects post-grasping. The effectiveness of the proposed soft multi-fingered hand is validated through five experiments: stiffness adjustment, finger stiffening under a specific orientation, bulb screwing, heavy object lifting, and bean curd grasping. The results demonstrate that the proposed soft multi-fingered hand exhibits robust grasping capabilities for various fragile objects.

Mixed selectivity in monkey anterior intraparietal area during visual and motor processes.

Classical studies suggest that the anterior intraparietal area (AIP) contributes to the encoding of specific information such as objects and actions of self and others, through a variety of neuronal classes, such as canonical, motor and mirror neurons. However, these studies typically focused on a single variable, leaving it unclear whether distinct sets of AIP neurons encode a single or multiple sources of information and how multimodal coding emerges. Here, we chronically recorded monkey AIP neurons in a variety of tasks and conditions classically employed in separate experiments. Most cells exhibited mixed selectivity for observed objects, executed actions, and observed actions, enhanced when this information came from the monkey's peripersonal working space. In contrast with the classical view, our findings indicate that multimodal coding emerges in AIP from partially-mixed selectivity of individual neurons for a variety of information relevant for planning actions directed to both physical objects and other subjects.

Reinforcement Learning Algorithms and Applications in Healthcare and Robotics: A Comprehensive and Systematic Review.

Reinforcement learning (RL) has emerged as a dynamic and transformative paradigm in artificial intelligence, offering the promise of intelligent decision-making in complex and dynamic environments. This unique feature enables RL to address sequential decision-making problems with simultaneous sampling, evaluation, and feedback. As a result, RL techniques have become suitable candidates for developing powerful solutions in various domains. In this study, we present a comprehensive and systematic review of RL algorithms and applications. This review commences with an exploration of the foundations of RL and proceeds to examine each algorithm in detail, concluding with a comparative analysis of RL algorithms based on several criteria. This review then extends to two key applications of RL: robotics and healthcare. In robotics manipulation, RL enhances precision and adaptability in tasks such as object grasping and autonomous learning. In healthcare, this review turns its focus to the realm of cell growth problems, clarifying how RL has provided a data-driven approach for optimizing the growth of cell cultures and the development of therapeutic solutions. This review offers a comprehensive overview, shedding light on the evolving landscape of RL and its potential in two diverse yet interconnected fields.

Healthy adults favor stable left/right hand choices over performance at an unconstrained reach-to-grasp task.

Reach-to-grasp actions are fundamental to the daily activities of human life, but few methods exist to assess individuals' reaching and grasping actions in unconstrained environments. The Block Building Task (BBT) provides an opportunity to directly observe and quantify these actions, including left/right hand choices. Here we sought to investigate the motor and non-motor causes of left/right hand choices, and optimize the design of the BBT, by manipulating motor and non-motor difficulty in the BBT's unconstrained reach-to-grasp task. We hypothesized that greater motor and non-motor (e.g. cognitive/perceptual) difficulty would drive increased usage of the dominant hand. To test this hypothesis, we modulated block size (large vs. small) to influence motor difficulty, and model complexity (10 vs. 5 blocks per model) to influence non-motor difficulty, in healthy adults (n = 57). Our data revealed that increased motor and non-motor difficulty led to lower task performance (slower task speed), but participants only increased use of their dominant hand only under the most difficult combination of conditions: in other words, participants allowed their performance to degrade before changing hand choices, even though participants were instructed only to optimize performance. These results demonstrate that hand choices during reach-to grasp actions are more stable than motor performance in healthy right-handed adults, but tasks with multifaceted difficulties can drive individuals to rely more on their dominant hand.