Biohybrid Robotic Hand to Investigate Tactile Encoding and Sensorimotor Integration.

For people who have experienced a spinal cord injury or an amputation, the recovery of sensation and motor control could be incomplete despite noteworthy advances with invasive neural interfaces. Our objective is to explore the feasibility of a novel biohybrid robotic hand model to investigate aspects of tactile sensation and sensorimotor integration with a pre-clinical research platform. Our new biohybrid model couples an artificial hand with biological neural networks (BNN) cultured in a multichannel microelectrode array (MEA). We decoded neural activity to control a finger of the artificial hand that was outfitted with a tactile sensor. The fingertip sensations were encoded into rapidly adapting (RA) or slowly adapting (SA) mechanoreceptor firing patterns that were used to electrically stimulate the BNN. We classified the coherence between afferent and efferent electrodes in the MEA with a convolutional neural network (CNN) using a transfer learning approach. The BNN exhibited the capacity for functional specialization with the RA and SA patterns, represented by significantly different robotic behavior of the biohybrid hand with respect to the tactile encoding method. Furthermore, the CNN was able to distinguish between RA and SA encoding methods with 97.84% ± 0.65% accuracy when the BNN was provided tactile feedback, averaged across three days in vitro (DIV). This novel biohybrid research platform demonstrates that BNNs are sensitive to tactile encoding methods and can integrate robotic tactile sensations with the motor control of an artificial hand. This opens the possibility of using biohybrid research platforms in the future to study aspects of neural interfaces with minimal human risk.

Multichannel Sensorimotor Integration with a Dexterous Artificial Hand.

Background: People use their hands to perform sophisticated tasks like playing a musical instrument by integrating manifold and diverse sensations of touch with motor control strategies. In contrast, prosthetic hands lack the capacity for multichannel haptic feedback and multitasking functionality remains rudimentary. There is a dearth of research exploring the potential of upper limb absent (ULA) people to integrate multiple channels of haptic feedback into dexterous prosthetic hand control strategies. Methods: In this paper, we designed a novel experimental paradigm for three ULA people and nine additional subjects to investigate their ability to integrate two simultaneously activated channels of context-specific haptic feedback into their dexterous artificial hand control strategies. Artificial neural networks (ANN) were designed for pattern recognition of the array of efferent electromyogram signals that controlled the dexterous artificial hand. ANNs were also used to classify the directions that objects were sliding across two tactile sensor arrays on the index (I) and little (L) fingertips of the robotic hand. The direction of sliding contact at each robotic fingertip was encoded by different stimulation frequencies of wearable vibrotactile actuators for haptic feedback. The subjects were tasked with implementing different control strategies with each finger simultaneously depending upon the perceived directions of sliding contact. This required the 12 subjects to concurrently control individual fingers of the artificial hand by successfully interpreting two channels of simultaneously activated context-specific haptic feedback. Results: Subjects were able to accomplish this complex feat of multichannel sensorimotor integration with an overall accuracy of 95.53% ± 0.23%. While there was no statistically significant difference in the classification accuracy between ULA people and the other subjects, the ULA people required more time to correctly respond to the simultaneous haptic feedback slip signals, suggesting a higher cognitive load required by the ULA people. Conclusion: ULA people can integrate multiple channels of simultaneously activated and nuanced haptic feedback with their control of individual fingers of an artificial hand. These findings provide a step toward empowering amputees to multitask with dexterous prosthetic hands, which remains an ongoing challenge.

Human neuromarkers of tactile perception: state of the art in methods and findings.

Tactile perception is a multifaceted sense with complicated convergent/divergent peripheral pathways. Its neuromarkers remain poorly understood, due to the sense's inherent complexity and the confounding factor of intricate motor, cognitive and affective correlates. This gap hinders research evaluating interventions to restore touch in artificial hands. We inventorize state-of-the-art and recent innovations in control systems with soft and hard robotics that are poised to unlock more targeted non-invasive stimulations. We review neuromarkers observed for pressure, vibration, brushing, texture discrimination, pain, heat and cold, complemented with the covariates from movement, attention, working memory, multisensory and sensorimotor integration or competition (audition, vision) and affect. We analyze neural oscillations during sensory and (peripheral and central) electro-magnetic stimulation. This review matures a framework of reverse prediction, in which non-invasive observation of neural activity robustly and unobtrusively quantifies tactile perception.

Multichannel haptic feedback unlocks prosthetic hand dexterity.

Loss of tactile sensations is a major roadblock preventing upper limb-absent people from multitasking or using the full dexterity of their prosthetic hands. With current myoelectric prosthetic hands, limb-absent people can only control one grasp function at a time even though modern artificial hands are mechanically capable of individual control of all five digits. In this paper, we investigated whether people could precisely control the grip forces applied to two different objects grasped simultaneously with a dexterous artificial hand. Toward that end, we developed a novel multichannel wearable soft robotic armband to convey artificial sensations of touch to the robotic hand users. Multiple channels of haptic feedback enabled subjects to successfully grasp and transport two objects simultaneously with the dexterous artificial hand without breaking or dropping them, even when their vision of both objects was obstructed. Simultaneous transport of the objects provided a significant time savings to perform the deliveries in comparison to a one-at-a-time approach. This paper demonstrated that subjects were able to integrate multiple channels of haptic feedback into their motor control strategies to perform a complex simultaneous object grasp control task with an artificial limb, which could serve as a paradigm shift in the way prosthetic hands are operated.

Robotically Embodied Biological Neural Networks to Investigate Haptic Restoration with Neuroprosthetic Hands.

Neuroprosthetic limbs reconnect severed neural pathways for control of (and increasingly sensation from) an artificial limb. However, the plastic interaction between robotic and biological components is poorly understood. To gain such insight, we developed a novel noninvasive neuroprosthetic research platform that enables bidirectional electrical communications (action, sensory perception) between a dexterous artificial hand and neuronal cultures living in a multichannel microelectrode array (MEA) chamber. Artificial tactile sensations from robotic fingertips were encoded to mimic slowly adapting (SA) or rapidly adapting (RA) mechanoreceptors. Afferent spike trains were used to stimulate neurons in a region of the neuronal culture. Electrical activity from neurons at another region in the MEA chamber was used as the motor control signal for the artificial hand. Results from artificial neural networks (ANNs) showed that the haptic model used to encode RA or SA fingertip sensations affected biological neural network (BNN) activity patterns, which in turn impacted the behavior of the artificial hand. That is, the exhibited finger tapping behavior of this closed-loop neurorobotic system showed statistical significance (p<0.01) between the haptic encoding methods across two different neuronal cultures and over multiple days. These findings suggest that our noninvasive neuroprosthetic research platform can be used to devise high-throughput experiments exploring how neural plasticity is affected by the mutual interactions between perception and action.

Hierarchical Tactile Sensation Integration from Prosthetic Fingertips Enables Multi-Texture Surface Recognition.

Multifunctional flexible tactile sensors could be useful to improve the control of prosthetic hands. To that end, highly stretchable liquid metal tactile sensors (LMS) were designed, manufactured via photolithography, and incorporated into the fingertips of a prosthetic hand. Three novel contributions were made with the LMS. First, individual fingertips were used to distinguish between different speeds of sliding contact with different surfaces. Second, differences in surface textures were reliably detected during sliding contact. Third, the capacity for hierarchical tactile sensor integration was demonstrated by using four LMS signals simultaneously to distinguish between ten complex multi-textured surfaces. Four different machine learning algorithms were compared for their successful classification capabilities: K-nearest neighbor (KNN), support vector machine (SVM), random forest (RF), and neural network (NN). The time-frequency features of the LMSs were extracted to train and test the machine learning algorithms. The NN generally performed the best at the speed and texture detection with a single finger and had a 99.2 ± 0.8% accuracy to distinguish between ten different multi-textured surfaces using four LMSs from four fingers simultaneously. The capability for hierarchical multi-finger tactile sensation integration could be useful to provide a higher level of intelligence for artificial hands.

Robotic Replica of a Human Spine Uses Soft Magnetic Sensor Array to Forecast Intervertebral Loads and Posture after Surgery.

Cervical disc implants are conventional surgical treatments for patients with degenerative disc disease, such as cervical myelopathy and radiculopathy. However, the surgeon still must determine the candidacy of cervical disc implants mainly from the findings of diagnostic imaging studies, which can sometimes lead to complications and implant failure. To help address these problems, a new approach was developed to enable surgeons to preview the post-operative effects of an artificial disc implant in a patient-specific fashion prior to surgery. To that end, a robotic replica of a person's spine was 3D printed, modified to include an artificial disc implant, and outfitted with a soft magnetic sensor array. The aims of this study are threefold: first, to evaluate the potential of a soft magnetic sensor array to detect the location and amplitude of applied loads; second, to use the soft magnetic sensor array in a 3D printed human spine replica to distinguish between five different robotically actuated postures; and third, to compare the efficacy of four different machine learning algorithms to classify the loads, amplitudes, and postures obtained from the first and second aims. Benchtop experiments showed that the soft magnetic sensor array was capable of precisely detecting the location and amplitude of forces, which were successfully classified by four different machine learning algorithms that were compared for their capabilities: Support Vector Machine (SVM), K-Nearest Neighbor (KNN), Random Forest (RF), and Artificial Neural Network (ANN). In particular, the RF and ANN algorithms were able to classify locations of loads applied 3.25 mm apart with 98.39% ± 1.50% and 98.05% ± 1.56% accuracies, respectively. Furthermore, the ANN had an accuracy of 94.46% ± 2.84% to classify the location that a 10 g load was applied. The artificial disc-implanted spine replica was subjected to flexion and extension by a robotic arm. Five different postures of the spine were successfully classified with 100% ± 0.0% accuracy with the ANN using the soft magnetic sensor array. All results indicated that the magnetic sensor array has promising potential to generate data prior to invasive surgeries that could be utilized to preoperatively assess the suitability of a particular intervention for specific patients and to potentially assist the postoperative care of people with cervical disc implants.

Diabetic Retinopathy (DR) Severity Level Classification Using Multimodel Convolutional Neural Networks.

Diabetic retinopathy (DR) is a progressive eye disease that affects a large portion of working-age adults. DR, which may progress to an irreversible state that causes blindness, can be diagnosed with a comprehensive dilated eye exam. With the eye dilated, the Doctor takes pictures of the inside of the eye via a medical procedure called Fluorescein Angiography, in which a dye is injected into the bloodstream. The dye highlights the blood vessels in the back of the eye so they can be photographed. In addition, the Doctor may request an Optical Coherence Tomography (OCT) exam, by which cross-sectional photos of the retina are produced to measure the thickness of the retina. Early prognostication is vital in treating the disease and preventing it from progressing into advanced irreversible stages. Skilled medical personnel and necessary medical facilities are required to detect DR in its five major stages. In this paper, we propose a diagnostic tool to detect Diabetic retinopathy from fundus images by using an ensemble of multi-inception CNN networks. Our inception block consists of three Convolutional layers with kernel sizes of 3x3, 5x5, and 1x1 that are concatenated deeply and forwarded to the max-pooling layer. We experimentally compare our proposed method with two pre-trained models: VGG16 and GoogleNets. The experiment results show that the proposed method can achieve an accuracy of 93.2% by an ensemble of 10 random networks, compared to 81% obtained with transfer learning based on VGG19.

Armband with Soft Robotic Actuators and Vibrotactile Stimulators for Bimodal Haptic Feedback from a Dexterous Artificial Hand.

The haptic sense relies upon a plurality of receptors and pathways to produce a complex perceptual experience of contact, pressure, taps, vibrations and flutters. This complexity is yet to be reproduced in haptic feedback interfaces that are used by people controlling a dexterous robotic hand, be it for limb-absence or teleoperation. The goal of the present bimodal haptic armband is to convey both low-frequency pressure changes and high-frequency vibrations from a dexterous robotic hand to a human's upper arm, so as to guide his/her control of the artificial limb. To that end, we design and manufacture four novel soft robotic armbands combining inflatable air chambers and vibrotactile stimulators. We develop control systems for both pathways. We conduct a series of benchtop tests to determine the pneumatic and vibrotactile performance and select from competing designs and materials. We test two of the resulting bimodal haptic armband on human subjects and confirm their ability to use both aspects of this haptic information. Arguing that dexterous artificial hands are presently not used to their fullest capability by the dearth of haptic information in users, this work aims to achieve a more realistic tactile experience for a fluent, more natural usage of robotic artificial hands.

Direction of Slip Detection for Adaptive Grasp Force Control with a Dexterous Robotic Hand.

A novel method of tactile communication among human-robot and robot-robot collaborative teams is developed for the purpose of adaptive grasp control of dexterous robotic hands. Neural networks are applied to the problem of classifying the direction objects slide against different tactile fingertip sensors in real-time. This ability to classify the direction that an object slides in a dexterous robotic hand was used for adaptive grasp synergy control to afford context dependent robotic reflexes in response to the direction of grasped object slip. Case studies with robot-robot and human-robot collaborative teams successfully demonstrated the feasibility; when object slip in the direction of gravity (towards the ground) was detected, the dexterous hand increased the grasp force to prevent dropping the object. When a human or robot applied an upward force to cause the grasped object to slip upward, the dexterous hand was programmed to release the object into the hand of the other team member. This method of adaptive grasp control using direction of slip detection can improve the efficiency of human-robot and robot-robot teams.