The integration of action-oriented multisensory information from target and limb within the movement planning and execution.

The planning and execution of a grasping or reaching movement toward targets we sense with the other hand requires integrating multiple sources of sensory information about the limb performing the movement and the target of the action. In the last two decades, several sensory and motor control theories have thoroughly described how this multisensory-motor integration process occurs. However, even though these theories were very influential in their respective field, they lack a clear, unified vision of how target-related and movement-related multisensory information integrates within the action planning and execution phases. This brief review aims to summarize the most influential theories in multisensory integration and sensory-motor control by underscoring their critical points and hidden connections, providing new ideas on the multisensory-motor integration process. Throughout the review, I wll propose an alternative view of how the multisensory integration process unfolds along the action planning and execution and I will make several connections with the existent multisensory-motor control theories.

Visual uncertainty unveils the distinct role of haptic cues in multisensory grasping.

Human multisensory grasping movements (i.e., seeing and feeling a handheld object while grasping it with the contralateral hand) are superior to movements guided by each separate modality. This multisensory advantage might be driven by the integration of vision with either the haptic position only or with both position and size cues. To contrast these two hypotheses, we manipulated visual uncertainty (central vs. peripheral vision) and the availability of haptic cues during multisensory grasping. We showed a multisensory benefit irrespective of the degree of visual uncertainty suggesting that the integration process involved in multisensory grasping can be flexibly modulated by the contribution of each modality. Increasing visual uncertainty revealed the role of the distinct haptic cues. The haptic position cue was sufficient to promote multisensory benefits evidenced by faster actions with smaller grip apertures, whereas the haptic size was fundamental in fine-tuning the grip aperture scaling. These results support the hypothesis that, in multisensory grasping, vision is integrated with all haptic cues, with the haptic position cue playing the key part. Our findings highlight the important role of non-visual sensory inputs in sensorimotor control and hint at the potential contributions of the haptic modality in developing and maintaining visuomotor functions.Significance statementThe longstanding view that vision is considered the primary sense we rely on to guide grasping movements relegates the equally important haptic inputs, such as touch and proprioception, to a secondary role. Here we show that by increasing visual uncertainty during visuo-haptic grasping, the central nervous system exploits distinct haptic inputs about the object position and size to optimize grasping performance. Specifically, we demonstrate that haptic inputs about the object position are fundamental to support vision in enhancing grasping performance, whereas haptic size inputs can further refine hand shaping. Our results provide strong evidence that non-visual inputs serve an important, previously under-appreciated, functional role in grasping.

A brief glimpse at a haptic target is sufficient for multisensory integration in reaching movements.

Goal-directed aiming movements toward visuo-haptic targets (i.e., seen and handheld targets) are generally more precise than those toward visual only or haptic only targets. This multisensory advantage stems from a continuous inflow of haptic and visual target information during the movement planning and execution phases. However, in everyday life, multisensory movements often occur without the support of continuous visual information. Here we investigated whether and to what extent limiting visual information to the initial stage of the action still leads to a multisensory advantage. Participants were asked to reach a handheld target while vision was briefly provided during the movement planning phase (50 ms, 100 ms, 200 ms of vision before movement onset), or during the planning and early execution phases (400 ms of vision), or during the entire movement. Additional conditions were performed in which only haptic target information was provided, or, only vision was provided either briefly (50 ms, 100 ms, 200 ms, 400 ms) or throughout the entire movement. Results showed that 50 ms of vision before movement onset were sufficient to trigger a direction-specific visuo-haptic integration process that increased endpoint precision. We conclude that, when a continuous support of vision is not available, endpoint precision is determined by the less recent, but most reliable multisensory information rather than by the latest unisensory (haptic) inputs.

Integration of haptics and vision in human multisensory grasping.

Grasping actions are directed not only toward objects we see but also toward objects we both see and touch (multisensory grasping). In this latter case, the integration of visual and haptic inputs improves movement performance compared to each sense alone. This performance advantage could be due to the integration of all the redundant positional and size cues or to the integration of only a subset of these cues. Here we selectively provided specific cues to tease apart how these different sensory sources contribute to visuo-haptic multisensory grasping. We demonstrate that the availability of the haptic positional cue together with the visual cues is sufficient to achieve the same grasping performance as when all cues are available. These findings provide strong evidence that the human sensorimotor system relies on non-visual sensory inputs and open new perspectives on their role in supporting vision during both development and adulthood.

Grasping adjustments to haptic, visual, and visuo-haptic object perturbations are contingent on the sensory modality.

Haptics provides information about the size and position of a handheld object. However, it is still unknown how haptics contributes to action correction if a sudden perturbation causes a change in the configuration of the handheld object. In this study, we have occasionally perturbed the size of an object that was the target of a right-hand reach-to-grasp movement. In some cases, participants were holding the target object with their left hand, which provided haptic information about the object perturbation. We compared the corrective responses to perturbations in three different sensory conditions: visual (participants had full vision of the object, but haptic information from the left hand was prevented), haptic (object size was sensed by the left hand and vision was prevented), and visuo-haptic (both visual and haptic information were available throughout the movement). We found that haptic inputs evoked faster contralateral corrections than visual inputs, although actions in haptic and visual conditions were similar in movement duration. Strikingly, the corrective responses in the visuo-haptic condition were as fast as those found in the haptic condition, a result that is contrary to that predicted by simple summation of unisensory signals. These results suggest the existence of a haptomotor reflex that can trigger automatic and efficient grasping corrections of the contralateral hand that are faster than those initiated by the well-known visuomotor reflex and the tactile-motor reflex.NEW & NOTEWORTHY We show that online grip aperture corrections during grasping actions are contingent on the sensory modality used to detect the object perturbation. We found that sensing perturbations with the contralateral hand only (haptics) leads to faster action corrections than when object perturbations are only visually sensed. Moreover, corrections following visuo-haptic perturbations were as fast as those to haptic perturbations. Thus a haptomotor reflex triggers faster automatic responses than the visuomotor reflex.

Grasping movements toward seen and handheld objects.

Grasping movements are typically performed toward visually sensed objects. However, planning and execution of grasping movements can be supported also by haptic information when we grasp objects held in the other hand. In the present study we investigated this sensorimotor integration process by comparing grasping movements towards objects sensed through visual, haptic or visuo-haptic signals. When movements were based on haptic information only, hand preshaping was initiated earlier, the digits closed on the object more slowly, and the final phase was more cautious compared to movements based on only visual information. Importantly, the simultaneous availability of vision and haptics led to faster movements and to an overall decrease of the grip aperture. Our findings also show that each modality contributes to a different extent in different phases of the movement, with haptics being more crucial in the initial phases and vision being more important for the final on-line control. Thus, vision and haptics can be flexibly combined to optimize the execution of grasping movement.

Expert players accurately detect an opponent's movement intentions through sound alone.

Sounds offer a rich source of information about events taking place in our physical and social environment. However, outside the domains of speech and music, little is known about whether humans can recognize and act upon the intentions of another agent's actions detected through auditory information alone. In this study we assessed whether intention can be inferred from the sound an action makes, and in turn, whether this information can be used to prospectively guide movement. In 2 experiments experienced and novice basketball players had to virtually intercept an attacker by listening to audio recordings of that player's movements. In the first experiment participants had to move a slider, while in the second one their body, to block the perceived passage of the attacker as they would in a real basketball game. Combinations of deceptive and nondeceptive movements were used to see if novice and/or experienced listeners could perceive the attacker's intentions through sound alone. We showed that basketball players were able to more accurately predict final running direction compared to nonplayers, particularly in the second experiment when the interceptive action was more basketball specific. We suggest that athletes present better action anticipation by being able to pick up and use the relevant kinematic features of deceptive movement from event-related sounds alone. This result suggests that action intention can be perceived through the sound a movement makes and that the ability to determine another person's action intention from the information conveyed through sound is honed through practice. (PsycINFO Database Record

To Hear or Not to Hear: Sound Availability Modulates Sensory-Motor Integration.

When we walk in place with our eyes closed after a few minutes of walking on a treadmill, we experience an unintentional forward body displacement (drift), called the sensory-motor aftereffect. Initially, this effect was thought to be due to the mismatch experienced during treadmill walking between the visual (absence of optic flow signaling body steadiness) and proprioceptive (muscle spindles firing signaling body displacement) information. Recently, the persistence of this effect has been shown even in the absence of vision, suggesting that other information, such as the sound of steps, could play a role. To test this hypothesis, six cochlear-implanted individuals were recruited and their forward drift was measured before (Control phase) and after (Post Exercise phase) walking on a treadmill while having their cochlear system turned on and turned off. The relevance in testing cochlear-implanted individuals was that when their system is turned off, they perceive total silence, even eliminating the sounds normally obtained from bone conduction. Results showed the absence of the aftereffect when the system was turned off, underlining the fundamental role played by sounds in the control of action and breaking new ground in the use of interactive sound feedback in motor learning and motor development.

Interactive footstep sounds modulate the perceptual-motor aftereffect of treadmill walking.

In this study, we investigated the role of interactive auditory feedback in modulating the inadvertent forward drift experienced while attempting to walk in place with closed eyes following a few minutes of treadmill walking. Simulations of footstep sounds upon surface materials such as concrete and snow were provided by means of a system composed of headphones and shoes augmented with sensors. In a control condition, participants could hear their actual footstep sounds. Results showed an overall enhancement of the forward drift after treadmill walking independent of the sound perceived, while the strength of the aftereffect, measured as the proportional increase (posttest/pretest) in forward drift, was higher under the influence of snow compared to both concrete and actual sound. In addition, a higher knee angle flexion was found during the snow sound condition both before and after treadmill walking. Behavioral results confirmed those of a perceptual questionnaire, which showed that the snow sound was effective in producing strong pseudo-haptic illusions. Our results provide evidence that the walking in place aftereffect results from a recalibration of haptic, visuo-motor but also sound-motor control systems. Self-motion perception is multimodal.

Might as well jump: sound affects muscle activation in skateboarding.

The aim of the study is to reveal the role of sound in action anticipation and performance, and to test whether the level of precision in action planning and execution is related to the level of sensorimotor skills and experience that listeners possess about a specific action. Individuals ranging from 18 to 75 years of age--some of them without any skills in skateboarding and others experts in this sport--were compared in their ability to anticipate and simulate a skateboarding jump by listening to the sound it produces. Only skaters were able to modulate the forces underfoot and to apply muscle synergies that closely resembled the ones that a skater would use if actually jumping on a skateboard. More importantly we showed that only skaters were able to plan the action by activating anticipatory postural adjustments about 200 ms after the jump event. We conclude that expert patterns are guided by auditory events that trigger proper anticipations of the corresponding patterns of movements.