Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions.

When exposed to complementary features of information across sensory modalities, our brains formulate cross-modal associations between features of stimuli presented separately to multiple modalities. For example, auditory pitch-visual size associations map high-pitch tones with small-size visual objects, and low-pitch tones with large-size visual objects. Preferential, or congruent, cross-modal associations have been shown to affect behavioural performance, i.e. choice accuracy and reaction time (RT) across multisensory decision-making paradigms. However, the neural mechanisms underpinning such influences in perceptual decision formation remain unclear. Here, we sought to identify when perceptual improvements from associative congruency emerge in the brain during decision formation. In particular, we asked whether such improvements represent 'early' sensory processing benefits, or 'late' post-sensory changes in decision dynamics. Using a modified version of the Implicit Association Test (IAT), coupled with electroencephalography (EEG), we measured the neural activity underlying the effect of auditory stimulus-driven pitch-size associations on perceptual decision formation. Behavioural results showed that participants responded significantly faster during trials when auditory pitch was congruent, rather than incongruent, with its associative visual size counterpart. We used multivariate Linear Discriminant Analysis (LDA) to characterise the spatiotemporal dynamics of EEG activity underpinning IAT performance. We found an 'Early' component (∼100-110 ms post-stimulus onset) coinciding with the time of maximal discrimination of the auditory stimuli), and a 'Late' component (∼330-340 ms post-stimulus onset) underlying IAT performance. To characterise the functional role of these components in decision formation, we incorporated a neurally-informed Hierarchical Drift Diffusion Model (HDDM), revealing that the Late component decreases response caution, requiring less sensory evidence to be accumulated, whereas the Early component increased the duration of sensory-encoding processes for incongruent trials. Overall, our results provide a mechanistic insight into the contribution of 'early' sensory processing, as well as 'late' post-sensory neural representations of associative congruency to perceptual decision formation.

Auditory information enhances post-sensory visual evidence during rapid multisensory decision-making.

Despite recent progress in understanding multisensory decision-making, a conclusive mechanistic account of how the brain translates the relevant evidence into a decision is lacking. Specifically, it remains unclear whether perceptual improvements during rapid multisensory decisions are best explained by sensory (i.e., 'Early') processing benefits or post-sensory (i.e., 'Late') changes in decision dynamics. Here, we employ a well-established visual object categorisation task in which early sensory and post-sensory decision evidence can be dissociated using multivariate pattern analysis of the electroencephalogram (EEG). We capitalize on these distinct neural components to identify when and how complementary auditory information influences the encoding of decision-relevant visual evidence in a multisensory context. We show that it is primarily the post-sensory, rather than the early sensory, EEG component amplitudes that are being amplified during rapid audiovisual decision-making. Using a neurally informed drift diffusion model we demonstrate that a multisensory behavioral improvement in accuracy arises from an enhanced quality of the relevant decision evidence, as captured by the post-sensory EEG component, consistent with the emergence of multisensory evidence in higher-order brain areas.

Biofeedback Signals for Robotic Rehabilitation: Assessment of Wrist Muscle Activation Patterns in Healthy Humans.

Electrophysiological recordings from human muscles can serve as control signals for robotic rehabilitation devices. Given that many diseases affecting the human sensorimotor system are associated with abnormal patterns of muscle activation, such biofeedback can optimize human-robot interaction and ultimately enhance motor recovery. To understand how mechanical constraints and forces imposed by a robot affect muscle synergies, we mapped the muscle activity of seven major arm muscles in healthy individuals performing goal-directed discrete wrist movements constrained by a wrist robot. We tested six movement directions and four force conditions typically experienced during robotic rehabilitation. We analyzed electromyographic (EMG) signals using a space-by-time decomposition and we identified a set of spatial and temporal modules that compactly described the EMG activity and were robust across subjects. For each trial, coefficients expressing the strength of each combination of modules and representing the underlying muscle recruitment, allowed for a highly reliable decoding of all experimental conditions. The decomposition provides compact representations of the observable muscle activation constrained by a robotic device. Results indicate that a low-dimensional control scheme incorporating EMG biofeedback could be an effective add-on for robotic rehabilitative protocols seeking to improve impaired motor function in humans.