Computation on demand: Action-specific representations of visual task features arise during distinct movement phases.

The intention to act influences the computations of various task-relevant features. However, little is known about the time course of these computations. Furthermore, it is commonly held that these computations are governed by conjunctive neural representations of the features. But, support for this view comes from paradigms arbitrarily combining task features and affordances, thus, requiring representations in working memory. Therefore, the present study used electroencephalography and a well-rehearsed task with features that afford minimal working memory representations to investigate the temporal evolution of feature representations and their potential integration in the brain. Female and male human participants viewed and grasped objects or touched them with a knuckle. Objects had different shapes and were made of heavy or light materials with shape and weight being relevant for grasping, not for "knuckling." Using multivariate analysis showed that representations of object shape were similar for grasping and knuckling. However, only for grasping did early shape representations reactivate at later phases of grasp planning, suggesting that sensorimotor control signals feed back to early visual cortex. Grasp-specific representations of material/weight only arose during grasp execution after object contact during the load phase. A trend for integrated representations of shape and material also became grasp-specific but only briefly during Movement onset. These results suggest that the brain generates action-specific representations of relevant features as required for the different subcomponents of its action computations. Our results argue against the view that goal-directed actions inevitably join all features of a task into a sustained and unified neural representation.Significance statement The idea that all the features of a task are integrated into a joint representation or event file is widely supported but importantly based on paradigms with arbitrary stimulus-response combinations. Our study is the first to investigate grasping using electroencephalography to search for the neural basis of feature integration in such a daily-life task with overlearned stimulus-response mappings. Contrary to the notion of event files we find limited evidence for integrated representations. Instead, we find that task-relevant features form representations at specific phases of the action, suggesting that action intentions reactivate representations of relevant features. Our results show that integrated representations do not occur universally for any kind of goal-directed behaviour but in a manner of computation on demand.

Saccades and presaccadic stimulus repetition alter cortical network topology and dynamics: evidence from EEG and graph theoretical analysis.

Parietal and frontal cortex are involved in saccade generation, and their output signals modify visual signals throughout cortex. Local signals associated with these interactions are well described, but their large-scale progression and network dynamics are unknown. Here, we combined source localized electroencephalography (EEG) and graph theory analysis (GTA) to understand how saccades and presaccadic visual stimuli interactively alter cortical network dynamics in humans. Twenty-one participants viewed 1-3 vertical/horizontal grids, followed by grid with the opposite orientation just before a horizontal saccade or continued fixation. EEG signals from the presaccadic interval (or equivalent fixation period) were used for analysis. Source localization-through-time revealed a rapid frontoparietal progression of presaccadic motor signals and stimulus-motor interactions, with additional band-specific modulations in several frontoparietal regions. GTA analysis revealed a saccade-specific functional network with major hubs in inferior parietal cortex (alpha) and the frontal eye fields (beta), and major saccade-repetition interactions in left prefrontal (theta) and supramarginal gyrus (gamma). This network showed enhanced segregation, integration, synchronization, and complexity (compared with fixation), whereas stimulus repetition interactions reduced synchronization and complexity. These cortical results demonstrate a widespread influence of saccades on both regional and network dynamics, likely responsible for both the motor and perceptual aspects of saccades.

Multivariate Analysis of Electrophysiological Signals Reveals the Time Course of Precision Grasps Programs: Evidence for Nonhierarchical Evolution of Grasp Control.

Current understanding of the neural processes underlying human grasping suggests that grasp computations involve gradients of higher to lower level representations and, relatedly, visual to motor processes. However, it is unclear whether these processes evolve in a strictly canonical manner from higher to intermediate and to lower levels given that this knowledge importantly relies on functional imaging, which lacks temporal resolution. To examine grasping in fine temporal detail here we used multivariate EEG analysis. We asked participants to grasp objects while controlling the time at which crucial elements of grasp programs were specified. We first specified the orientation with which participants should grasp objects, and only after a delay we instructed participants about which effector to use to grasp, either the right or the left hand. We also asked participants to grasp with both hands because bimanual and left-hand grasping share intermediate-level grasp representations. We observed that grasp programs evolved in a canonical manner from visual representations, which were independent of effectors to motor representations that distinguished between effectors. However, we found that intermediate representations of effectors that partially distinguished between effectors arose after representations that distinguished among all effector types. Our results show that grasp computations do not proceed in a strictly hierarchically canonical fashion, highlighting the importance of the fine temporal resolution of EEG for a comprehensive understanding of human grasp control.SIGNIFICANCE STATEMENT A long-standing assumption of the grasp computations is that grasp representations progress from higher to lower level control in a regular, or canonical, fashion. Here, we combined EEG and multivariate pattern analysis to characterize the temporal dynamics of grasp representations while participants viewed objects and were subsequently cued to execute an unimanual or bimanual grasp. Interrogation of the temporal dynamics revealed that lower level effector representations emerged before intermediate levels of grasp representations, thereby suggesting a partially noncanonical progression from higher to lower and then to intermediate level grasp control.

Do motor plans affect sensorimotor state estimates during temporal decision-making with crossed vs. uncrossed hands? Failure to replicate the dynamic crossed-hand effect.

Hermosillo et al. (J Neurosci 31: 10019-10022, 2011) have suggested that action planning of hand movements impacts decisions about the temporal order judgments regarding vibrotactile stimulation of the hands. Specifically, these authors reported that the crossed-hand effect, a confusion about which hand is which when held in a crossed posture, gradually reverses some 320 ms before the arms begin to move from an uncrossed to a crossed posture or vice versa, such that the crossed-hand is reversed at the time of movement onset in anticipation of the movement's end position. However, to date, no other study has attempted to replicate this dynamic crossed-hand effect. Therefore, in the present study, we conducted four experiments to revisit the question whether preparing uncrossed-to-crossed or crossed-to-uncrossed movements affects the temporo-spatial perception of tactile stimulation of the hands. We used a temporal order judgement (TOJ) task at different time stages during action planning to test whether TOJs are more difficult with crossed than uncrossed hands ("static crossed-hand effect") and, crucially, whether planning to cross or uncross the hands shows the opposite pattern of difficulties ("dynamic crossed-hand effect"). As expected, our results confirmed the static crossed-hand effect. However, the dynamic crossed-hand effect could not be replicated. In addition, we observed that participants delayed their movements with late somatosensory stimulation from the TOJ task, even when the stimulations were meaningless, suggesting that the TOJ task resulted in cross-modal distractions. Whereas the current findings are not inconsistent with a contribution of motor signals to posture perception, they cast doubt on observations that motor signals impact state estimates well before movement onset.

Multivariate Analysis of Electrophysiological Signals Reveals the Temporal Properties of Visuomotor Computations for Precision Grips.

The frontoparietal networks underlying grasping movements have been extensively studied, especially using fMRI. Accordingly, whereas much is known about their cortical locus much less is known about the temporal dynamics of visuomotor transformations. Here, we show that multivariate EEG analysis allows for detailed insights into the time course of visual and visuomotor computations of precision grasps. Male and female human participants first previewed one of several objects and, upon its reappearance, reached to grasp it with the thumb and index finger along one of its two symmetry axes. Object shape classifiers reached transient accuracies of 70% at ∼105 ms, especially based on scalp sites over visual cortex, dropping to lower levels thereafter. Grasp orientation classifiers relied on a system of occipital-to-frontal electrodes. Their accuracy rose concurrently with shape classification but ramped up more gradually, and the slope of the classification curve predicted individual reaction times. Further, cross-temporal generalization revealed that dynamic shape representation involved early and late neural generators that reactivated one another. In contrast, grasp computations involved a chain of generators attaining a sustained state about 100 ms before movement onset. Our results reveal the progression of visual and visuomotor representations over the course of planning and executing grasp movements.SIGNIFICANCE STATEMENT Grasping an object requires the brain to perform visual-to-motor transformations of the object's properties. Although much of the neuroanatomic basis of visuomotor transformations has been uncovered, little is known about its time course. Here, we orthogonally manipulated object visual characteristics and grasp orientation, and used multivariate EEG analysis to reveal that visual and visuomotor computations follow similar time courses but display different properties and dynamics.

Mapping the anatomy of perceptual pseudoneglect. A multivariate approach.

Fundamental to the understanding of the functions of spatial cognition and attention is to clarify the underlying neural mechanisms. It is clear that relatively right-dominant activity in ventral and dorsal parieto-frontal cortex is associated with attentional reorienting, certain forms of mental imagery and spatial working memory for higher loads, while lesions mostly to right ventral areas cause spatial neglect with pathological attentional biases to the right side. In contrast, complementary leftward biases in healthy people, called pseudoneglect, have been associated with varying patterns of cortical activity. Notably, this inconsistency may be explained, at least in part, by the fact that pseudoneglect studies have often employed experimental paradigms that do not control sufficiently for cognitive processes unrelated to pseudoneglect. To address this issue, here we administered a carefully designed continuum of pseudoneglect and control tasks in healthy adults while using functional magnetic resonance imaging (fMRI). Data submitted to partial least square (PLS) imaging analysis yielded a significant latent variable that identified a right-dominant network of brain regions along the intra-occipital and -parietal sulci, frontal eye fields and right ventral cortex in association with perceptual pseudoneglect. Our results shed new light on the interplay of attentional and cognitive systems in pseudoneglect.

Multimodal evidence on shape and surface information in individual face processing.

The significance of shape and surface information for face perception is well established, yet their relative contribution to recognition and their neural underpinnings await clarification. Here, we employ image reconstruction to retrieve, assess and visualize such information using behavioral, electroencephalography and functional magnetic resonance imaging data. Our results indicate that both shape and surface information can be successfully recovered from each modality but that the latter is better recovered than the former, consistent with its key role for face representations. Further, shape and surface information exhibit similar spatiotemporal profiles, rely on the extraction of specific visual features, such as eye shape or skin tone, and reveal a systematic representational structure, albeit with more cross-modal consistency for shape than surface. More generally, the present work illustrates a novel approach to relating and comparing different modalities in terms of perceptual information content. Thus, our results help elucidate the representational basis of individual face recognition while, methodologically, they showcase the utility of image reconstruction and clarify its reliance on diagnostic visual information.

Shared right-hemispheric representations of sensorimotor goals in dynamic task environments.

Functional behaviour affords that we form goals to integrate sensory information about the world around us with suitable motor actions, such as when we plan to grab an object with a hand. However, much research has tested grasping in static scenarios where goals are pursued with repetitive movements, whereas dynamic contexts require goals to be pursued even when changes in the environment require a change in the actions to attain them. To study grasp goals in dynamic environments here, we employed a task where the goal remained the same but the execution of the movement changed; we primed participants to grasp objects either with their right or left hand, and occasionally they had to switch to grasping with both. Switch costs should be minimal if grasp goal representations were used continuously, for example, within the left dominant hemisphere. But remapped or re-computed goal representations should delay movements. We found that switching from right-hand grasping to bimanual grasping delayed reaction times but switching from left-hand grasping to bimanual grasping did not. Further, control experiments showed that the lateralized switch costs were not caused by asymmetric inhibition between hemispheres or switches between usual and unusual tasks. Our results show that the left hemisphere does not serve a general role of sensorimotor grasp goal representation. Instead, sensorimotor grasp goals appear to be represented at intermediate levels of abstraction, downstream from cognitive task representations, yet upstream from the control of the grasping effectors.

Working memory in action: inspecting the systematic and unsystematic errors of spatial memory across saccades.

Our ability to interact with the world depends on memory buffers that flexibly store and process information for short periods of time. Current working memory research, however, mainly uses tasks that avoid eye movements, whereas in daily life we need to remember information across saccades. Because saccades disrupt perception and attention, the brain might use special transsaccadic memory systems. Therefore, to compare working memory systems between and across saccades, the current study devised transsaccadic memory tasks that evaluated the influence of memory load on several kinds of systematic and unsystematic spatial errors, and tested whether these measures predicted performance in more established working memory paradigms. Experiment 1 used a line intersection task that had people integrate lines shown before and after saccades, and it administered a 2-back task. Experiments 2 and 3 asked people to point at one of several locations within a memory array flashed before an eye movement, and we tested change detection and 2-back performance. We found that unsystematic transsaccadic errors increased with memory load and were correlated with 2-back performance. Systematic errors produced similar results, although effects varied as a function of the geometric layout of the memory arrays. Surprisingly, transsaccadic errors did not predict change detection performance despite the latter being a widely accepted measure of working memory capacity. Our results suggest that working memory systems between and across saccades share, in part, similar neural resources. Nevertheless, our data highlight the importance of investigating working memory across saccades.

Parietal area BA7 integrates motor programs for reaching, grasping, and bimanual coordination.

Skillful interaction with the world requires that the brain uses a multitude of sensorimotor programs and subroutines, such as for reaching, grasping, and the coordination of the two body halves. However, it is unclear how these programs operate together. Networks for reaching, grasping, and bimanual coordination might converge in common brain areas. For example, Brodmann area 7 (BA7) is known to activate in disparate tasks involving the three types of movements separately. Here, we asked whether BA7 plays a key role in integrating coordinated reach-to-grasp movements for both arms together. To test this, we applied transcranial magnetic stimulation (TMS) to disrupt BA7 activity in the left and right hemispheres, while human participants performed a bimanual size-perturbation grasping task using the index and middle fingers of both hands to grasp a rectangular object whose orientation (and thus grasp-relevant width dimension) might or might not change. We found that TMS of the right BA7 during object perturbation disrupted the bimanual grasp and transport/coordination components, and TMS over the left BA7 disrupted unimanual grasps. These results show that right BA7 is causally involved in the integration of reach-to-grasp movements of the two arms. NEW & NOTEWORTHY: Our manuscript describes a role of human Brodmann area 7 (BA7) in the integration of multiple visuomotor programs for reaching, grasping, and bimanual coordination. Our results are the first to suggest that right BA7 is critically involved in the coordination of reach-to-grasp movements of the two arms. The results complement previous reports of right-hemisphere lateralization for bimanual grasps.

The time course of individual face recognition: A pattern analysis of ERP signals.

An extensive body of work documents the time course of neural face processing in the human visual cortex. However, the majority of this work has focused on specific temporal landmarks, such as N170 and N250 components, derived through univariate analyses of EEG data. Here, we take on a broader evaluation of ERP signals related to individual face recognition as we attempt to move beyond the leading theoretical and methodological framework through the application of pattern analysis to ERP data. Specifically, we investigate the spatiotemporal profile of identity recognition across variation in emotional expression. To this end, we apply pattern classification to ERP signals both in time, for any single electrode, and in space, across multiple electrodes. Our results confirm the significance of traditional ERP components in face processing. At the same time though, they support the idea that the temporal profile of face recognition is incompletely described by such components. First, we show that signals associated with different facial identities can be discriminated from each other outside the scope of these components, as early as 70ms following stimulus presentation. Next, electrodes associated with traditional ERP components as well as, critically, those not associated with such components are shown to contribute information to stimulus discriminability. And last, the levels of ERP-based pattern discrimination are found to correlate with recognition accuracy across subjects confirming the relevance of these methods for bridging brain and behavior data. Altogether, the current results shed new light on the fine-grained time course of neural face processing and showcase the value of novel methods for pattern analysis to investigating fundamental aspects of visual recognition.

Colour expectations during object perception are associated with early and late modulations of electrophysiological activity.

It is well known that visual expectation and attention modulate object perception. Yet, the mechanisms underlying these top-down influences are not completely understood. Event-related potentials (ERPs) indicate late contributions of expectations to object processing around the P2 or N2. This is true independent of whether people expect objects (vs. no objects) or specific shapes, hence when expectations pertain to complex visual features. However, object perception can also benefit from expecting colour information, which can facilitate figure/ground segregation. Studies on attention to colour show attention-sensitive modulations of the P1, but are limited to simple transient detection paradigms. The aim of the current study was to examine whether expecting simple features (colour information) during challenging object perception tasks produce early or late ERP modulations. We told participants to expect an object defined by predominantly black or white lines that were embedded in random arrays of distractor lines and then asked them to report the object's shape. Performance was better when colour expectations were met. ERPs revealed early and late phases of modulation. An early modulation at the P1/N1 transition arguably reflected earlier stages of object processing. Later modulations, at the P3, could be consistent with decisional processes. These results provide novel insights into feature-specific contributions of visual expectations to object perception.

The time course for visual extinction after a 'virtual' lesion of right posterior parietal cortex.

Our understanding of the attentional networks in the human brain largely relies on neuropsychological studies in patients with lesions to the posterior parietal cortex (PPC), particularly in the right hemisphere, that may cause severe disruptions of attentional functions. However, lesion studies only capture a point in time when the dysfunctions caused by the damage have triggered a chain of adaptive responses in the brain. To disentangle deficits and ensuing cortical plasticity, here we examined the time course for one's ability to detect objects in the visual periphery after an inhibitory continuous theta-burst stimulation (cTBS) protocol to the left or right PPC. Our results showed that cTBS of right PPC caused participants to be less sensitive to objects appearing on the left side as well as to objects appearing on both sides at the same time, consistent with an overall shift of attention to the right side of space. In addition, we found that participants missed more objects during bilateral presentations similar to patients with visual extinction. Critically, extinction evolved over time; that is, visual extinction for ipsilateral objects improved after 10 min whereas contralateral extinction peaked around 15-25 min after cTBS. Our findings suggest that lesions to the PPC impair competition between the two visual hemifields, resulting in contralateral extinction as a secondary response, arguably due to ensuing disruptions in interhemispheric balance.

The right anterior intraparietal sulcus is critical for bimanual grasping: a TMS study.

Grasping with 2 limbs in opposition to one another is older than the hand, yet the neural mechanisms for bimanual grasps remain unclear. Similar to unimanual grasping, bimanual grasping may require regions in the parietal cortex that use visual object-feature information to find matching stable grasp points on the object. The localization of matching points is computationally expensive, so it might make sense for the signals to converge in a single cortical area. To examine this, we use transcranial magnetic stimulation (TMS) to probe the contribution of cortical areas known to be associated with unimanual grasping, while participants performed bimanual grasps. We applied TMS to the anterior and caudal portion of the intra-parietal sulcus (aIPS and cIPS) in each hemisphere during a size-perturbation task using the index fingers of both hands to grasp an object whose orientation might or might not change. We found significant interaction effects between TMS and perturbation of the grasp-relevant object dimension that increased grip aperture only for the right aIPS. These results indicate that the aIPS is involved not only in unimanual, but also bimanual grasping, and the right aIPS is critically involved in bimanual grasps. This suggests that information from both hemispheres converges in the right hemisphere to achieve bimanual grasps.

Do head-on-trunk signals modulate disengagement of spatial attention?

Body schema is indispensable for sensorimotor control and learning, but whether it is associated with cognitive functions, such as allocation of spatial attention, remains unclear. Observations in patients with unilateral spatial neglect support this view, yet data from neurologically normal participants are inconsistent. Here, we investigated the influence of head-on-trunk positions (30° left or right, straight ahead) on disengagement of attention in healthy participants. Five experiments examined the effects of valid or invalid cues on spatial shifts of attention using the Posner paradigm. Experiment 1 used a forced-choice task. Participants quickly reported the location of a target that appeared left or right of the fixation point, preceded by a cue on the same (valid) or opposite side (invalid). Experiments 2, 3, and 4 also used valid and invalid cues but required participants to simply detect a target appearing on the left or right side. Experiment 5 used a speeded discrimination task, in which participants quickly reported the orientation of a Gabor. We observed expected influences of validity and stimulus onset asynchrony as well as inhibition of return; however, none of the experiments suggested that head-on-trunk position created or changed visual field advantages, contrary to earlier reports. Our results showed that the manipulations of the body schema did not modulate attentional processes in the healthy brain, unlike neuropsychological studies on neglect patients. Our findings suggest that spatial neglect reflects a state of the lesioned brain that is importantly different from that of the normally functioning brain.

Distractor removal amplifies spatial frequency-specific crossover of the attentional bias: a psychophysical and Monte Carlo simulation study.

Rarely noticed in daily life, attention may prefer the left side of space. Such attentional biases offer key insights into functions of spatial attention and visual awareness because they complement pathological biases in patients with spatial neglect who become largely unaware of the left side after right-brain damage. Yet there is little comprehensive understanding of these normal and pathological biases and how they relate to other attentional functions. Here we used a grating-scales task (GST) to test whether leftward biases and their spatial frequency-dependent crossover interact with attentional mechanisms of distractor removal. We asked healthy participants to make perceptual judgements to capture attentional biases in a high and a low spatial frequency condition (GST-HI and GST-LO), and we degraded stimuli with distracting pixel noise. We found that with noise, crossover grew, while biases remained positively correlated. Using Monte Carlo simulations, we probed the feasibility of three models and conclude that our data can only be explained by two, or more, biasing mechanisms, arguably interacting with each other through interhemispheric competition. Our study sets the stage for a new systematic approach to investigating the visuospatial mechanisms of the right hemisphere.

A right hemisphere dominance for bimanual grasps.

To find points on the surface of an object that ensure a stable grasp, it would be most effective to employ one area in one cortical hemisphere. But grasping the object with both hands requires control through both hemispheres. To better understand the control mechanisms underlying this "bimanual grasping", here we examined how the two hemispheres coordinate their control processes for bimanual grasping depending on visual field. We asked if bimanual grasping involves both visual fields equally or one more than the other. To test this, participants fixated either to the left or right of an object and then grasped or pushed it off a pedestal. We found that when participants grasped the object in the right visual field, maximum grip aperture (MGA) was larger and more variable, and participants were slower to react and to show MGA compared to when they grasped the object in the left visual field. In contrast, when participants pushed the object we observed no comparable visual field effects. These results suggest that grasping with both hands, specifically the computation of grasp points on the object, predominantly involves the right hemisphere. Our study provides new insights into the interactions of the two hemispheres for grasping.

Left visual field preference for a bimanual grasping task with ecologically valid object sizes.

Grasping using two forelimbs in opposition to one another is evolutionary older than the hand with an opposable thumb (Whishaw and Coles in Behav Brain Res 77:135-148, 1996); yet, the mechanisms for bimanual grasps remain unclear. Similar to unimanual grasping, the localization of matching stable grasp points on an object is computationally expensive and so it makes sense for the signals to converge in a single cortical hemisphere. Indeed, bimanual grasps are faster and more accurate in the left visual field, and are disrupted if there is transcranial stimulation of the right hemisphere (Le and Niemeier in Exp Brain Res 224:263-273, 2013; Le et al. in Cereb Cortex. doi: 10.1093/cercor/bht115, 2013). However, research so far has tested the right hemisphere dominance based on small objects only, which are usually grasped with one hand, whereas bimanual grasping is more commonly used for objects that are too big for a single hand. Because grasping large objects might involve different neural circuits than grasping small objects (Grol et al. in J Neurosci 27:11877-11887, 2007), here we tested whether a left visual field/right hemisphere dominance for bimanual grasping exists with large and thus more ecologically valid objects or whether the right hemisphere dominance is a function of object size. We asked participants to fixate to the left or right of an object and to grasp the object with the index and middle fingers of both hands. Consistent with previous observations, we found that for objects in the left visual field, the maximum grip apertures were scaled closer to the object width and were smaller and less variable, than for objects in the right visual field. Our results demonstrate that bimanual grasping is predominantly controlled by the right hemisphere, even in the context of grasping larger objects.

Working memory in action: Transsaccadic working memory deficits in the left visual field and after transcallosal remapping.

Every waking second, we make three saccadic eye movements that move our retinal images. Thus, to attain a coherent image of the world we need to remember visuo-spatial information across saccades. But transsaccadic working memory (tWM) remains poorly understood. Crucially, there has been a debate whether there are any differences in tWM for the left vs. right visual field and depending on saccade direction. However, previous studies have probed tWM with minimal loads whereas spatial differences might arise with higher loads. Here we employed a task that probed higher memory load for spatial information in the left and right visual field and with horizontal as well as vertical saccades. We captured several measures of precision and accuracy of performance that, when submitted to principal component analysis, produced two components. Component 1, mainly associated with precision, yielded greater error for the left than the right visual field. Component 2 was associated with performance accuracy and unexpectedly produced a disadvantage after rightward saccades. Both components showed that performance was worse when rightward or leftward saccades afforded a shift of memory representations between visual fields compared to remapping within the same field. Our study offers several novel findings. It is the first to show that tWM involves at least two components likely reflecting working memory capacity and strategic aspects of working memory, respectively. Reduced capacity for the left, rather than the right visual field is consistent with how the left and right visual fields are known to be represented in the two hemispheres. Remapping difficulties between visual fields is consistent with the limited information transfer across the corpus callosum. Finally, the impact of rightward saccades on working memory might be due to greater interference of the accompanying shifts of attention. Our results highlight the dynamic nature of transsaccadic working memory.