Same action, different meaning: neural substrates of action semantic meaning.

Voluntary actions are shaped by desired goals and internal intentions. Multiple factors, including the planning of subsequent actions and the expectation of sensory outcome, were shown to modulate kinetics and neural activity patterns associated with similar goal-directed actions. Notably, in many real-world tasks, actions can also vary across the semantic meaning they convey, although little is known about how semantic meaning modulates associated neurobehavioral measures. Here, we examined how behavioral and functional magnetic resonance imaging measures are modulated when subjects execute similar actions (button presses) for two different semantic meanings-to answer "yes" or "no" to a binary question. Our findings reveal that, when subjects answer using their right hand, the two semantic meanings are differentiated based on voxel patterns in the frontoparietal cortex and lateral-occipital complex bilaterally. When using their left hand, similar regions were found, albeit only with a more liberal threshold. Although subjects were faster to answer "yes" versus "no" when using their right hand, the neural differences cannot be explained by these kinetic differences. To the best of our knowledge, this is the first evidence showing that semantic meaning is embedded in the neural representation of actions, independent of alternative modulating factors such as kinetic and sensory features.

Better-than-chance classification for signal detection.

The estimated accuracy of a classifier is a random quantity with variability. A common practice in supervised machine learning, is thus to test if the estimated accuracy is significantly better than chance level. This method of signal detection is particularly popular in neuroimaging and genetics. We provide evidence that using a classifier's accuracy as a test statistic can be an underpowered strategy for finding differences between populations, compared to a bona fide statistical test. It is also computationally more demanding than a statistical test. Via simulation, we compare test statistics that are based on classification accuracy, to others based on multivariate test statistics. We find that the probability of detecting differences between two distributions is lower for accuracy-based statistics. We examine several candidate causes for the low power of accuracy-tests. These causes include: the discrete nature of the accuracy-test statistic, the type of signal accuracy-tests are designed to detect, their inefficient use of the data, and their suboptimal regularization. When the purpose of the analysis is the evaluation of a particular classifier, not signal detection, we suggest several improvements to increase power. In particular, to replace V-fold cross-validation with the Leave-One-Out Bootstrap.

Action-locked Neural Responses in Auditory Cortex to Self-generated Sounds.

Sensory perception is a product of interactions between the internal state of an organism and the physical attributes of a stimulus. It has been shown across the animal kingdom that perception and sensory-evoked physiological responses are modulated depending on whether or not the stimulus is the consequence of voluntary actions. These phenomena are often attributed to motor signals sent to relevant sensory regions that convey information about upcoming sensory consequences. However, the neurophysiological signature of action-locked modulations in sensory cortex, and their relationship with perception, is still unclear. In the current study, we recorded neurophysiological (using Magnetoencephalography) and behavioral responses from 16 healthy subjects performing an auditory detection task of faint tones. Tones were either generated by subjects' voluntary button presses or occurred predictably following a visual cue. By introducing a constant temporal delay between button press/cue and tone delivery, and applying source-level analysis, we decoupled action-locked and auditory-locked activity in auditory cortex. We show action-locked evoked-responses in auditory cortex following sound-triggering actions and preceding sound onset. Such evoked-responses were not found for button-presses that were not coupled with sounds, or sounds delivered following a predictive visual cue. Our results provide evidence for efferent signals in human auditory cortex that are locked to voluntary actions coupled with future auditory consequences.

Single-cell activity in human STG during perception of phonemes is organized according to manner of articulation.

One of the central tasks of the human auditory system is to extract sound features from incoming acoustic signals that are most critical for speech perception. Specifically, phonological features and phonemes are the building blocks for more complex linguistic entities, such as syllables, words and sentences. Previous ECoG and EEG studies showed that various regions in the superior temporal gyrus (STG) exhibit selective responses to specific phonological features. However, electrical activity recorded by ECoG or EEG grids reflects average responses of large neuronal populations and is therefore limited in providing insights into activity patterns of single neurons. Here, we recorded spiking activity from 45 units in the STG from six neurosurgical patients who performed a listening task with phoneme stimuli. Fourteen units showed significant responsiveness to the stimuli. Using a Naïve-Bayes model, we find that single-cell responses to phonemes are governed by manner-of-articulation features and are organized according to sonority with two main clusters for sonorants and obstruents. We further find that 'neural similarity' (i.e. the similarity of evoked spiking activity between pairs of phonemes) is comparable to the 'perceptual similarity' (i.e. to what extent two phonemes are judged as sounding similar) based on perceptual confusion, assessed behaviorally in healthy subjects. Thus, phonemes that were perceptually similar also had similar neural responses. Taken together, our findings indicate that manner-of-articulation is the dominant organization dimension of phoneme representations at the single-cell level, suggesting a remarkable consistency across levels of analyses, from the single neuron level to that of large neuronal populations and behavior.

Voluntary Actions Modulate Perception and Neural Representation of Action-Consequences in a Hand-Dependent Manner.

Evoked neural activity in sensory regions and perception of sensory stimuli are modulated when the stimuli are the consequence of voluntary movement, as opposed to an external source. It has been suggested that such modulations are due to motor commands that are sent to relevant sensory regions during voluntary movement. However, given the anatomical-functional laterality bias of the motor system, it is plausible that the pattern of such behavioral and neural modulations will also exhibit a similar bias, depending on the effector triggering the stimulus (e.g., right/left hand). Here, we examined this issue in the visual domain using behavioral and neural measures (fMRI). Healthy participants judged the relative brightness of identical visual stimuli that were either self-triggered (using right/left hand button presses), or triggered by the computer. Stimuli were presented either in the right or left visual field. Despite identical physical properties of the visual consequences, we found stronger perceptual modulations when the triggering hand was ipsi- (rather than contra-) lateral to the stimulated visual field. Additionally, fMRI responses in visual cortices differentiated between stimuli triggered by right/left hand. Our findings support a model in which voluntary actions induce sensory modulations that follow the anatomical-functional bias of the motor system.

A novel tool for time-locking study plans to results.

Often researchers wish to mark an objective line between study plans that were specified before data acquisition and decisions that were made following data exploration. Contrary to common perception, registering study plans to an online platform prior to data collection does not by itself provide such an objective distinction, even when the registration is time-stamped. Here, we adapt a method from the field of cryptography to allow encoding of study plans and predictions within random aspects of the data acquisition process. Doing so introduces a causal link between the preregistered content and objective attributes of the acquired data, such as the timing and location of brain activations. This guarantees that the preregistered plans and predictions are indeed specified prior to data collection. Our time-locking system does not depend on any external party and can be performed entirely in-lab. We provide code for easy implementation and a detailed example from the field of functional Magnetic Resonance Imaging (fMRI).

Behavioral and neural effects of congruency of visual feedback during short-term motor learning.

Visual feedback can facilitate or interfere with movement execution. Here, we describe behavioral and neural mechanisms by which the congruency of visual feedback during physical practice of a motor skill modulates subsequent performance gains. 18 healthy subjects learned to execute rapid sequences of right hand finger movements during fMRI scans either with or without visual feedback. Feedback consisted of a real-time, movement-based display of virtual hands that was either congruent (right virtual hand movement), or incongruent (left virtual hand movement yoked to the executing right hand). At the group level, right hand performance gains following training with congruent visual feedback were significantly higher relative to training without visual feedback. Conversely, performance gains following training with incongruent visual feedback were significantly lower. Interestingly, across individual subjects these opposite effects correlated. Activation in the Supplementary Motor Area (SMA) during training corresponded to individual differences in subsequent performance gains. Furthermore, functional coupling of SMA with visual cortices predicted individual differences in behavior. Our results demonstrate that some individuals are more sensitive than others to congruency of visual feedback during short-term motor learning and that neural activation in SMA correlates with such inter-individual differences.

What's in a pattern? Examining the type of signal multivariate analysis uncovers at the group level.

Multivoxel pattern analysis (MVPA) has gained enormous popularity in the neuroimaging community over the past few years. At the group level, most MVPA studies adopt an "information based" approach in which the sign of the effect of individual subjects is discarded and a non-directional summary statistic is carried over to the second level. This is in contrast to a directional "activation based" approach typical in univariate group level analysis, in which both signal magnitude and sign are taken into account. The transition from examining effects in one voxel at a time vs. several voxels (univariate vs. multivariate) has thus tacitly entailed a transition from directional to non-directional signal definition at the group level. While a directional group-level MVPA approach implies that individuals have similar multivariate spatial patterns of activity, in a non-directional approach each individual may have a distinct spatial pattern. Using an experimental dataset, we show that directional and non-directional group-level MVPA approaches uncover distinct brain regions with only partial overlap. We propose a method to quantify the degree of spatial similarity in activation patterns over subjects. Applied to an auditory task, we find higher values in auditory regions compared to control regions.

Enhanced auditory evoked activity to self-generated sounds is mediated by primary and supplementary motor cortices.

Accumulating evidence demonstrates that responses in auditory cortex to auditory consequences of self-generated actions are modified relative to the responses evoked by identical sounds generated by an external source. Such modifications have been suggested to occur through a corollary discharge sent from the motor system, although the exact neuroanatomical origin is unknown. Furthermore, since tactile input has also been shown to modify responses in auditory cortex, it is not even clear whether the source of such modifications is motor output or somatosensory feedback. We recorded functional magnetic resonance imaging (fMRI) data from healthy human subjects (n = 11) while manipulating the rate at which they performed sound-producing actions with their right hand. In addition, we manipulated the amount of tactile feedback to examine the relative roles of motor and somatosensory cortices in modifying evoked activity in auditory cortex (superior temporal gyrus). We found an enhanced fMRI signal in left auditory cortex during perception of self-generated sounds relative to passive listening to identical sounds. Moreover, the signal difference between active and passive conditions in left auditory cortex covaried with the rate of sound-producing actions and was invariant to the amount of tactile feedback. Together with functional connectivity analysis, our results suggest motor output from supplementary motor area and left primary motor cortex as the source of signal modification in auditory cortex during perception of self-generated sounds. Motor signals from these regions could represent a predictive signal of the expected auditory consequences of the performed action.

Differentiating intended sensory outcome from underlying motor actions in the human brain.

To achieve a certain sensory outcome, multiple actions can be executed. For example, unlocking a door might require clockwise or counterclockwise key turns depending on regional norms. Using fMRI in healthy human subjects, we examined the neural networks that dissociate intended sensory outcome from underlying motor actions. Subjects controlled a figure on a computer screen by performing pen traces on an MR-compatible digital tablet. Our design allowed us to dissociate intended sensory outcome (moving the figure in a certain direction) from the underlying motor action (horizontal/vertical pen traces). Using multivoxel pattern analysis and a whole-brain searchlight strategy, we found that activity patterns in left (contralateral) motor and parietal cortex and also right (ipsilateral) motor cortex significantly discriminated direction of pen traces regardless of intended direction of figure movement. Conversely, activity patterns in right superior parietal lobule and premotor cortex, and also left frontopolar cortex, significantly discriminated intended direction of figure movement regardless of underlying direction of hand movement. Together, these results highlight the role of ipsilateral motor cortex in coding movement directions and point to a network of brain regions involved in high order representation of intended sensory outcome that is dissociated from specific motor plans.

Preconscious prediction of a driver's decision using intracranial recordings.

While driving, we make numerous conscious decisions such as route and turn direction selection. Although drivers are held responsible, the neural processes that govern such decisions are not clear. We recorded intracranial EEG signals from six patients engaged in a computer-based driving simulator. Patients decided which way to turn (left/right) and subsequently reported the time of the decision. We show that power modulations of gamma band oscillations (30-100 Hz) preceding the reported time of decision (up to 5.5 sec) allow prediction of decision content with high accuracy (up to 82.4%) on a trial-by-trial basis, irrespective of subsequent motor output. Moreover, these modulations exhibited a spatiotemporal gradient, differentiating left/right decisions earliest in premotor cortices and later in more anterior and lateral regions. Our results suggest a preconscious role for the premotor cortices in early stages of decision-making, which permits foreseeing and perhaps modifying the content of real-life human choices before they are consciously made.

Decoding speech perception from single cell activity in humans.

Deciphering the content of continuous speech is a challenging task performed daily by the human brain. Here, we tested whether activity of single cells in auditory cortex could be used to support such a task. We recorded neural activity from auditory cortex of two neurosurgical patients while presented with a short video segment containing speech. Population spiking activity (~20 cells per patient) allowed detection of word onset and decoding the identity of perceived words with significantly high accuracy levels. Oscillation phase of local field potentials (8-12Hz) also allowed decoding word identity although with lower accuracy levels. Our results provide evidence that the spiking activity of a relatively small population of cells in human primary auditory cortex contains significant information for classification of words in ongoing speech. Given previous evidence for overlapping neural representation during speech perception and production, this may have implications for developing brain-machine interfaces for patients with deficits in speech production.

Memory enhancement and deep-brain stimulation of the entorhinal area.

BACKGROUND: The medial temporal structures, including the hippocampus and the entorhinal cortex, are critical for the ability to transform daily experience into lasting memories. We tested the hypothesis that deep-brain stimulation of the hippocampus or entorhinal cortex alters memory performance. METHODS: We implanted intracranial depth electrodes in seven subjects to identify seizure-onset zones for subsequent epilepsy surgery. The subjects completed a spatial learning task during which they learned destinations within virtual environments. During half the learning trials, focal electrical stimulation was given below the threshold that elicits an afterdischarge (i.e., a neuronal discharge that occurs after termination of the stimulus). RESULTS: Entorhinal stimulation applied while the subjects learned locations of landmarks enhanced their subsequent memory of these locations: the subjects reached these landmarks more quickly and by shorter routes, as compared with locations learned without stimulation. Entorhinal stimulation also resulted in a resetting of the phase of the theta rhythm, as shown on the hippocampal electroencephalogram. Direct hippocampal stimulation was not effective. In this small series, no adverse events associated with the procedure were observed. CONCLUSIONS: Stimulation of the entorhinal region enhanced memory of spatial information when applied during learning. (Funded by the National Institutes of Health and the Dana Foundation.).

High or low expectations: Expected intensity of action outcome is embedded in action kinetics.

Goal-directed actions are performed in order to attain certain sensory consequences in the world. However, expected attributes of these consequences can affect the kinetics of the action. In a set of three studies (n = 120), we examined how expected attributes of stimulus outcome (intensity) shape the kinetics of the triggering action (applied force), even when the action kinetic and attribute are independent. We show that during action execution (button presses), the expected intensity of sensory outcome affects the applied force of the stimulus-producing action in an inverse fashion. Thus, participants applied more force when the expected intensity of the outcome was low (vs. high intensity outcome). In the absence of expectations or when actions were performed in response to the sensory event, no intensity-dependent force modulations were found. Thus, expectations of stimulus intensity and causality play an important role in shaping action kinetics. Finally, we examined the relationship between kinetics and perception and found no influence of applied force level on perceptual detection of low intensity (near-threshold) outcome stimuli, suggesting no causal link between the two. Taken together, our results demonstrate that action kinetics are embedded with high-level context such as the expectation of consequence intensity and the causal relationship with environmental cues.

Human intracranial recordings and cognitive neuroscience.

The ultimate goal of neuroscience research is to understand the operating mechanism of the human brain and to exploit this understanding to devise methods for repair when it malfunctions. A key feature of this operating mechanism is electrical activity of single brain cells and cell assemblies. For obvious ethical reasons, scientists rely mostly on animal research in the study of such signals. Research in humans is often limited to electrical signals that can be recorded at the scalp or to surrogates of electrical activity, namely magnetic source imaging and measures of regional blood flow and metabolism. Invasive brain recordings performed in patients during various clinical procedures provide a unique opportunity to record high-resolution signals in vivo from the human brain-data that are otherwise unavailable. Of special value are the rare opportunities to record in awake humans the activity of single brain cells and small cellular assemblies. These recordings provide a unique view on aspects of human cognition that are impossible to study in animals, including language, imagery, episodic memory, volition, and even consciousness. In the current review we discuss the unique contribution of invasive recordings from patients to the field of cognitive neuroscience.

A neural substrate in the human hippocampus for linking successive events.

Memory formation requires the placement of experienced events in the same order in which they appeared. A large body of evidence from human studies indicates that structures in the medial temporal lobe are critically involved in forming and maintaining such memories, and complementing evidence from lesion and electrophysiological work in animals support these findings. However, it remains unclear how single cells and networks of cells can signal this temporal relationship between events. Here we used recordings from single cells in the human brain obtained while subjects viewed repeated presentations of cinematic episodes. We found that neuronal activity in successive time segments became gradually correlated, and, as a result, activity in a given time window became a faithful predictor of the activity to follow. This correlation emerged rapidly, within two to three presentations of an episode and exceeded both context-independent and pure stimulus-driven correlations. The correlation was specific for hippocampal neurons, did not occur in the amygdala and anterior cingulate cortex, and was found for single cells, cell pairs, and triplets of cells, supporting the notion that cell assemblies code for the temporal relationships between sensory events. Importantly, this neuronal measure of temporal binding successfully predicted subjects' ability to recall and verbally report the viewed episodes later. Our findings suggest a neuronal substrate for the formation of memory of the temporal order of events.

Temporal hierarchy of observed goal-directed actions.

During social interactions, we continuously integrate current and previous information over varying timescales to infer other people's action intentions. Motor cognition theories argue for a hierarchical organization of goal-directed actions based on temporal scales. Accordingly, transient motor primitives are represented at lower levels of the hierarchy, a combination of primitives building motor sequences at subordinate levels, and more stable overarching action goals at superordinate levels. A neural topography of hierarchal timescales for information accumulation was previously shown in the visual and auditory domains. However, whether such a temporal hierarchy can also account for observed goal-directed action representations in motor pathways remains to be determined. Thus, the current study examined the neural architecture underlying the processing of observed goal-directed actions using inter-subject correlation (ISC) of fMRI activity. Observers (n = 24) viewed sequential hand movements presented in their intact order or piecewise scrambled at three timescales pertaining to goal-directed action evolution (Primitives: ± 1.5 s, Sub-Goals: ± 4 s, and High-Goals: ± 10 s). The results revealed differential intrinsic temporal capacities for integrating goal-directed action information across brain areas engaged in action observation. Longer timescales (> ± 10 s) were found in the posterior parietal and dorsal premotor compared to the ventral premotor (± 4 s) and anterior parietal (± 1.5 s) cortex. Moreover, our results revealed a hemispheric bias with more extended timescales in the right MT+, primary somatosensory, and early visual cortices compared to their homotopic regions in the left hemisphere. Our findings corroborate a hierarchical neural mapping of observed actions based on temporal scales of goals and provide further support for a ubiquitous time-dependent neural organization of information processing across multiple modalities.

The power of multivariate approach in identifying EEG correlates of interlimb coupling.

Interlimb coupling refers to the interaction between movements of one limb and movements of other limbs. Understanding mechanisms underlying this effect is important to real life because it reflects the level of interdependence between the limbs that plays a role in daily activities including tool use, cooking, or playing musical instruments. Interlimb coupling involves multiple brain regions working together, including coordination of neural activity in sensory and motor regions across the two hemispheres. Traditional neuroscience research took a univariate approach to identify neural features that correspond to behavioural coupling measures. Yet, this approach reduces the complexity of the neural activity during interlimb tasks to one value. In this brief research report, we argue that identifying neural correlates of interlimb coupling would benefit from a multivariate approach in which full patterns from multiple sources are used to predict behavioural coupling. We demonstrate the feasibility of this approach in an exploratory EEG study where participants (n = 10) completed 240 trials of a well-established drawing paradigm that involves interlimb coupling. Using artificial neural network (ANN), we show that multivariate representation of the EEG signal significantly captures the interlimb coupling during bimanual drawing whereas univariate analyses failed to identify such correlates. Our findings demonstrate that analysing distributed patterns of multiple EEG channels is more sensitive than single-value techniques in uncovering subtle differences between multiple neural signals. Using such techniques can improve identification of neural correlates of complex motor behaviours.

Playing with your ears: Audio-motor skill learning is sensitive to the lateral relationship between trained hand and ear.

A salient feature of motor and sensory circuits in the brain is their contralateral hemispheric bias-a feature that might play a role in integration and learning of sensorimotor skills. In the current behavioral study, we examined whether the lateral configuration between sound-producing hand and feedback-receiving ear affects performance and learning of an audio-motor skill. Right-handed participants (n = 117) trained to play a piano sequence using their right or left hand while auditory feedback was presented monaurally, either to the right or left ear. Participants receiving auditory feedback to the contralateral ear during training performed better than participants receiving ipsilateral feedback (with respect to the training hand). Furthermore, in the Left-Hand training groups, the contralateral training advantage persisted in a generalization task. Our results demonstrate that audio-motor learning is sensitive to the lateral configuration between motor and sensory circuits and suggest that integration of neural activity across hemispheres facilitates such learning.

Real-time processes in the development of action planning.

Across species and ages, planning multi-step actions is a hallmark of intelligence and critical for survival. Traditionally, researchers adopt a "top-down" approach to action planning by focusing on the ability to create an internal representation of the world that guides the next step in a multi-step action. However, a top-down approach does not inform on underlying mechanisms, so researchers can only speculate about how and why improvements in planning occur. The current study takes a "bottom-up" approach by testing developmental changes in the real-time, moment-to-moment interplay among perceptual, neural, and motor components of action planning using simultaneous video, motion-tracking, head-mounted eye tracking, and electroencephalography (EEG). Preschoolers (n = 32) and adults (n = 22) grasped a hammer with their dominant hand to pound a peg when the hammer handle pointed in different directions. When the handle pointed toward their non-dominant hand, younger children ("nonadaptive planners") used a habitual overhand grip that interfered with wielding the hammer, whereas adults and older children ("adaptive planners") used an adaptive underhand grip. Adaptive and nonadaptive children differed in when and where they directed their gaze to obtain visual information, neural activation of the motor system before reaching, and straightness of their reach trajectories. Nonadaptive children immediately used a habitual overhand grip before gathering visual information, leaving insufficient time to form a plan before acting. Our novel bottom-up approach transcends mere speculation by providing converging evidence that the development of action planning depends on a real-time "tug of war" between habits and information gathering and processing.