The neural representations of valence transformation in indole processing.

Indole is often associated with a sweet and floral odor typical of jasmine flowers at low concentrations and an unpleasant, animal-like odor at high concentrations. However, the mechanism whereby the brain processes this opposite valence of indole is not fully understood yet. In this study, we aimed to investigate the neural mechanisms underlying indole valence encoding in conversion and nonconversion groups using the smelling task to arouse pleasantness. For this purpose, 12 conversion individuals and 15 nonconversion individuals participated in an event-related functional magnetic resonance imaging paradigm with low (low-indole) and high (high-indole) indole concentrations in which valence was manipulated independent of intensity. The results of this experiment showed that neural activity in the right amygdala, orbitofrontal cortex and insula was associated with valence independent of intensity. Furthermore, activation in the right orbitofrontal cortex in response to low-indole was positively associated with subjective pleasantness ratings. Conversely, activation in the right insula and amygdala in response to low-indole was positively correlated with anticipatory hedonic traits. Interestingly, while amygdala activation in response to high-indole also showed a positive correlation with these hedonic traits, such correlation was observed solely with right insula activation in response to high-indole. Additionally, activation in the right amygdala in response to low-indole was positively correlated with consummatory pleasure and hedonic traits. Regarding olfactory function, only activation in the right orbitofrontal cortex in response to high-indole was positively correlated with olfactory identification, whereas activation in the insula in response to low-indole was negatively correlated with the level of self-reported olfactory dysfunction. Based on these findings, valence transformation of indole processing in the right orbitofrontal cortex, insula, and amygdala may be associated with individual hedonic traits and perceptual differences.

Multiple and Dissociable Effects of Sensory History on Working-Memory Performance.

Behavioral reports of sensory information are biased by stimulus history. The nature and direction of such serial-dependence biases can differ between experimental settings; both attractive and repulsive biases toward previous stimuli have been observed. How and when these biases arise in the human brain remains largely unexplored. They could occur either via a change in sensory processing itself and/or during postperceptual processes such as maintenance or decision-making. To address this, we tested 20 participants (11 female) and analyzed behavioral and magnetoencephalographic (MEG) data from a working-memory task in which participants were sequentially presented with two randomly oriented gratings, one of which was cued for recall at the end of the trial. Behavioral responses showed evidence for two distinct biases: (1) a within-trial repulsive bias away from the previously encoded orientation on the same trial, and (2) a between-trial attractive bias toward the task-relevant orientation on the previous trial. Multivariate classification of stimulus orientation revealed that neural representations during stimulus encoding were biased away from the previous grating orientation, regardless of whether we considered the within-trial or between-trial prior orientation, despite opposite effects on behavior. These results suggest that repulsive biases occur at the level of sensory processing and can be overridden at postperceptual stages to result in attractive biases in behavior.SIGNIFICANCE STATEMENT Recent experience biases behavioral reports of sensory information, possibly capitalizing on the temporal regularity in our environment. It is still unclear at what stage of stimulus processing such serial biases arise. Here, we recorded behavior and neurophysiological [magnetoencephalographic (MEG)] data to test whether neural activity patterns during early sensory processing show the same biases seen in participants' reports. In a working-memory task that produced multiple biases in behavior, responses were biased toward previous targets, but away from more recent stimuli. Neural activity patterns were uniformly biased away from all previously relevant items. Our results contradict proposals that all serial biases arise at an early sensory processing stage. Instead, neural activity exhibited mostly adaptation-like responses to recent stimuli.

Drifting assemblies for persistent memory: Neuron transitions and unsupervised compensation.

Change is ubiquitous in living beings. In particular, the connectome and neural representations can change. Nevertheless, behaviors and memories often persist over long times. In a standard model, associative memories are represented by assemblies of strongly interconnected neurons. For faithful storage these assemblies are assumed to consist of the same neurons over time. Here we propose a contrasting memory model with complete temporal remodeling of assemblies, based on experimentally observed changes of synapses and neural representations. The assemblies drift freely as noisy autonomous network activity and spontaneous synaptic turnover induce neuron exchange. The gradual exchange allows activity-dependent and homeostatic plasticity to conserve the representational structure and keep inputs, outputs, and assemblies consistent. This leads to persistent memory. Our findings explain recent experimental results on temporal evolution of fear memory representations and suggest that memory systems need to be understood in their completeness as individual parts may constantly change.

The Neural Representations of Emotional Experiences Are More Similar Than Those of Neutral Experiences.

Stimuli that evoke the same feelings can nevertheless look different and have different semantic meanings. Although we know much about the neural representation of emotion, the neural underpinnings of emotional similarity are unknown. One possibility is that the same brain regions represent similarity between emotional and neutral stimuli, perhaps with different strengths. Alternatively, emotional similarity could be coded in separate regions, possibly those sensitive to emotional valence and arousal. In behavior, the extent to which people consider similarity along emotional dimensions when they evaluate the overall similarity between stimuli has never been investigated. Although the emotional features of stimuli may dominate explicit ratings of similarity, it is also possible that people neglect emotional dimensions as irrelevant to that judgment. We contrasted these hypotheses in (male and female) healthy controls using two measures of similarity and two picture databases of complex negative and neutral scenes, the second of which provided exquisite control over semantic and visual attributes. The similarity between emotional stimuli was greater than between neutral stimuli in the inferior temporal cortex, the fusiform face area, and the precuneus. Additionally, only the similarity between emotional stimuli was significantly represented in early visual cortex, anterior insula and dorsal anterior cingulate cortex. Intriguingly, despite the stronger neural similarity between emotional stimuli, the same participants did not rate them as more similar to each other than neutral stimuli. These results contribute to our understanding of how emotion is represented within a general conceptual workspace and of the overgeneralization bias in anxiety disorders.SIGNIFICANCE STATEMENT We tested differences in similarity between emotional and neutral scenes. Arousal and negative valence did not increase similarity ratings. When conditions were equated on semantic similarity, participants rated emotional stimuli as similar to each other as neutral ones. Despite this equivalence, the similarity among the neural representations of emotional compared with neutral stimuli was higher in regions, which also expressed similarity between neutral stimuli and in unique regions. We report a striking difference between behavioral and neural similarity; strong neural similarity between emotional pictures did not influence similarity judgements in the same participants in the behavioral rating task after the scan. These findings may have an impact on research about the neural representations of emotional categories and the overgeneralization bias in anxiety disorders.

Testing the generalization of neural representations.

Multivariate analysis methods are widely used in neuroscience to investigate the presence and structure of neural representations. Representational similarities across time or contexts are often investigated using pattern generalization, e.g. by training and testing multivariate decoders in different contexts, or by comparable pattern-based encoding methods. It is however unclear what conclusions can be validly drawn on the underlying neural representations when significant pattern generalization is found in mass signals such as LFP, EEG, MEG, or fMRI. Using simulations, we show how signal mixing and dependencies between measurements can drive significant pattern generalization even though the true underlying representations are orthogonal. We suggest that, using an accurate estimate of the expected pattern generalization given identical representations, it is nonetheless possible to test meaningful hypotheses about the generalization of neural representations. We offer such an estimate of the expected magnitude of pattern generalization and demonstrate how this measure can be used to assess the similarity and differences of neural representations across time and contexts.

Representational Neural Mapping of Dexterous Grasping Before Lifting in Humans.

The ability of humans to reach and grasp objects in their environment has been the mainstay paradigm for characterizing the neural circuitry driving object-centric actions. Although much is known about hand shaping, a persistent question is how the brain orchestrates and integrates the grasp with lift forces of the fingers in a coordinated manner. The objective of the current study was to investigate how the brain represents grasp configuration and lift force during a dexterous object-centric action in a large sample of male and female human subjects. BOLD activity was measured as subjects used a precision-grasp to lift an object with a center of mass (CoM) on the left or right with the goal of minimizing tilting the object. The extent to which grasp configuration and lift force varied between left and right CoM conditions was manipulated by grasping the object collinearly (requiring a non-collinear force distribution) or non-collinearly (requiring more symmetrical forces). Bayesian variational representational similarity analyses on fMRI data assessed the evidence that a set of cortical and cerebellar regions were sensitive to grasp configuration or lift force differences between CoM conditions at differing time points during a grasp to lift action. In doing so, we reveal strong evidence that grasping and lift force are not represented by spatially separate functionally specialized regions, but by the same regions at differing time points. The coordinated grasp to lift effort is shown to be under dorsolateral (PMv and AIP) more than dorsomedial control, and under SPL7, somatosensory PSC, ventral LOC and cerebellar control.SIGNIFICANCE STATEMENT Clumsy disasters such as spilling, dropping, and crushing during our daily interactions with objects are a rarity rather than the norm. These disasters are avoided in part as a result of our orchestrated anticipatory efforts to integrate and coordinate grasping and lifting of object interactions, all before the lift of an object even commences. How the brain orchestrates this integration process has been largely neglected by historical approaches independently and solely focusing on reaching and grasping and the neural principles that guide them. Here, we test the extent to which grasping and lifting are represented in a spatially or temporally distinct manner and identified strong evidence for the consecutive emergence of sensitivity to grasping, then lifting within the same region.

Representational dynamics preceding conscious access.

Our senses are continuously bombarded with more information than our brain can process up to the level of awareness. The present study aimed to enhance understanding on how attentional selection shapes conscious access under conditions of rapidly changing input. Using an attention task, EEG, and multivariate decoding of individual target- and distractor-defining features, we specifically examined dynamic changes in the representation of targets and distractors as a function of conscious access and the task-relevance (target or distractor) of the preceding item in the RSVP stream. At the behavioral level, replicating previous work and suggestive of a flexible gating mechanism, we found a significant impairment in conscious access to targets (T2) that were preceded by a target (T1) followed by one or two distractors (i.e., the attentional blink), but striking facilitation of conscious access to targets shown directly after another target (i.e., lag-1 sparing and blink reversal). At the neural level, conscious access to T2 was associated with enhanced early- and late-stage T1 representations and enhanced late-stage D1 representations, and interestingly, could be predicted based on the pattern of EEG activation well before T1 was presented. Yet, across task conditions, we did not find convincing evidence for the notion that conscious access is affected by rapid top-down selection-related modulations of the strength of early sensory representations induced by the preceding visual event. These results cannot easily be explained by existing accounts of how attentional selection shapes conscious access under rapidly changing input conditions, and have important implications for theories of the attentional blink and consciousness more generally.

Neural Representations of Procedural Knowledge.

Although declarative concepts (e.g., apple) have been shown to be identifiable from their functional MRI (fMRI) signatures, the correspondence has yet to be established for executing a complex procedure such as tying a knot. In this study, 7 participants were trained to tie seven knots. Their neural representations of these seven procedures were assessed with fMRI as they imagined tying each knot. A subset of the trained participants physically tied each knot in a later fMRI session. Findings demonstrated that procedural knowledge of tying a particular knot can be reliably identified from its fMRI signature, and such procedural signatures were found here in frontal, parietal, motor, and cerebellar regions. In addition, a classifier trained on mental tying signatures was able to reliably identify when participants were planning to tie knots before they physically tied them, which suggests that the mental-tying and physical-tying procedural signatures are similar. These findings indicate that fMRI activation patterns can illuminate the representation and organization of procedural knowledge.

Estimating the functional dimensionality of neural representations.

Recent advances in multivariate fMRI analysis stress the importance of information inherent to voxel patterns. Key to interpreting these patterns is estimating the underlying dimensionality of neural representations. Dimensions may correspond to psychological dimensions, such as length and orientation, or involve other coding schemes. Unfortunately, the noise structure of fMRI data inflates dimensionality estimates and thus makes it difficult to assess the true underlying dimensionality of a pattern. To address this challenge, we developed a novel approach to identify brain regions that carry reliable task-modulated signal and to derive an estimate of the signal's functional dimensionality. We combined singular value decomposition with cross-validation to find the best low-dimensional projection of a pattern of voxel-responses at a single-subject level. Goodness of the low-dimensional reconstruction is measured as Pearson correlation with a test set, which allows to test for significance of the low-dimensional reconstruction across participants. Using hierarchical Bayesian modeling, we derive the best estimate and associated uncertainty of underlying dimensionality across participants. We validated our method on simulated data of varying underlying dimensionality, showing that recovered dimensionalities match closely true dimensionalities. We then applied our method to three published fMRI data sets all involving processing of visual stimuli. The results highlight three possible applications of estimating the functional dimensionality of neural data. Firstly, it can aid evaluation of model-based analyses by revealing which areas express reliable, task-modulated signal that could be missed by specific models. Secondly, it can reveal functional differences across brain regions. Thirdly, knowing the functional dimensionality allows assessing task-related differences in the complexity of neural patterns.

Within-Category Decoding of Information in Different Attentional States in Short-Term Memory.

A long-standing assumption of cognitive neuroscience has been that working memory (WM) is accomplished by sustained, elevated neural activity. More recently, theories of WM have expanded this view by describing different attentional states in WM with differing activation levels. Several studies have used multivariate pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) data to study neural activity corresponding to these WM states. Intriguingly, no evidence was found for active neural representations for information held in WM outside the focus of attention ("unattended memory items," UMIs), suggesting that only attended memory items (AMIs) are accompanied by an active trace. However, these results depended on category-level decoding, which lacks sensitivity to neural representations of individual items. Therefore, we employed a WM task in which subjects remembered the directions of motion of two dot arrays, with a retrocue indicating which was relevant for an imminent memory probe (the AMI). This design allowed MVPA decoding of delay-period fMRI signal at the stimulus-item level, affording a more sensitive test of the neural representation of UMIs. Whereas evidence for the AMI was reliably high, evidence for the UMI dropped to baseline, consistent with the notion that different WM attentional states may have qualitatively different mechanisms of retention.

Neural representations of the concepts in simple sentences: Concept activation prediction and context effects.

Although it has been possible to identify individual concepts from a concept's brain activation pattern, there have been significant obstacles to identifying a proposition from its fMRI signature. Here we demonstrate the ability to decode individual prototype sentences from readers' brain activation patterns, by using theory-driven regions of interest and semantic properties. It is possible to predict the fMRI brain activation patterns evoked by propositions and words which are entirely new to the model with reliably above-chance rank accuracy. The two core components implemented in the model that reflect the theory were the choice of intermediate semantic features and the brain regions associated with the neurosemantic dimensions. This approach also predicts the neural representation of object nouns across participants, studies, and sentence contexts. Moreover, we find that the neural representation of an agent-verb-object proto-sentence is more accurately characterized by the neural signatures of its components as they occur in a similar context than by the neural signatures of these components as they occur in isolation.

Predicting the brain activation pattern associated with the propositional content of a sentence: Modeling neural representations of events and states.

Even though much has recently been learned about the neural representation of individual concepts and categories, neuroimaging research is only beginning to reveal how more complex thoughts, such as event and state descriptions, are neurally represented. We present a predictive computational theory of the neural representations of individual events and states as they are described in 240 sentences. Regression models were trained to determine the mapping between 42 neurally plausible semantic features (NPSFs) and thematic roles of the concepts of a proposition and the fMRI activation patterns of various cortical regions that process different types of information. Given a semantic characterization of the content of a sentence that is new to the model, the model can reliably predict the resulting neural signature, or, given an observed neural signature of a new sentence, the model can predict its semantic content. The models were also reliably generalizable across participants. This computational model provides an account of the brain representation of a complex yet fundamental unit of thought, namely, the conceptual content of a proposition. In addition to characterizing a sentence representation at the level of the semantic and thematic features of its component concepts, factor analysis was used to develop a higher level characterization of a sentence, specifying the general type of event representation that the sentence evokes (e.g., a social interaction versus a change of physical state) and the voxel locations most strongly associated with each of the factors. Hum Brain Mapp 38:4865-4881, 2017. © 2017 Wiley Periodicals, Inc.

Mapping the unconscious maintenance of a lost first language.

Optimal periods during early development facilitate the formation of perceptual representations, laying the framework for future learning. A crucial question is whether such early representations are maintained in the brain over time without continued input. Using functional MRI, we show that internationally adopted (IA) children from China, exposed exclusively to French since adoption (mean age of adoption, 12.8 mo), maintained neural representations of their birth language despite functionally losing that language and having no conscious recollection of it. Their neural patterns during a Chinese lexical tone discrimination task matched those observed in Chinese/French bilinguals who have had continual exposure to Chinese since birth and differed from monolingual French speakers who had never been exposed to Chinese. They processed lexical tone as linguistically relevant, despite having no Chinese exposure for 12.6 y, on average, and no conscious recollection of that language. More specifically, IA participants recruited left superior temporal gyrus/planum temporale, matching the pattern observed in Chinese/French bilinguals. In contrast, French speakers who had never been exposed to Chinese did not recruit this region and instead activated right superior temporal gyrus. We show that neural representations are not overwritten and suggest a special status for language input obtained during the first year of development.

Contour junctions underlie neural representations of scene categories in high-level human visual cortex.

Humans efficiently grasp complex visual environments, making highly consistent judgments of entry-level category despite their high variability in visual appearance. How does the human brain arrive at the invariant neural representations underlying categorization of real-world environments? We here show that the neural representation of visual environments in scene-selective human visual cortex relies on statistics of contour junctions, which provide cues for the three-dimensional arrangement of surfaces in a scene. We manipulated line drawings of real-world environments such that statistics of contour orientations or junctions were disrupted. Manipulated and intact line drawings were presented to participants in an fMRI experiment. Scene categories were decoded from neural activity patterns in the parahippocampal place area (PPA), the occipital place area (OPA) and other visual brain regions. Disruption of junctions but not orientations led to a drastic decrease in decoding accuracy in the PPA and OPA, indicating the reliance of these areas on intact junction statistics. Accuracy of decoding from early visual cortex, on the other hand, was unaffected by either image manipulation. We further show that the correlation of error patterns between decoding from the scene-selective brain areas and behavioral experiments is contingent on intact contour junctions. Finally, a searchlight analysis exposes the reliance of visually active brain regions on different sets of contour properties. Statistics of contour length and curvature dominate neural representations of scene categories in early visual areas and contour junctions in high-level scene-selective brain regions.

Neural substrates of sensorimotor processes: letter writing and letter perception.

Writing and perceiving letters are thought to share similar neural substrates; however, what constitutes a neural representation for letters is currently debated. One hypothesis is that letter representation develops from sensorimotor experience resulting in an integrated set of modality-specific regions, whereas an alternative account suggests that letter representations may be abstract, independent of modality. Studies reviewed suggest that letter representation consists of a network of modality-responsive brain regions that may include an abstract component.

Neural Representations of Physics Concepts.

We used functional MRI (fMRI) to assess neural representations of physics concepts (momentum, energy, etc.) in juniors, seniors, and graduate students majoring in physics or engineering. Our goal was to identify the underlying neural dimensions of these representations. Using factor analysis to reduce the number of dimensions of activation, we obtained four physics-related factors that were mapped to sets of voxels. The four factors were interpretable as causal motion visualization, periodicity, algebraic form, and energy flow. The individual concepts were identifiable from their fMRI signatures with a mean rank accuracy of .75 using a machine-learning (multivoxel) classifier. Furthermore, there was commonality in participants' neural representation of physics; a classifier trained on data from all but one participant identified the concepts in the left-out participant (mean accuracy = .71 across all nine participant samples). The findings indicate that abstract scientific concepts acquired in an educational setting evoke activation patterns that are identifiable and common, indicating that science education builds abstract knowledge using inherent, repurposed brain systems.

SDF4CHD: Generative modeling of cardiac anatomies with congenital heart defects.

Congenital heart disease (CHD) encompasses a spectrum of cardiovascular structural abnormalities, often requiring customized treatment plans for individual patients. Computational modeling and analysis of these unique cardiac anatomies can improve diagnosis and treatment planning and may ultimately lead to improved outcomes. Deep learning (DL) methods have demonstrated the potential to enable efficient treatment planning by automating cardiac segmentation and mesh construction for patients with normal cardiac anatomies. However, CHDs are often rare, making it challenging to acquire sufficiently large patient cohorts for training such DL models. Generative modeling of cardiac anatomies has the potential to fill this gap via the generation of virtual cohorts; however, prior approaches were largely designed for normal anatomies and cannot readily capture the significant topological variations seen in CHD patients. Therefore, we propose a type- and shape-disentangled generative approach suitable to capture the wide spectrum of cardiac anatomies observed in different CHD types and synthesize differently shaped cardiac anatomies that preserve the unique topology for specific CHD types. Our DL approach represents generic whole heart anatomies with CHD type-specific abnormalities implicitly using signed distance fields (SDF) based on CHD type diagnosis. To capture the shape-specific variations, we then learn invertible deformations to morph the learned CHD type-specific anatomies and reconstruct patient-specific shapes. After training with a dataset containing the cardiac anatomies of 67 patients spanning 6 CHD types and 14 combinations of CHD types, our method successfully captures divergent anatomical variations across different types and the meaningful intermediate CHD states across the spectrum of related CHD diagnoses. Additionally, our method demonstrates superior performance in CHD anatomy generation in terms of CHD-type correctness and shape plausibility. It also exhibits comparable generalization performance when reconstructing unseen cardiac anatomies. Moreover, our approach shows potential in augmenting image-segmentation pairs for rarer CHD types to significantly enhance cardiac segmentation accuracy for CHDs. Furthermore, it enables the generation of CHD cardiac meshes for computational simulation, facilitating a systematic examination of the impact of CHDs on cardiac functions.

Computational role of structure in neural activity and connectivity.

One major challenge of neuroscience is identifying structure in seemingly disorganized neural activity. Different types of structure have different computational implications that can help neuroscientists understand the functional role of a particular brain area. Here, we outline a unified approach to characterize structure by inspecting the representational geometry and the modularity properties of the recorded activity and show that a similar approach can also reveal structure in connectivity. We start by setting up a general framework for determining geometry and modularity in activity and connectivity and relating these properties with computations performed by the network. We then use this framework to review the types of structure found in recent studies of model networks performing three classes of computations.

Representational formats of human memory traces.

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Orthogonal neural representations support perceptual judgements of natural stimuli.

In natural behavior, observers must separate relevant information from a barrage of irrelevant information. Many studies have investigated the neural underpinnings of this ability using artificial stimuli presented on simple backgrounds. Natural viewing, however, carries a set of challenges that are inaccessible using artificial stimuli, including neural responses to background objects that are task-irrelevant. An emerging body of evidence suggests that the visual abilities of humans and animals can be modeled through the linear decoding of task-relevant information from visual cortex. This idea suggests the hypothesis that irrelevant features of a natural scene should impair performance on a visual task only if their neural representations intrude on the linear readout of the task relevant feature, as would occur if the representations of task-relevant and irrelevant features are not orthogonal in the underlying neural population. We tested this hypothesis using human psychophysics and monkey neurophysiology, in response to parametrically variable naturalistic stimuli. We demonstrate that 1) the neural representation of one feature (the position of a central object) in visual area V4 is orthogonal to those of several background features, 2) the ability of human observers to precisely judge object position was largely unaffected by task-irrelevant variation in those background features, and 3) many features of the object and the background are orthogonally represented by V4 neural responses. Our observations are consistent with the hypothesis that orthogonal neural representations can support stable perception of objects and features despite the tremendous richness of natural visual scenes.