Semantic fMRI neurofeedback of emotions: from basic principles to clinical applications.

During fMRI neurofeedback participants learn to self-regulate activity in relevant brain areas and networks based on ongoing feedback extracted from measured responses in those regions. This closed-loop approach has been successfully applied to reduce symptoms in mood disorders such as depression by showing participants a thermometer-like display indicating the strength of activity in emotion-related brain areas. The hitherto employed conventional neurofeedback is, however, 'blind' with respect to emotional content, i.e. patients instructed to engage in a specific positive emotion could drive the neurofeedback signal by engaging in a different (positive or negative) emotion. In this future perspective, we present a new form of neurofeedback that displays semantic information of emotions to the participant. Semantic information is extracted online using real-time representational similarity analysis of emotion-specific activity patterns. The extracted semantic information can be provided to participants in a two-dimensional semantic map depicting the current mental state as a point reflecting its distance to pre-measured emotional mental states (e.g. 'happy', 'content', 'sad', 'angry'). This new approach provides transparent feedback during self-regulation training, and it has the potential to enable more specific training effects for future therapeutic applications such as clinical interventions in mood disorders.This article is part of the theme issue 'Neurofeedback: new territories and neurocognitive mechanisms of endogenous neuromodulation'.

The topology and geometry of neural representations.

A central question for neuroscience is how to characterize brain representations of perceptual and cognitive content. An ideal characterization should distinguish different functional regions with robustness to noise and idiosyncrasies of individual brains that do not correspond to computational differences. Previous studies have characterized brain representations by their representational geometry, which is defined by the representational dissimilarity matrix (RDM), a summary statistic that abstracts from the roles of individual neurons (or responses channels) and characterizes the discriminability of stimuli. Here, we explore a further step of abstraction: from the geometry to the topology of brain representations. We propose topological representational similarity analysis, an extension of representational similarity analysis that uses a family of geotopological summary statistics that generalizes the RDM to characterize the topology while de-emphasizing the geometry. We evaluate this family of statistics in terms of the sensitivity and specificity for model selection using both simulations and functional MRI (fMRI) data. In the simulations, the ground truth is a data-generating layer representation in a neural network model and the models are the same and other layers in different model instances (trained from different random seeds). In fMRI, the ground truth is a visual area and the models are the same and other areas measured in different subjects. Results show that topology-sensitive characterizations of population codes are robust to noise and interindividual variability and maintain excellent sensitivity to the unique representational signatures of different neural network layers and brain regions.

The Neural Underpinnings of Aphantasia: A Case Study of Identical Twins.

Aphantasia is a condition characterized by reduced voluntary mental imagery. As this lack of mental imagery disrupts visual memory, understanding the nature of this condition can provide important insight into memory, perception, and imagery. Here, we leveraged the power of case studies to better characterize this condition by running a pair of identical twins, one with aphantasia and one without, through mental imagery tasks in an fMRI scanner. We identified objective, neural measures of aphantasia, finding less visual information in their memories which may be due to lower connectivity between frontoparietal and occipitotemporal lobes of the brain. However, despite this difference, we surprisingly found more visual information in the aphantasic twin's memory than anticipated, suggesting that aphantasia is a spectrum rather than a discrete condition.

Better Together: Integrating Multivariate with Univariate Methods, and MEG with EEG to Study Language Comprehension.

We used MEG and EEG to examine the effects of Plausibility (anomalous vs. plausible) and Animacy (animate vs. inanimate) on activity to incoming words during language comprehension. We conducted univariate event-related and multivariate spatial similarity analyses on both datasets. The univariate and multivariate results converged in their time course and sensitivity to Plausibility. However, only the spatial similarity analyses detected effects of Animacy. The MEG and EEG findings largely converged between 300-500ms, but diverged in their univariate and multivariate responses to the anomalies between 600-1000ms. We interpret the full set of results within a predictive coding framework. In addition to the theoretical significance of these findings, we discuss the methodological implications of the convergence and divergence between the univariate and multivariate results, as well as between the MEG and EEG results. We argue that a deeper understanding of language processing can be achieved by integrating different analysis approaches and techniques.

DIFFERENTIAL CONTRIBUTIONS OF THE SPECTRO-TEMPORAL AND VOCAL CHARACTERISTICS OF AUDITORY PSEUDOWORDS TO MULTIPLE SOUND-SYMBOLIC MAPPINGS.

Sound symbolism, the idea that the sound of a word alone can convey its meaning, is often studied using auditory pseudowords. For example, people reliably assign the auditory pseudowords "bouba" and "kiki" to rounded and pointed shapes, respectively. Previously we showed that representational dissimilarity matrices (RDMs) of the shape ratings of auditory pseudowords correlated significantly with RDMs of acoustic parameters reflecting spectro-temporal variations; the ratings also correlated significantly with voice quality features. Here, participants rated auditory pseudowords on scales representing categorical opposites across seven meaning domains, including shape. Examination of the relationships of the perceptual ratings to spectro-temporal and vocal parameters of the pseudowords essentially replicated our previous findings for shape while varying patterns emerged for the other domains. Thus, the spectro-temporal and vocal properties of spoken pseudowords contribute differentially to sound-symbolic mapping depending on the meaning domain.

Microstructural and functional substrates underlying dispositional greed and its link with trait but not state impulsivity.

The interplay between personality traits and impulsivity has long been a central theme in psychology and psychiatry. However, the potential association between Greed Personality Traits (GPT) and impulsivity, encompassing both trait and state impulsivity and future time perspective, remains largely unexplored. To address these issues, we employed questionnaires and an inter-temporal choice task to estimate corresponding trait/state impulsivity and collected multi-modal neuroimaging data (resting-state functional imaging: n = 430; diffusion-weighted imaging: n = 426; task-related functional imaging: n = 53) to investigate the underlying microstructural and functional substrates. Behavioral analyses revealed that GPT mediated the association between time perspective (e.g., present fatalism) and trait impulsivity (e.g., motor impulsivity). Functional imaging analyses further identified that brain activation strengths and patterns related to delay length, particularly in the dorsomedial prefrontal cortex, superior parietal lobule, and cerebellum, were associated with GPT. Moreover, individuals with similar levels of greed exhibited analogous spontaneous brain activity patterns, predominantly in the Default Mode Network (DMN), Fronto-Parietal Network (FPN), and Visual Network (VIS). Diffusion imaging analysis observed specific microstructural characteristics in the spinocerebellar/pontocerebellar fasciculus, internal/external capsule, and corona radiata that support the formation of GPT. Furthermore, the corresponding neural activation pattern, spontaneous neural activity pattern, and analogous functional couplings among the aforementioned brain regions mediated the relationships between time perspective and GPT and between GPT and motor impulsivity. These findings provide novel insights into the possible pathway such as time perspective → dispositional greed → impulsivity and uncover their underlying microstructural and functional substrates.

Commonalities and variations in emotion representation across modalities and brain regions.

Humans express emotions through various modalities such as facial expressions and natural language. However, the relationships between emotions expressed through different modalities and their correlations with neural activities remain uncertain. Here, we aimed to unveil some of these uncertainties by investigating the similarity of emotion representations across modalities and brain regions. First, we represented various emotion categories as multi-dimensional vectors derived from visual (face), linguistic, and visio-linguistic data, and used representational similarity analysis to compare these modalities. Second, we examined the linear transferability of emotion representation from other modalities to the visual modality. Third, we compared the representational structure derived in the first step with those from brain activities across 360 regions. Our findings revealed that emotion representations share commonalities across modalities with modality-type dependent variations, and they can be linearly mapped from other modalities to the visual modality. Additionally, emotion representations in uni-modalities showed relatively higher similarity with specific brain regions, while multi-modal emotion representation was most similar to representations across the entire brain region. These findings suggest that emotional experiences are represented differently across various brain regions with varying degrees of similarity to different modality types, and that they may be multi-modally conveyable in visual and linguistic domains.

The primacy of taxonomic semantic organization over thematic semantic organization during picture naming.

Different organizational structures have been argued to underlie semantic knowledge about concepts; taxonomic organization, based on shared features, and thematic organization based on co-occurrence in common scenes and scenarios. The goal of the current study is to examine which of the two organizational systems are more engaged in the semantic context of a picture naming task. To address this question, we examined the representational structure underlying the semantic space in different picture naming tasks by applying representational similarity analysis (RSA) to electroencephalography (EEG) datasets. In a series of experiments, EEG signals were collected while participants named pictures under different semantic contexts. Study 1 reanalyzes existing data from semantic contexts directing attention to taxonomic organization and semantic contexts that are not biased towards either taxonomic or thematic organization. In Study 2 we keep the stimuli the same and vary semantic contexts to draw attention to either taxonomic or thematic organization. The RSA approach allows us to examine the pairwise similarity in scalp-recorded amplitude patterns at each time point following the onset of the picture and relate it to theoretical taxonomic and thematic measures derived from computational models of semantics. Across all tasks, the similarity structure of scalp-recorded neural activity correlated better with taxonomic than thematic measures, in time windows associated with semantic processing. Most strikingly, we found that the scalp-recorded patterns of neural activity between taxonomically related items were more similar to each other than the scalp-recorded patterns of neural activity for thematically related or unrelated items, even in tasks that makes thematic information more salient. These results suggest that the principle semantic organization of these concepts during picture naming is taxonomic, at least in the context of picture naming.

Representational dissimilarity component analysis (ReDisCA).

The principle of Representational Similarity Analysis (RSA) posits that neural representations reflect the structure of encoded information, allowing exploration of spatial and temporal organization of brain information processing. Traditional RSA when applied to EEG or MEG data faces challenges in accessing activation time series at the brain source level due to modeling complexities and insufficient geometric/anatomical data. To overcome this, we introduce Representational Dissimilarity Component Analysis (ReDisCA), a method for estimating spatial-temporal components in EEG or MEG responses aligned with a target representational dissimilarity matrix (RDM). ReDisCA yields informative spatial filters and associated topographies, offering insights into the location of "representationally relevant" sources. Applied to evoked response time series, ReDisCA produces temporal source activation profiles with the desired RDM. Importantly, while ReDisCA does not require inverse modeling its output is consistent with EEG and MEG observation equation and can be used as an input to rigorous source localization procedures. Demonstrating ReDisCA's efficacy through simulations and comparison with conventional methods, we show superior source localization accuracy and apply the method to real EEG and MEG datasets, revealing physiologically plausible representational structures without inverse modeling. ReDisCA adds to the family of inverse modeling free methods such as independent component analysis (Makeig, 1995), Spatial spectral decomposition (Nikulin, 2011), and Source power comodulation (Dähne, 2014) designed for extraction sources with desired properties from EEG or MEG data. Extending its utility beyond EEG and MEG analysis, ReDisCA is likely to find application in fMRI data analysis and exploration of representational structures emerging in multilayered artificial neural networks.

Multivariate Neural Patterns of Reward and Anxiety in Adolescents with Anorexia Nervosa.

People with anorexia nervosa (AN) commonly exhibit elevated anxiety and atypical reward responsiveness. To examine multivariate neural patterns associated with reward and the impact of anxiety on reward, we analyzed fMRI data from a monetary reward task using representational similarity analysis, a multivariate approach that measures trial-by-trial consistency of neural responses. Twenty-five adolescent girls with AN and 22 mildly anxious controls lacking any history of AN were presented personalized anxiety-provoking or neutral words before receiving a reward, and neural response patterns in reward regions were analyzed. Consistent with our preregistered hypothesis, AN participants showed lower representational similarity than controls during neutral-word rewarded trials. Within groups, controls showed significant representational similarity in reward circuit regions including the left nucleus accumbens, left basolateral amygdala, and left medial orbitofrontal cortex, which were not observed in AN. Further, reward-related prefrontal cognitive control areas - left ventrolateral prefrontal cortex and left dorsolateral prefrontal cortex - showed significant representational similarity in both groups, but a larger spatial extent in controls. Contrary to predictions, there were no significant between-group differences for the effects of anxiety-words on reward representational similarity, and representational similarity did not predict longitudinal symptom change over six months. Overall, the results demonstrate relatively inconsistent trial-by-trial responses to reward receipt in the neutral state in AN compared with controls in both reward circuit and cognitive control regions, but no significant differential effects of anxiety states on reward responses. These results add to dynamic understandings of reward processing in AN that have potential implications for planning and guiding reward-focused interventions.

The computational and neural substrates of individual differences in impulsivity under loss framework.

Numerous neuroimaging studies have identified significant individual variability in intertemporal choice, often attributed to three neural mechanisms: (1) increased reward circuit activity, (2) decreased cognitive control, and (3) prospection ability. These mechanisms that explain impulsivity, however, have been primarily studied in the gain domain. This study extends this investigation to the loss domain. We employed a hierarchical Bayesian drift-diffusion model (DDM) and the inter-subject representational similarity approach (IS-RSA) to investigate the potential computational neural substrates underlying impulsivity in loss domain across two experiments (n = 155). These experiments utilized a revised intertemporal task that independently manipulated the amounts of immediate and delayed-loss options. Behavioral results demonstrated positive correlations between the drift rate, measured by the DDM, and the impulsivity index K in Exp. 1 (n = 97) and were replicated in Exp. 2 (n = 58). Imaging analyses further revealed that the drift rate significantly mediated the relations between brain properties (e.g., prefrontal cortex activations and gray matter volume in the orbitofrontal cortex and precuneus) and K in Exp. 1. IS-RSA analyses indicated that variability in the drift rate also mediated the associations between inter-subject variations in activation patterns and individual differences in K. These findings suggest that individuals with similar impulsivity levels are likely to exhibit similar value processing patterns, providing a potential explanation for individual differences in impulsivity within a loss framework.

Emotional concepts shape the perceptual representation of body expressions.

Emotion perception interacts with how we think and speak, including our concept of emotions. Body expression is an important way of emotion communication, but it is unknown whether and how its perception is modulated by conceptual knowledge. In this study, we employed representational similarity analysis and conducted three experiments combining semantic similarity, mouse-tracking task, and one-back behavioral task with electroencephalography and functional magnetic resonance imaging techniques, the results of which show that conceptual knowledge predicted the perceptual representation of body expressions. Further, this prediction effect occurred at approximately 170 ms post-stimulus. The neural encoding of body expressions in the fusiform gyrus and lingual gyrus was impacted by emotion concept knowledge. Taken together, our results indicate that conceptual knowledge of emotion categories shapes the configural representation of body expressions in the ventral visual cortex, which offers compelling evidence for the constructed emotion theory.

A Multidimensional Neural Representation of Face Impressions.

From a glimpse of a face, people form trait impressions that operate as facial stereotypes, which are largely inaccurate yet nevertheless drive social behavior. Behavioral studies have long pointed to dimensions of trustworthiness and dominance that are thought to underlie face impressions due to their evolutionarily adaptive nature. Using human neuroimaging (N = 26, 19 female, 7 male), we identify a two-dimensional representation of faces' inferred traits in the middle temporal gyrus (MTG), a region involved in domain-general conceptual processing including the activation of social concepts. The similarity of neural-response patterns for any given pair of faces in the bilateral MTG was predicted by their proximity in trustworthiness-dominance space, an effect that could not be explained by mere visual similarity. This MTG trait-space representation occurred automatically, was relatively invariant across participants, and did not depend on the explicit endorsement of face impressions (i.e., beliefs that face impressions are valid and accurate). In contrast, regions involved in high-level social reasoning (the bilateral temporoparietal junction and posterior superior temporal sulcus; TPJ-pSTS) and entity-specific social knowledge (the left anterior temporal lobe; ATL) also exhibited this trait-space representation but only among participants who explicitly endorsed forming these impressions. Together, the findings identify a two-dimensional neural representation of face impressions and suggest that multiple implicit and explicit mechanisms give rise to biases based on facial appearance. While the MTG implicitly represents a multidimensional trait space for faces, the TPJ-pSTS and ATL are involved in the explicit application of this trait space for social evaluation and behavior.

What makes an event significant: an fMRI study on self-defining memories.

Self-defining memories are highly significant personal memories that contribute to an individual's life story and identity. Previous research has identified 4 key subcomponents of self-defining memories: content, affect, specificity, and self-reflection. However, these components were not tested under functional neuroimaging. In this study, we first explored how self-defining memories distinguish themselves from everyday memories (non-self-defining) through their associated brain activity. Next, we evaluated the different self-defining memory subcomponents through their activity in the underlying brain system. Participants recalled both self-defining and non-self-defining memories under functional MRI and evaluated the 4 subcomponents for each memory. Multivoxel pattern analysis uncovered a brain system closely related to the default mode network to discriminate between self-defining and non-self-defining memories. Representational similarity analysis revealed the neural coding of each subcomponent. Self-reflection was coded mainly in the precuneus, middle and inferior frontal gyri, and cingulate, lateral occipital, and insular cortices. To a much lesser extent, content coding was primarily in the left angular gyrus and fusiform gyrus. No region was found to represent information on affect and specificity. Our findings highlight the marked difference in brain processing between significant and non-significant memories, and underscore self-reflection as a predominant factor in the formation and maintenance of self-defining memories, inviting a reassessment of what constitutes significant memories.

The neural representation of stereotype content.

Judgments about social groups are characterized by their position in a representational space defined by two axes, warmth and competence. We examined serial dependence (SD) in evaluations of warmth and competence while measuring participants' electroencephalographic (EEG) activity, as a means to address the independence between these two psychological axes. SD is the attraction of perceptual reports towards things seen in the recent past and has recently been intensely investigated in vision. SD occurs at multiple levels of visual processing, from basic features to meaningful objects. The current study aims to (1) measure whether SD occurs between non-visual objects, in particular social groups and (2) uncover the neural correlates of social group evaluation and SD using EEG. Participants' judgments about social groups such as "nurses" or "accountants" were serially dependent, but only when the two successive groups were close in representational space. The pattern of results argues in favor of a non-separability between the two axes, because groups nearby on one dimension but far on the other were not subject to SD, even though that other dimension was irrelevant to the task at hand. Using representational similarity analysis, we found a brain signature that differentiated social groups as a function of their position in the representational space. Our results thus argue that SD may be a ubiquitous cognitive phenomenon, that social evaluations are serially dependent, and that reproducible neural signatures of social evaluations can be uncovered.

Sensitivity and specificity of the action observation network to kinematics, target object, and gesture meaning.

Hierarchical models have been proposed to explain how the brain encodes actions, whereby different areas represent different features, such as gesture kinematics, target object, action goal, and meaning. The visual processing of action-related information is distributed over a well-known network of brain regions spanning separate anatomical areas, attuned to specific stimulus properties, and referred to as action observation network (AON). To determine the brain organization of these features, we measured representational geometries during the observation of a large set of transitive and intransitive gestures in two independent functional magnetic resonance imaging experiments. We provided evidence for a partial dissociation between kinematics, object characteristics, and action meaning in the occipito-parietal, ventro-temporal, and lateral occipito-temporal cortex, respectively. Importantly, most of the AON showed low specificity to all the explored features, and representational spaces sharing similar information content were spread across the cortex without being anatomically adjacent. Overall, our results support the notion that the AON relies on overlapping and distributed coding and may act as a unique representational space instead of mapping features in a modular and segregated manner.

Multivariate pattern analysis of cooperation and competition in constructive action.

The neural underpinning of cooperative and competitive constructive activity has been investigated using mass-univariate approaches. In this study, we sought to compare the results of these approaches with the results of multivariate pattern analysis (MVPA). In particular, we wanted to test whether MVPA supports the claim made in previous studies that cooperation is associated with the activity of reward-related brain circuits. Participants were required to construct a pattern on the screen either individually or in cooperation or competition with another person during an fMRI scan. Both the MVPA classification methods and the representational similarity analysis indicated the involvement of orbitofrontal and ventromedial prefrontal areas in processes that distinguish between cooperation and competition, and activation analysis showed that these areas are more active during cooperation than during competition. However, a single trial analysis showed that the effect was reversed when only winning trials were considered. In these trials, activation of reward-related areas was higher during competition than during cooperation. Moreover, the contrast between won and lost trials in terms of reward circuits involvement was sharper under competition than under cooperation. Thus, although cooperation can be generally more rewarding than competition, it is associated with smaller difference between trials lost and trials won in terms of reward circuits activation. One may speculate that in cooperation, victory and defeat are shared with the partner and, contrary to competition, are not experienced as personal achievement or failure.

Are older adults less generous? Age differences in emotion-related social decision making.

In social interaction, age-related differences in emotional processing may lead to varied social decision making between young and older adults. However, previous studies of social decision making have paid less attention to the interactants' emotions, leaving age differences and underlying neural mechanisms unexplored. To address this gap, the present study combined functional and structural magnetic resonance imaging, employing a modified dictator game task with recipients displaying either neutral or sad facial expressions. Behavioral results indicated that although older adults' overall allocations did not differ significantly from those of young adults, older adults' allocations showing a decrease in emotion-related generosity compared to young adults. Using representational similarity analysis, we found that older adults showed reduced neural representations of recipients' emotions and gray matter volume in the right anterior cingulate gyrus (ACC), right insula, and left dorsomedial prefrontal cortex (DMPFC) compared to young adults. More importantly, mediation analyses indicated that age influenced allocations not only through serial mediation of neural representations of the right insula and left DMPFC, but also through serial mediation of the mean gray matter volume of the right ACC and left DMPFC. This study identifies the potential neural pathways through which age affects emotion-related social decision making, advancing our understanding of older adults' social interaction behavior that they may not be less generous unless confronted with individuals with specific emotions.

Contributions of repeated learning to memory in humans: insights from single-neuron recordings in the hippocampus and amygdala.

Despite the well-established phenomenon of improved memory performance through repeated learning, studies investigating the associated neural mechanisms have yielded complex and sometimes contradictory findings, and direct evidence from human neuronal recordings has been lacking. This study employs single-neuron recordings with exceptional spatial-temporal resolution, combined with representational similarity analysis, to explore the neural dynamics within the hippocampus and amygdala during repeated learning. Our results demonstrate that in the hippocampus, repetition enhances both representational specificity and fidelity, with these features predicting learning times. Conversely, the amygdala exhibits heightened representational specificity and fidelity during initial learning but does not show improvement with repetition, suggesting functional specialization of the hippocampus and amygdala during different stages of the learning repetition. Specifically, the hippocampus appears to contribute to sustained engagement necessary for benefiting from repeated learning, while the amygdala may play a role in the representation of novel items. These findings contribute to a comprehensive understanding of the intricate interplay between these brain regions in memory processes. Significance statement  For over a century, understanding how repetition contributes to memory enhancement has captivated researchers, yet direct neuronal evidence has been lacking, with a primary focus on the hippocampus and a neglect of the neighboring amygdala. Employing advanced single-neuron recordings and analytical techniques, this study unveils a nuanced functional specialization within the amygdala-hippocampal circuit during various learning repetition. The results highlight the hippocampus's role in sustaining engagement for improved memory with repetition, contrasting with the amygdala's superior ability in representing novel items. This exploration not only deepens our comprehension of memory enhancement intricacies but also sheds light on potential interventions to optimize learning and memory processes.

The neural representation of body part concepts.

Neuropsychological and neuroimaging studies provide evidence for a degree of category-related organization of conceptual knowledge in the brain. Some of this evidence indicates that body part concepts are distinctly represented from other categories; yet, the neural correlates and mechanisms underlying these dissociations are unclear. We expand on the limited prior data by measuring functional magnetic resonance imaging responses induced by body part words and performing a series of analyses investigating the cortical representation of this semantic category. Across voxel-level contrasts, pattern classification, representational similarity analysis, and vertex-wise encoding analyses, we find converging evidence that the posterior middle temporal gyrus, the supramarginal gyrus, and the ventral premotor cortex in the left hemisphere play important roles in the preferential representation of this category compared to other concrete objects.