Audiovisual integration in the human brain: a coordinate-based meta-analysis.

People can seamlessly integrate a vast array of information from what they see and hear in the noisy and uncertain world. However, the neural underpinnings of audiovisual integration continue to be a topic of debate. Using strict inclusion criteria, we performed an activation likelihood estimation meta-analysis on 121 neuroimaging experiments with a total of 2,092 participants. We found that audiovisual integration is linked with the coexistence of multiple integration sites, including early cortical, subcortical, and higher association areas. Although activity was consistently found within the superior temporal cortex, different portions of this cortical region were identified depending on the analytical contrast used, complexity of the stimuli, and modality within which attention was directed. The context-dependent neural activity related to audiovisual integration suggests a flexible rather than fixed neural pathway for audiovisual integration. Together, our findings highlight a flexible multiple pathways model for audiovisual integration, with superior temporal cortex as the central node in these neural assemblies.

Functional lateralization of the medial temporal lobe in novel associative processing during creativity evaluation.

Although hemispheric lateralization of creativity has been a longstanding topic of debate, the underlying neurocognitive mechanism remains poorly understood. Here we designed 2 types of novel stimuli-"novel useful and novel useless," adapted from "familiar useful" designs taken from daily life-to demonstrate how the left and right medial temporal lobe (MTL) respond to novel designs of different usefulness. Taking the "familiar useful" design as a baseline, we found that the right MTL showed increased activation in response to "novel useful" designs, followed by "novel useless" ones, while the left MTL only showed increased activation in response to "novel useful" designs. Calculating an asymmetry index suggests that usefulness processing is predominant in the left MTL, whereas the right MTL is predominantly involved in novelty processing. Moreover, the left parahippocampal gyrus (PHG) showed stronger functional connectivity with the anterior cingulate cortex when responding to "novel useless" designs. In contrast, the right PHG showed stronger connectivity with the amygdala, midbrain, and hippocampus. Critically, multivoxel representational similarity analyses revealed that the left MTL was more effective than the right MTL at distinguishing the usefulness differences in novel stimuli, while representational patterns in the left PHG positively predicted the post-behavior evaluation of "truly creative" products. These findings suggest an apparent dissociation of the left and right MTL in integrating the novelty and usefulness information and novel associative processing during creativity evaluation, respectively. Our results provide novel insights into a longstanding and controversial question in creativity research by demonstrating functional lateralization of the MTL in processing novel associations.

Social Support, Internalized HIV Stigma, Resilience and Depression Among People Living with HIV: A Moderated Mediation Analysis.

Internalized HIV stigma has been associated with depression among people living with HIV (PLWH). However, it is still unclear whether resilience would mediate the association between internalized HIV stigma and depression and how this indirect effect would be moderated by social support. Data were collected from 402 PLWH in South Carolina using a cross-sectional survey. Data were fitted using a path model that specified the extent to which internalized HIV stigma and depression were related through resilience and how this effect was moderated by social support. Sociodemographic characteristics were included in the model as covariates. The indirect effect of internalized HIV stigma on depression through resilience was statistically significant for high social support but not for low social support. To mitigate negative impacts of internalized HIV stigma on mental health of PLWH, intervention efforts should integrate multilevel components for promoting both resilience and social support.

Negative emotion amplifies retrieval practice effect for both task-relevant and task-irrelevant information.

Selective retrieval of task-relevant information often facilitates memory retention of that information. However, it is still unclear if selective retrieval of task-relevant information can alter memory for task-irrelevant information, and the role of emotional arousal in it. In two experiments, we used emotional and neutral faces as stimuli, and participants were asked to memorise the name (who is this person?) and location (where does he/she come from?) associated with each face in initial study. Then, half of the studied faces were presented as cues, and participants were asked to retrieve the corresponding names (Experiment 1) or locations (Experiment 2). Finally, all the faces were presented and participants were asked to retrieve both the corresponding names and locations. The results of the final test showed that retrieval practice not only enhanced memory of task-relevant information but also enhanced memory of task-irrelevant information. More importantly, negative emotion amplified the retrieval practice effect overall, with a larger retrieval-induced benefit for the negative than neutral condition. These findings demonstrated an emotional arousal amplification effect on retrieval-induced enhancement effects, suggesting that the advantage of the retrieved memory representations can be amplified by emotional arousal even without explicit goals in a task setting.

SCOPE: The South Carolina psycholinguistic metabase.

The number of databases that provide various measurements of lexical properties for psycholinguistic research has increased rapidly in recent years. The proliferation of lexical variables, and the multitude of associated databases, makes the choice, comparison, and standardization of these variables in psycholinguistic research increasingly difficult. Here, we introduce The South Carolina Psycholinguistic Metabase (SCOPE), which is a metabase (or a meta-database) containing an extensive, curated collection of psycholinguistic variable values from major databases. The metabase currently contains 245 lexical variables, organized into seven major categories: General (e.g., frequency), Orthographic (e.g., bigram frequency), Phonological (e.g., phonological uniqueness point), Orth-Phon (e.g., consistency), Semantic (e.g., concreteness), Morphological (e.g., number of morphemes), and Response variables (e.g., lexical decision latency). We hope that SCOPE will become a valuable resource for researchers in psycholinguistics and affiliated disciplines such as cognitive neuroscience of language, computational linguistics, and communication disorders. The availability and ease of use of the metabase with comprehensive set of variables can facilitate the understanding of the unique contribution of each of the variables to word processing, and that of interactions between variables, as well as new insights and development of improved models and theories of word processing. It can also help standardize practice in psycholinguistics. We demonstrate use of the metabase by measuring relationships between variables in multiple ways and testing their individual contribution towards a number of dependent measures, in the most comprehensive analysis of this kind to date. The metabase is freely available at go.sc.edu/scope.

Evaluating non-affective cross-modal congruence effects on emotion perception.

Although numerous studies have shown that people are more likely to integrate consistent visual and auditory signals, the role of non-affective congruence in emotion perception is unclear. This registered report examined the influence of non-affective cross-modal congruence on emotion perception. In Experiment 1, non-affective congruence was manipulated by matching or mismatching gender between visual and auditory modalities. Participants were instructed to attend to emotion information from only one modality while ignoring the other modality. Experiment 2 tested the inverse effectiveness rule by including both noise and noiseless conditions. Across two experiments, we found the effects of task-irrelevant emotional signals from one modality on emotional perception in the other modality, reflected in affective congruence, facilitation, and affective incongruence effects. The effects were stronger for the attend-auditory compared to the attend-visual condition, supporting a visual dominance effect. The effects were stronger for the noise compared to the noiseless condition, consistent with the inverse effectiveness rule. We did not find evidence for the effects of non-affective congruence on audiovisual integration of emotion across two experiments, suggesting that audiovisual integration of emotion may not require automatic integration of non-affective congruence information.

An fMRI Study of Affective Congruence across Visual and Auditory Modalities.

Evaluating multisensory emotional content is a part of normal day-to-day interactions. We used fMRI to examine brain areas sensitive to congruence of audiovisual valence and their overlap with areas sensitive to valence. Twenty-one participants watched audiovisual clips with either congruent or incongruent valence across visual and auditory modalities. We showed that affective congruence versus incongruence across visual and auditory modalities is identifiable on a trial-by-trial basis across participants. Representations of affective congruence were widely distributed with some overlap with the areas sensitive to valence. Regions of overlap included bilateral superior temporal cortex and right pregenual anterior cingulate. The overlap between the regions identified here and in the emotion congruence literature lends support to the idea that valence may be a key determinant of affective congruence processing across a variety of discrete emotions.

The function of the hippocampus and middle temporal gyrus in forming new associations and concepts during the processing of novelty and usefulness features in creative designs.

Creative thought relies on the reorganization of existing knowledge to generate novel and useful concepts. However, how these new concepts are formed, especially through the processing of novelty and usefulness (which are usually regarded as the key properties of creativity), is not clear. Taking familiar and useful (FU) objects/designs as the starting point or fundamental baseline, we modified them into novel and useless (NS) objects/designs or novel and useful (NU) ones (i.e., truly creative ones) to investigate how the features of novelty and usefulness are processed (processing of novelty: NU minus FU; processing of usefulness: NU minus NS). Specifically, we predicted that the creative integration of novelty and usefulness entails not only the formation of new associations, which could be critically mediated by the hippocampus and adjacent medial temporal lobe (MTL) areas, but also the formation of new concepts or categories, which is supported by the middle temporal gyrus (MTG). We found that both the MTL and the MTG were involved in the processing of novelty and usefulness. The MTG showed distinctive patterns of information processing, reflected by strengthened functional connectivity with the hippocampus to construct new concepts and strengthened functional connectivity with the executive control system to break the boundaries of old concepts. Additionally, participants' subjective evaluations of concept distance showed that the distance between the familiar concept (FU) and the successfully constructed concept (NU) was larger than that between the FU and the unsuccessfully constructed concept (NS), and this pattern was found to correspond to the patterns of their neural representations in the MTG. These findings demonstrate the critical mechanism by which new associations and concepts are formed during novelty and usefulness processing in creative design; this mechanism may be critically mediated by the hippocampus-MTG connection.

The role of emotion arousal in the retrieval practice effect.

Many studies have shown that practicing retrieval produces better memory retention compared to restudy. Though previous literature has provided valuable insights about the retrieval practice effect, it is still unclear how emotion arousal influences the retrieval practice effect, and whether the effect would be manifested in recollection or familiarity processes. To answer these questions, in the current study, negative and neutral words were used as stimuli and participants were asked to perform a recognition test or restudy the words after initial study. At the end of the experiment, a final recognition test with involving the remember-know paradigm was shown. Behavioral data were collected with EEG recorded throughout the experiment. The behavioral retrieval practice effect was only found for the neutral but not the negative words. Consistently, significant ERP differences between the restudy and retrieval practice conditions were only found for neutral, but not negative items, which was a component from 700 to 900 ms at left-posterior electrode cluster. Moreover, we found that the effects of emotion arousal on the retrieval practice effect were mainly reflected in the recollection process. These findings provide behavioral and neural evidence that emotion arousal can influence the retrieval practice effect.

The temporal dynamics of perceptual and conceptual fluency on recognition memory.

Recognition memory can be driven by both perceptual and conceptual fluency, but when and to what extent they contribute to recognition memory remains an open question. The present study used event-related potentials (ERPs) to investigate the neural correlates of perceptual and conceptual fluency, when they gave rise to recognition. We manipulated the perceptual and conceptual fluency of retrieval cues in the recognition test independently to obtain the effects of different types of fluency. Behavioral results showed that perceptual fluency selectively affected K hits, while conceptual fluency affected R hits and K false alarms. In addition, conceptual fluency facilitated the response times of R hits. The ERP results showed that perceptual fluency effect appeared at 100-200 ms and conceptual fluency effect appeared at 300-500 ms. The parietal LPC peaked earlier for conceptually primed trials compared to unprimed trials. These results suggest that perceptual and conceptual fluency had different effects on recognition judgments, and these two types of fluency can be delineated by distinct ERP correlates. The current finding indicates that unconscious memory processes can support recognition and have provided insights into the underlying mechanism involved in recognition memory.

Modelling audiovisual integration of affect from videos and music.

Two experiments examined how affective values from visual and auditory modalities are integrated. Experiment 1 paired music and videos drawn from three levels of valence while holding arousal constant. Experiment 2 included a parallel combination of three levels of arousal while holding valence constant. In each experiment, participants rated their affective states after unimodal and multimodal presentations. Experiment 1 revealed a congruency effect in which stimulus combinations of the same extreme valence resulted in more extreme state ratings than component stimuli presented in isolation. An interaction between music and video valence reflected the greater influence of negative affect. Video valence was found to have a significantly greater effect on combined ratings than music valence. The pattern of data was explained by a five parameter differential weight averaging model that attributed greater weight to the visual modality and increased weight with decreasing values of valence. Experiment 2 revealed a congruency effect only for high arousal combinations and no interaction effects. This pattern was explained by a three parameter constant weight averaging model with greater weight for the auditory modality and a very low arousal value for the initial state. These results demonstrate key differences in audiovisual integration between valence and arousal.

Electrophysiological signals associated with fluency of different levels of processing reveal multiple contributions to recognition memory.

Processing fluency appears to influence recognition memory judgements, and the manipulation of fluency, if misattributed to an effect of prior exposure, can result in illusory memory. Although it is well established that fluency induced by masked repetition priming leads to increased familiarity, manipulations of conceptual fluency have produced conflicting results, variously affecting familiarity or recollection. Some recent studies have found that masked conceptual priming increases correct recollection (Taylor & Henson, 2012), and the magnitude of this behavioural effect correlates with analogous fMRI BOLD priming effects in brain regions associated with recollection (Taylor, Buratto, & Henson, 2013). However, the neural correlates and time-courses of masked repetition and conceptual priming were not compared directly in previous studies. The present study used event-related potentials (ERPs) to identify and compare the electrophysiological correlates of masked repetition and conceptual priming and investigate how they contribute to recognition memory. Behavioural results were consistent with previous studies: Repetition primes increased familiarity, whereas conceptual primes increased correct recollection. Masked repetition and conceptual priming also decreased the latency of late parietal component (LPC). Masked repetition priming was associated with an early P200 effect and a later parietal maximum N400 effect, whereas masked conceptual priming was only associated with a central-parietal maximum N400 effect. In addition, the topographic distributions of the N400 repetition priming and conceptual priming effects were different. These results suggest that fluency at different levels of processing is associated with different ERP components, and contributes differentially to subjective recognition memory experiences.

Masked repetition priming hinders subsequent recollection but not familiarity: A behavioral and event-related potential study.

The present study used the masked repetition priming paradigm in the study phase and the R/K paradigm in the test phase to investigate whether repetition priming can hinder recognition memory and which recognition process (familiarity or recollection) is hindered. Event-related potentials (ERPs) in the study and test phase were recorded to explore the temporal course of how repetition priming hinders subsequent recognition memory and which old/new effect (FN400 or LPC) is affected. Converging behavioral and ERP results indicated that masked repetition priming hindered subsequent recollection but not familiarity. The analysis of ERP priming effects in the study phase indicated that primed words were associated with less negative N400 and less positive LPC compared to unprimed words. The analysis of the priming effect as a function of subsequent memory revealed that only the LPC priming effect was predictive of priming effect on subsequent memory, which suggested that the "prediction-error" account might be a possible explanation of how repetition priming affects subsequent recognition memory.

The role of retrieval mode and retrieval orientation in retrieval practice: insights from comparing recognition memory testing formats and restudying.

The effectiveness of retrieval practice for aiding long-term memory, referred to as the testing effect, has been widely demonstrated. However, the specific neurocognitive mechanisms underlying this phenomenon remain unclear. In the present study, we sought to explore the role of pre-retrieval processes at initial testing on later recognition performance by using event-related potentials (ERPs). Subjects studied two lists of words (Chinese characters) and then performed a recognition task or a source memory task, or restudied the word lists. At the end of the experiment, subjects received a final recognition test based on the remember-know paradigm. Behaviorally, initial testing (active retrieval) enhanced memory retention relative to restudying (passive retrieval). The retrieval mode at initial testing was indexed by more positive-going ERPs for unstudied items in the active-retrieval tasks than in passive retrieval from 300 to 900 ms. Follow-up analyses showed that the magnitude of the early ERP retrieval mode effect (300-500 ms) was predictive of the behavioral testing effect later on. In addition, the ERPs for correctly rejected new items during initial testing differed between the two active-retrieval tasks from 500 to 900 ms, and this ERP retrieval orientation effect predicted differential behavioral testing gains between the two active-retrieval conditions. Our findings confirm that initial testing promotes later retrieval relative to restudying, and they further suggest that adopting pre-retrieval processing in the forms of retrieval mode and retrieval orientation might contribute to these memory enhancements.

Memory enhancement for emotional words is attributed to both valence and arousal.

We do not memorize items in our surroundings with equal priority. Previous literature has widely shown that emotional stimuli are better remembered than neutral stimuli. However, given emotional stimuli and neutral stimuli often differ in both valence and arousal dimensions, it remains unclear whether the enhancement effects can be attributed to valence, or just to arousal. Importantly, most prior studies relied on a relatively small number of stimuli and non-emotional factors such as word length, imageability and other confounds were hard to control. To address these challenges, we analyzed multiple large databases of recognition memory and free recall tasks from previous research by items with many lexical and semantic covariates included, examining the effects of valence or arousal when controlling for each other. Our results showed a U-shaped relationship between valence and memory performance for both recognition and free recall, and a linear relationship between arousal and memory performance for both tasks. These findings showed that the memory enhancement effects can be attributed to both valence and arousal. We demonstrated these effects with generalizability across many stimuli and controlled for non-emotional factors. Together, these findings disentangle the contribution of valence and arousal in emotional memory enhancement effects and provide insights for current major theories of emotional memory.

The effect of auditory valence on subsequent visual semantic processing.

Emotion influences many cognitive processes and plays an important role in our daily life. Previous studies focused on the effects of arousal on subsequent cognitive processing, but the effect of valence on subsequent semantic processing is still not clear. The present study examined the effect of auditory valence on subsequent visual semantic processing when controlling for arousal. We used instrumental music clips varying in valence while matching in arousal to induce valence states and asked participants to make natural or man-made judgements on subsequent neutral objects. We found that positive and negative valences similarly impaired subsequent semantic processing compared with neutral valence. The linear ballistic accumulator model analyses showed that the valence effects can be attributed to drift rate differences, suggesting that the effects are likely related to attentional selection. Our findings are consistent with a motivated attention model, indicating comparable attentional capture by both positive and negative valences in modulating subsequent cognitive processes.

Affective valence of words differentially affects visual and auditory word recognition.

Recognizing written or spoken words involves a sequence of processing stages, transforming sensory features into lexical-semantic representations. Whereas the later processing stages are common across modalities, the initial stages are modality-specific. In the visual modality, previous studies have shown that words with positive valence are recognized faster than neutral words. Here, we examined whether the effects of valence on word recognition are specific to the visual modality or are common across visual and auditory modalities. To address this question, we analyzed multiple large databases of visual and auditory lexical decision tasks, relating the valence of words to lexical decision times while controlling for a large number of variables, including arousal and frequency. We found that valence differentially influenced visual and auditory word recognition. Valence had an asymmetric effect on visual lexical decision times, primarily speeding up recognition of positive words. By contrast, valence had a symmetric effect on auditory lexical decision times, with both negative and positive words speeding up word recognition relative to neutral words. The modality-specificity of valence effects were consistent across databases and were observed when the same sets of words were compared across modalities. We interpret these findings as indicating that valence influences word recognition partly at the sensory-perceptual stage. We relate these effects to the effects of positive (reward) and negative (punishment) reinforcers on perception. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Crossmodal negativity bias in semantic processing.

Negative events have greater influence on cognitive processing compared with positive events, a phenomenon referred to as the negativity bias. Previous studies have shown that reaction times (RTs) to negatively valenced items are slower in semantic tasks. According to the automatic vigilance hypothesis, these effects are caused by preferential attention to negative stimuli or features diverting cognitive resources away from semantic processing. However, it is still unclear whether the negativity bias can be modulated by affective context in a crossmodal setting and how that occurs. Experiment 1 examined individually presented pictures and words and established that participants were slower to judge negatively valenced picture and word targets in a semantic task. Experiments 2 and 3 probed the crossmodal influences of valence on subsequent semantic processing by using short music clips as primes and valenced pictures or words as targets. Both experiments demonstrated that priming negative versus positive music slowed RTs in a semantic task, irrespective of target valence. Hierarchical Bayesian drift diffusion model analyses suggest that the slow-down effects for negative conditions are mainly attributed to reduced drift rates. Together, these experiments demonstrate that negative auditory valence can impair subsequent semantic processing of visual targets in an additive fashion. These results provide further support for the automatic vigilance hypothesis. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Modality-general and modality-specific audiovisual valence processing.

A fundamental question in affective neuroscience is whether there is a common hedonic system for valence processing independent of modality, or there are distinct neural systems for different modalities. To address this question, we used both region of interest and whole-brain representational similarity analyses on functional magnetic resonance imaging data to identify modality-general and modality-specific brain areas involved in valence processing across visual and auditory modalities. First, region of interest analyses showed that the superior temporal cortex was associated with both modality-general and auditory-specific models, while the primary visual cortex was associated with the visual-specific model. Second, the whole-brain searchlight analyses also identified both modality-general and modality-specific representations. The modality-general regions included the superior temporal, medial superior frontal, inferior frontal, precuneus, precentral, postcentral, supramarginal, paracentral lobule and middle cingulate cortices. The modality-specific regions included both perceptual cortices and higher-order brain areas. The valence representations derived from individualized behavioral valence ratings were consistent with these results. Together, these findings suggest both modality-general and modality-specific representations of valence.

Effects of retrieval and emotion on within-item associative memory - Evidence from ERP and oscillatory subsequent memory effects.

Previous studies have shown the effects of retrieval practice and emotion on associative memory separately. However, it is yet not clear what are the related neural mechanisms and how the two factors together influence associative memory? We examined this question by instructing participants to memorize emotional or neutral words using different ways of learning. Behaviorally, the source memory was enhanced by the retrieval practice compared to the restudy condition, and impaired by the negative compared to the neutral condition without an interaction. Consistent neural effects of retrieval practice were also found, in which subsequent memory effects (SME) of 500-700 ms parietal ERPs and alpha desynchronization were found for the retrieval practice but not for the restudy. No significant difference of SME for ERPs and time-frequency analyses regarding the emotion effect was found. These results demonstrated the neural mechanism for the effects of emotion and retrieval practice on subsequent memory.